Server Optimizations - provide a "server only" download option + some guidance

[Benevolent27]

Hello. Well so this is a two part suggestion aimed at helping server operators.

1. This suggestion is relatively simple. It is to have a separate jar file which operates ONLY as a server. It could be run exactly as the server is now, with the same switches, except when it updates it ONLY downloads files the server actually needs and excludes files needed by the client, such as the tutorial videos. This would save on bandwidth, update time, and allow for much smaller backups.

2. Provide some guidance to server operators for configs. For instance, give recommended values for a few different system configurations. For example, let's say someone has a quad-core xeon which runs at 2.4ghz a core and 32 gb of RAM. What would be a good starting point for this server to maximize the power of the server? What if a server has a newer and more powerful six-core xeon processor running at 3.2 ghz but only 8gb of RAM? What might a good config look like? Talking to other server operators, there seems to be a lot of confusion about how to fully optimize their servers. When setting up LvD, we basically kept changing various settings and testing with different ships, but it's hard to fully understand the impact each of these settings has. We've done our best here to create as much balance as possible and to utilize the power available to us, but I know there is more tweaking that can be done. A little more guidance and verbosity in the descriptions (such as NIO size) would be rather helpful as well.

 

[Erth Paradine]

Hello. Well so this is a two part suggestion aimed at helping server operators.

1. This suggestion is relatively simple. It is to have a separate jar file which operates ONLY as a server. It could be run exactly as the server is now, with the same switches, except when it updates it ONLY downloads files the server actually needs and excludes files needed by the client, such as the tutorial videos. This would save on bandwidth, update time, and allow for much smaller backups.

Why not simply exclude the video files from your backups?

As for duration of time, the video files add 10-15 seconds to our overall update time. That's not really a concern from where I'm sitting, especially when it takes 15+ minutes for the mandatory creation of a .zip file prior to update (a process which does already exclude video files).

2. Provide some guidance to server operators for configs. For instance, give recommended values for a few different system configurations. For example, let's say someone has a quad-core xeon which runs at 2.4ghz a core and 32 gb of RAM. What would be a good starting point for this server to maximize the power of the server? What if a server has a newer and more powerful six-core xeon processor running at 3.2 ghz but only 8gb of RAM? What might a good config look like? Talking to other server operators, there seems to be a lot of confusion about how to fully optimize their servers. When setting up LvD, we basically kept changing various settings and testing with different ships, but it's hard to fully understand the impact each of these settings has. We've done our best here to create as much balance as possible and to utilize the power available to us, but I know there is more tweaking that can be done. A little more guidance and verbosity in the descriptions (such as NIO size) would be rather helpful as well.

I've supported datacenter environments for over two decades, and in my experience, the most basic of baseline suggestions (such as what already exists) are more than adequate. Success using different settings will only occur if you're using an environment identical to one with the suggested settings. Otherwise, you're right back where you started, with blind guesswork. Any success at server tuning really requires a skilled approach, with adequate systems performance profiling, and a strong understanding of the interactions between systems, and hosted applications/services.

As for specific settings to adjust, I feel this could be documented better, but then again this is alpha code, where documentation is typically sparse. So perhaps this helps you:

These timeouts can be shortened, if your system's I/O can be improved, such as by using SATA or PCIe based SSDs, as they are disk I/O intensive routines, and can interrupt gameplay if shortened too much:

SECTOR_INACTIVE_TIMEOUT
SECTOR_INACTIVE_CLEANUP_TIMEOUT
SECTOR_AUTOSAVE_SEC

Adjust these to accommodate additional cores (follow suggestions in server.cfg):

CHUNK_REQUEST_THREAD_POOL_SIZE_TOTAL
CHUNK_REQUEST_THREAD_POOL_SIZE_CPU

More ram (and in many cases CPU) equals more pew & boom:

CONCURRENT_SIMULATION
MAX_SIMULTANIOUS_EXPLOSIONS

Additional things to evaluate (but tend to be more universally appropriate for any game server):

• Disk configurations (disk types, caching, buffering, RAID, mirroring, etc)
• Storage settings (block size, filesystem type, mount options, elevator selection)
• Networking (bandwidth, firewalling, NIC settings, TCP settings)

As for sharing very specific values. While I can't speak for all server operators, such info is held closely around here, primarily because of a very small but noisy and extremely toxic subset of this community that would undoubtedly try exploiting configuration nuances to crash servers or otherwise ruin gameplay for all others. With minimal tools to manage and mitigate such toxicity (see my signature for thoughts on tool improvements), the negatives of sharing outweigh any perceived positives. Further, with this being an alpha game...you're generally better off leveraging your own skills, to determine what works best for your particular equipment and community, as settings will undoubtedly need to be adjusted from release-to-release, and having a good grasp of the basics will help you approach specific system nuances more intelligently and expediently.

 

[Benevolent27]

Why not simply exclude the video files from your backups?

As for duration of time, the video files add 10-15 seconds to our overall update time. That's not really a concern from where I'm sitting, especially when it takes 15+ minutes for the mandatory creation of a .zip file prior to update (a process which does already exclude video files).

I do, but it still makes the update process much slower. Also, I have noticed that many server operators are not as savvy as you and I might be with scripting. This is meant to solve a broad problem.

I've supported datacenter environments for over two decades, and in my experience, the most basic of baseline suggestions (such as what already exists) are more than adequate. Success using different settings will only occur if you're using an environment identical to one with the suggested settings. Otherwise, you're right back where you started, with blind guesswork. Any success at server tuning really requires a skilled approach, with adequate systems performance profiling, and a strong understanding of the interactions between systems, and hosted applications/services.

As for specific settings to adjust, I feel this could be documented better, but then again this is alpha code, where documentation is typically sparse. So perhaps this helps you:

These timeouts can be shortened, if your system's I/O can be improved, such as by using SATA or PCIe based SSDs, as they are disk I/O intensive routines, and can interrupt gameplay if shortened too much:

SECTOR_INACTIVE_TIMEOUT
SECTOR_INACTIVE_CLEANUP_TIMEOUT
SECTOR_AUTOSAVE_SEC

Adjust these to accommodate additional cores (follow suggestions in server.cfg):

CHUNK_REQUEST_THREAD_POOL_SIZE_TOTAL
CHUNK_REQUEST_THREAD_POOL_SIZE_CPU

More ram (and in many cases CPU) equals more pew & boom:

CONCURRENT_SIMULATION
MAX_SIMULTANIOUS_EXPLOSIONS

Additional things to evaluate (but tend to be more universally appropriate for any game server):

• Disk configurations (disk types, caching, buffering, RAID, mirroring, etc)
• Storage settings (block size, filesystem type, mount options, elevator selection)
• Networking (bandwidth, firewalling, NIC settings, TCP settings)

As for sharing very specific values. While I can't speak for all server operators, such info is held closely around here, primarily because of a very small but noisy and extremely toxic subset of this community that would undoubtedly try exploiting configuration nuances to crash servers or otherwise ruin gameplay for all others. With minimal tools to manage and mitigate such toxicity (see my signature for thoughts on tool improvements), the negatives of sharing outweigh any perceived positives. Further, with this being an alpha game...you're generally better off leveraging your own skills, to determine what works best for your particular equipment and community, as settings will undoubtedly need to be adjusted from release-to-release, and having a good grasp of the basics will help you approach specific system nuances more intelligently and expediently.

How many games have you played where the settings auto-adjust for quality based on the specs of the system it is run on? When done right, these games are enhancing the fluidity of gameplay for the end-user, right? Though a tool like this would be ideal, I think, for server operators, I don't think it'd be practical at this point, which is why I suggest just having a few recommended settings files for common server setups which have big differences. For example, I've noticed some servers running on home PC's that have "ok" CPU's, lots of RAM, regular hard drives (not SSD) and slower internet. Then there are the servers which run with 4 xeon cores (2.4ghz), 8GB of RAM, and SSD's. Then there are servers with 4 xeon cores, regular hard drives, and 64GB of RAM. Then there are servers with dual xeon CPU's (4 cores apiece, 16 threads), SSD's, and 64GB of RAM. These systems could all very well have different bottlenecks, and should be configured much differently. Having an example settings file would be a great place to start to get the most out of the system configuration being used.

Regarding the server settings you mentioned, I appreciate it, and I'm sure other server operators could use the info, but I know about these already, I have adjusted them back and forth after a lot of testing for our server. We've come up with one that runs pretty darn well I think, but I know more could be done. Then there are other settings that are not documented anywhere, such as with the HSQL settings for the world file which still seems to be optimized for computers made in 1998.

Part of the reason I suggest that there be some suggested server setting config files, is because I had a heck of a time looking around for different configurations hosting options for LvD. The problem is that nobody really seems to know what the biggest bottlenecks are for StarMade, so trying to find a good balance, while maintaining a reasonable cost is difficult. Sure we could rent a top of the line server with a great dual CPU setup with 16 cores total and 32 threads, with 128gb of RAM, terrabytes of backup storage and oodles of high end SSD for the server files, but that would cost an immense amount of money each month. If I could have more clearly known where to put the most focus, then the selection process would've been much easier. By having a few config files tailored for specific setups, I could have used that as a baseline for choice, and then also as a baseline to start with before tweaking.

Now, currently one of the issues I've been having is that I'm trying to ensure the CPU actually remains busy when there is a lot going on with the server. Of course, it could be remaining low due to certain bottlenecks, but I've found my options are a bit limited in the settings files. Sure I could keep boosting the chuck request threads or increase the max simultaneous explosions, but I'm not the one that coded the game. I can only keep testing, rebooting the server over and over (while players are trying to play). If I were to code a game like StarMade, I'd imagine that I should have a much clearer understanding of what the bottlenecks are and how to really use the existing settings to get the most out of a server computer. I'd also be aware of the caveats of changing certain settings (such as what happens if the CHUNK_REQUEST_THREAD_POOL_SIZE_TOTAL setting is set too high? Perhaps it won't cause any issues at all till too many people are mining at once. Or maybe physics calculations will slow to a crawl if this setting is too high. Or maybe it will increase everyone's ping time across the board at all times, but will make large-scale combat a lot more streamlined. Who knows? (I imagine the developers might)

Right now, server operators are just kind of reading these somewhat cryptic descriptions in the config files and willy nilly changing them. I think establishing a baseline for servers with different configurations and better documentation will have a positive, general impact on the community as a whole. I do not agree that it should be some test of skill (and patience), where only skilled server operators are able to have a functional server. Right now, the out of the box settings are most definitely NOT sufficient to run a server on and what a person ends up setting it to will be their best guesstimate, and not ideal.

 

[Erth Paradine]

I do, but it still makes the update process much slower. Also, I have noticed that many server operators are not as savvy as you and I might be with scripting. This is meant to solve a broad problem.

How many games have you played where the settings auto-adjust for quality based on the specs of the system it is run on? When done right, these games are enhancing the fluidity of gameplay for the end-user, right? Though a tool like this would be ideal, I think, for server operators, I don't think it'd be practical at this point, which is why I suggest just having a few recommended settings files for common server setups which have big differences. For example, I've noticed some servers running on home PC's that have "ok" CPU's, lots of RAM, regular hard drives (not SSD) and slower internet. Then there are the servers which run with 4 xeon cores (2.4ghz), 8GB of RAM, and SSD's. Then there are servers with 4 xeon cores, regular hard drives, and 64GB of RAM. Then there are servers with dual xeon CPU's (4 cores apiece, 16 threads), SSD's, and 64GB of RAM. These systems could all very well have different bottlenecks, and should be configured much differently. Having an example settings file would be a great place to start to get the most out of the system configuration being used.

In your current state: what settings are working for you?

As for the topic of SM's main server-side bottlenecks? AABB checks are #1 in my book. Next would be blocking threads related to sector loads, blueprint, and faction turns (trade threads are much better behaved now). Various logic & rail implementations can also be problematic, especially when applied to giantism designs. There's not much on the server-side that we can do about threading issues though, other than to cull any obvious abusers, and throw money (e.g. more & and faster CPU threads) at the problem.

With exception to memory leaks, I can't remember the last time our server's heap bumped higher than ~8GB of consumed RAM. With SSDs, even filesystem buffering doesn't typically eat more than ~4GB, add in non-heap memory consumption, networking overheads, and other system needs, and I'd put 16-24GB as a bare-minimum requirement for any popular server...although this can vary dependent upon a server's intended uses, its community, and the age of its DB.

As for adjusting SQL_NIO_FILE_SIZE, that alone will vary based upon DB sizes, and its effectiveness can be influenced by 32bit vs 64bit installs, as well as choices about the underlying OS, the supporting filesystem, the storage methods, etc. This value itself illustrates an excellent point though: its something that needs to be adjusted as a server's DB grows, making for no "ideal" value (aside from "stay ahead of .data"). It's also another reason why I feel that server operators need to have a basic understanding of what they're doing, instead of blindly applying someone else's specific recommendations. For instance, fiddling with NIO forces a fallback to OS & hardware buffering if set too low, or there's insufficient memory; the symptoms of which will often masquerade as other issues. Setting too high just wastes resources.

Regarding the server settings you mentioned, I appreciate it, and I'm sure other server operators could use the info, but I know about these already, I have adjusted them back and forth after a lot of testing for our server. We've come up with one that runs pretty darn well I think, but I know more could be done. Then there are other settings that are not documented anywhere, such as with the HSQL settings for the world file which still seems to be optimized for computers made in 1998.

Mmap I/O abstraction in Java's standard library APIs sucks: returning a MappedByteBuffer, which (aside from not being threadsafe) typically has an 2^31-1 element constraint...a very 90's-esque way to suck the fun out of what should be a brilliant I/O abstraction option.

Part of the reason I suggest that there be some suggested server setting config files, is because I had a heck of a time looking around for different configurations hosting options for LvD. The problem is that nobody really seems to know what the biggest bottlenecks are for StarMade, so trying to find a good balance, while maintaining a reasonable cost is difficult. Sure we could rent a top of the line server with a great dual CPU setup with 16 cores total and 32 threads, with 128gb of RAM, terrabytes of backup storage and oodles of high end SSD for the server files, but that would cost an immense amount of money each month. If I could have more clearly known where to put the most focus, then the selection process would've been much easier. By having a few config files tailored for specific setups, I could have used that as a baseline for choice, and then also as a baseline to start with before tweaking.

From what I see: primarily CPU-bound, I/O is secondary.

IMHO: if you're planning for a popular server, get as many cores as you can afford, throw at least 24GB of RAM at it, and tune the hardware (NICs, storage, etc) for the lowest possible CPU demands, giving attention to interrupt coalescence, NAPI, and all relevant hardware and protocol queue, caching, and buffer settings, reduce layer 3/4 timeouts, increase filesystem block sizes, make sure your storage is fully-aligned (pages, blocks, offsets, stripes, right down to sectors).

Now, currently one of the issues I've been having is that I'm trying to ensure the CPU actually remains busy when there is a lot going on with the server. Of course, it could be remaining low due to certain bottlenecks, but I've found my options are a bit limited in the settings files. Sure I could keep boosting the chuck request threads or increase the max simultaneous explosions, but I'm not the one that coded the game. I can only keep testing, rebooting the server over and over (while players are trying to play). If I were to code a game like StarMade, I'd imagine that I should have a much clearer understanding of what the bottlenecks are and how to really use the existing settings to get the most out of a server computer. I'd also be aware of the caveats of changing certain settings (such as what happens if the CHUNK_REQUEST_THREAD_POOL_SIZE_TOTAL setting is set too high? Perhaps it won't cause any issues at all till too many people are mining at once. Or maybe physics calculations will slow to a crawl if this setting is too high. Or maybe it will increase everyone's ping time across the board at all times, but will make large-scale combat a lot more streamlined. Who knows? (I imagine the developers might)

Right now, server operators are just kind of reading these somewhat cryptic descriptions in the config files and willy nilly changing them. I think establishing a baseline for servers with different configurations and better documentation will have a positive, general impact on the community as a whole. I do not agree that it should be some test of skill (and patience), where only skilled server operators are able to have a functional server. Right now, the out of the box settings are most definitely NOT sufficient to run a server on and what a person ends up setting it to will be their best guesstimate, and not ideal.

Busy CPU is great...sometimes...which threads are getting the most time?