Tips N Tricks for MP-Servers

[Tsnonak]

This is a broadcast message from your local server, please do not adjust the screen...

Last Update: March 19, 2021
Game version: 0.202.87

This thread is an attemp to raise the quality of life for server-owners, administrators, and players!
StarMade is in Alpha, not optimised and buggy, there some things you should (maybe) know.

Player count plays a big role in how much is going on at the same time all over the universe.
All sectors around the player-sector (27 total), are loaded /unloaded as players gallivant through the universe.

Here are some tips for running a smooth server, Starting with the server.cfg.
These are some of the settings we use on the LK-server...

CONCURRENT_SIMULATION = 8 // Reduce the default number.
BLUEPRINT_DEFAULT_PRIVATE = true // This should always be true on public servers.
DEFAULT_BLUEPRINT_ENEMY_USE = false // This should always be false on public servers.


Some tips for keeping npc interaction interesting instead of painfull...
AI_ENGAGEMENT_RANGE_OF_MIN_WEAPON_RANGE = 0.5 //brings AI closer to players, helps with grueling chases

NPC_LOADED_SHIP_MAX_SPEED_MULT = 0.5 // ships that are too fast(custom npcs) are grueling to catch, slow them down some


AI_WEAPON_AIMING_ACCURACY = 200 // This is a little low and allows for AI errors (Official server uses:1000, 2000 has been common on servers in the past)
current v0.202.86 release seems intent on setting this back to default 200 for singleplayer, seems ok for servers*

Asteroid Duplicate Bug:
last tests duplicate asteroids are still prevalent in v0.202.87
They often enter endless loops of crashing, warp away, return and crash again, or occupy the same coordinates, jitter verry strangely in an endless collision-cycle.
ASTEROID_SECTOR_REPLENISH_TIME_SEC = -1 // -1 = OFF

Do not make asteroids too big, but if you really want them I recommend not bigger than MAX = 128
ASTEROID_RADIUS_MAX = 128

crashing asteroids can be problematic @ default settings...
If you leave asteroid-respawn on: do not increase from default settings, (smaller is ok), but it may still cause problems on busy servers!

PLANETS:
Keep Planets small, under no circumstance make them bigger!
(Planets)
PLANET_SIZE_MEAN_VALUE = 50.0
PLANET_SIZE_DEVIATION_VALUE = 5.0

PLAYER_MAX_BUILD_AREA = 30 // Brush sizes that are too big can cause problems depending on your rig and player count
Players setting 50x50x50x8 blocks at a time is going to cause problems!


CHUNK_REQUEST_THREAD_POOL_SIZE_CPU = 3 // Adjust as needed, Rule of Thumb is number of proccessors - 1

ONLY_ALLOW_FACTION_SHIPS_ADDED_TO_FLEET = true // This should always be true on public servers

Prevent Logic-bombs: we have had no problems with players needing more than this in almost two years of service.
MAX_LOGIC_SIGNAL_QUEUE_PER_OBJECT =10000
MAX_LOGIC_ACTIVATIONS_AT_ONCE_PER_OBJECT_WARN = 8000
MAX_LOGIC_ACTIVATIONS_AT_ONCE_PER_OBJECT_STOP = 10000


To Turn NPC factions off:
NPC_FACTION_SPAWN_LIMIT = 0
(but, if you really want NPCs then zhey should really be slown down a bit, see below*)


The GameConfig.xml: set max-limits and adjust start-inventory.
The following has been copied from another post...

"
I have played on servers that had no lag reduction policies, it didn't end well.
I have played on a server with an enforced turret count limit and mass limit, it was the most stable I ever played on.

Fact is, StarMade doesn't have a hard cap for ship size and docked entity count, but the game's engine and server/player hardware definitely has a softcap.
"

PATH: \Starmade\GameConfig.xml


Max ship / station size / mass-limits:

<MaxDimensionShip>
<X>600</X>
<Y>600</Y>
<Z>600</Z>
</MaxDimensionShip>

<MaxDimensionStation>
<X>2000</X>
<Y>2000</Y>
<Z>2000</Z>
</MaxDimensionStation>

<ShipLimits>
<Mass>250000.0</Mass>
<Blocks>2000000</Blocks>
</ShipLimits>

<StationLimits>
<Mass>2000000.0</Mass>
<Blocks>5000000</Blocks>
</StationLimits>

Max weapon / support systems can also be set: ID_list

<GroupLimits>
<Controller>
<ID>4</ID> // ID of the computer
<GroupMax>125</GroupMax> // MAX allowed groups
</Controller>
<Controller>
<ID>4</ID>
<ComputerMax>2</ComputerMax> // MAX allowed computers
</Controller>
</GroupLimits>

What does Schine say about Salvagers:


Add a blueprint to start inventory:
<Blueprint>
<Name>STARTER_SALVAGE_CRAFT</Name>
<Filled>true</Filled>
<Slot>inventory</Slot>
</Blueprint>




NPC Faction Problem:
"they are the most destructive feature of this game in terms of server stability"...

They accumulate in masses when left unchecked.
They leave docked parts all over the universe.
They grow too fast and crash too much.

regular cleansing of the database via admin-commands / scripts is recommended, best time is right before or after restart.
Note, if this has not been done and the world is older or already full of NPCs this can cause a server crash and possibly death!

/despawn_all MOB_ all true
/despawn_all FLTSHP_ all true
/despawn_all GFLTSHP_ all true
/despawn_all GNPCFLT_ all true
/despawn_all [System_ all true

I have even gone as far as checking the server-database:
\StarMade\server-database\YOUR-WORLD
... and manually deleting any "unwanted" NPC-entities...
(No Server-problems could be observed*).


You may / or may not want to adjust NPC-Faction-growth in the appropriate npc-faction-folders...

Initial territory is set with the npcSpawnConfig.xml...
PATH: StarMade\data\npcFactions\npcSpawnConfig.xml

<InitialGrowth>-1<!--How many systems are initially taken--></InitialGrowth>
-1 is random generation, remove the (-) minus to give the faction 1 InitialGrowth-system, (they will start with 2 systems*).

Reduce Faction-turns in the npcConfig.xml...
PATH: StarMade\server-database\YOUR_WORLD\npcFactions\faction\npcCofig.xml

(line 287*)
<TimeBetweenTurnsMS>9000000<!--Time between faction turns in milliseconds--></TimeBetweenTurnsMS>
We've just added another 0 to make it 9 million instead of 900,000 milliseconds.


NPCs / Pirate:

When replacing NPC assets always remember to have them in-place before running the first time!

Currently Pirate stations will bug-out if they are updated: updated-pirate-stations

Regular NPCs are not affected by this, (to my knowledge*).
Updated NPC Assets



Hardware:
The most important thing to consider when selecting hardware: above RAM or SSDs, is CPU-clockspeed.

Golden-Rule:
"Don't just throw more RAM at the issue, or a faster SSD or anything like that unless you're sure those are the root cause of your performance issues." therimmer96™

ISP: Good Bandwidth is also important.

Server-[RAM]-Settings according to Schine™:
4-8 G ist the sweet-spot for Java (depending on the system, (more than 8G can have negative effects & always in multiples of 2*)).
Only add more if you have ca. 10 players constant w/ restarts 8 - 12 hours.
Regular-Restarts are very important for performance...
Recently I also noticed loading Blueprint designs in Shipyards lag more the longer the server hass been running.

suggested-server-settings: java -server -Xms1G -Xmx8G -jar StarMade.jar -server
Keep start-RAM [Xms] low.


The following has been copied from another post...
I do not have the resources or time to validate everything stated below, but can confirm that it still applies to an extent today.

"The simplified version of the multitude of problems of running a server at that time"... ca. 2015

1. It took an absurd amount of bandwidth spiking in excess of 500mb/s for the roughly 50-60 people online.
2. It took an absurd amount of disk space. The server would grow by as much as 5GB per day, if left unattended.
3. The server was in a constant state of very low performance even when run on top of the line hardware.
4. The server would implode if no admin was on to watch it 24/7 due to people doing seemingly simple mistakes like getting too many ships near each other, or having too much stuff near a planet.
5. The catalog getting corrupted was nearly a daily occurrence.
6. Sectors would randomly corrupt themselves.
7. Planets, asteroids and NPC stations / shops would disappear or randomly move (ghost stations are still a thing*)
8. Unrecoverable crashes at least 1 time a month resulting in rollbacks
9. Admin commands were basically useless since they either didn't work, or reported invalid information, or sometimes crashed the server.
10. The sheer volume of log output could cripple the server if not running on SSDs.

Gigantism:
Is still a big problem for MP-servers...
Giant bases and ships with high block / entity count cause server / client lag.

stolen from: how-to-avoid-server-lag

The biggest sources of lag that I've seen are:

1) Collision Checks
Any time the server has to check to see if two things hit each other, especially two really big things, it grinds to a halt as it starts having to check millions of blocks against millions of other blocks. This is why docking and undocking is so hard on the server, once you start getting inside of bounding boxes, constant collision checks occur until its done.

2) Mass Block Changes
Any time more blocks change status than the code can handle, such as massive salvage arrays or huge missile explosions that vaporize extensive tracts of land.

Turret lag appears to be mostly #1, as is blowing up a planet (because its having to calculate the planetary plates and their close initial proximity to each other for collision purposes).

So what to do?

Use the minimal configuration that is possible. Use the smallest sectors. The smallest planets. Disable all movement like asteroids. Limit the amount of systems you can add to a computer. Limit the maximum amount of players online. Very few server can handle over 30 players it just scales badly. 29x29 connections or 30x30 connections is not the same. Every added player adds more drain on the system then the one before him.

... more good stuff regarding fleets / drones in the post as well.
Changes to reactors / chambers in the tens to hundred thousands cause server spikes in the thousands and may crash the server!
(I've done it!)!
Changes to giant ships / stations / and too many entities cause server lag, giant ships flying through space cause lag...
it is best to avoid gigantism, and inform players of the problems it causes.


Generally in all software: failed saves result in file corruption!


Full list of Admin Commands

Feel free to post anything affecting server-stability that you have encountered below , thanks for your participation!


Brierie

WIP*

See you in Space!

 

[therimmer96]

At the end of the day, the thing that has the most impact on performance above any of the things listed here is to have appropriate hardware.

Starmade, and most game servers for that matter, is largely single threaded. If you're shopping for hardware directly, find the CPU with the best single thread performance you can afford and use that. If you're using a Game Server Provider <Cough>PingPerfect.com*</Cough> and you're having performance issues, contact their staff and find out if you can upgrade to a higher clock speed system/tier. Especially if your performance issues occur during moments of combat, collisions and the likes.

Don't just throw more RAM at the issue, or a faster SSD or anything like that unless you're sure those are the root cause of your performance issues.

 

[Tsnonak]

At the end of the day, the thing that has the most impact on performance above any of the things listed here is to have appropriate hardware.

Starmade, and most game servers for that matter, is largely single threaded. If you're shopping for hardware directly, find the CPU with the best single thread performance you can afford and use that. If you're using a Game Server Provider <Cough>PingPerfect.com*</Cough> and you're having performance issues, contact their staff and find out if you can upgrade to a higher clock speed system/tier. Especially if your performance issues occur during moments of combat, collisions and the likes.

Don't just throw more RAM at the issue, or a faster SSD or anything like that unless you're sure those are the root cause of your performance issues.

thanks, and quite correct, hardware stuff also plays a big role!

At the end of the day, StarMade is known to bring top-line hardware to its knees...

 

[Tsnonak]

Update March 15, 2021...
Due to my recent experiences on MP-Servers...
I felt an update was needed, same old xxxx problem with servers...
and I would maybe like try to help and bring an end to the difference of opinions regarding Asteroid-Settings.

cannot log in, Ping 40,000+
... because of asteroid duplicates.

1. If you feel the need to enable asteroid respawn:
ASTEROID_SECTOR_REPLENISH_TIME_SEC = 300

Then:
You need to keep them small (vanilla),or maybe even reduce size (do not make them bigger! )!
You need to enable physiks:
ASTEROIDS_ENABLE_DYNAMIC_PHYSICS = true

This allows collision avoidance, allowing them to warp around and (hopefully) settle in a good spot.

2. If you do this:
ASTEROIDS_ENABLE_DYNAMIC_PHYSICS =false

you are disabling (as far as I can tell) collision avoidance resulting in multiple asteroids occupying the same coordinates!
(they are now "Stuck In Space"™, you may have seen them... they jitter and act wonky).

... cont. from above: cannot log in... finally logged-in (after a few tries and several minutes of Ping= xx,xxx).
3 - 5.9k mass Asteroids occupying the same coordinates... they need to die a horrible death ASAP!
Attempt to mine/salvage them:
My salvager is small, <500 groups... ping goes through the roof: 4,000 -5,000+!
mining them is a server - wide problem for everyone!

How do I know it was the cause?... the server Admin was online!
Server was fine, until I tried to login... It was a sector near me!
... problem solved this.time, but more spawn whenever a player flies around... it will not end well for the server.
... usually, on a "nearby" server:
Idle ping = 25 -35, when I mine (457 groups) the ping spikes and doubles = 65 - 75
*** I can also see this leading to: Server-file-faction-corruption ***...
A server that is "stuck" in endless collision-calculations cannot save proper...
failed saves lead to file corruption.
Brierie

which brings me back to the next problem: Gigantismus...
I can only imagine what a "Miner/Salvager" with 4,500+ groups causes... (I do not know the hardware/settings of all servers*).
Personally: I do not use such beasts (tested them on my old server, was not impressed*).
... and to be honest, my miner/salvager with 457 groups does fine, it gobbles asteroids in no time, it is enough.
Servers need to set limits in the gameconfig.xml!
Taken from the original Post above:

I understand/assume... in reality this refers to load time for all groups in the weapons-menu-window...
but what does it mean for AlphaMade in general...
It is like a weapon: with >500 Outputs !
...and we know how that ends: not performance friendly.
Although salvaging has seen some awesome improvements... I have no interest debating over salvagers with 4,500+ groups, it CANNOT be good for server performance, I have experienced it online, alone, public server.
(20 was the server-limit "back in the day")

Next Topic: Asteroid-duplicate-bug...
Alendon wrote a fix for his wrapper: https://starmadedock.net/content/duplicate-asteroid-fix-plugin.7757/
Also...
Once upon a time... I had contact to Schema (discord)for a short period...
this was the result: (although slightly misleading, should have read "not touched").

sadly, it returned with the next update, but what does that mean... to me as a programming layman...
it is a bug which can be easily fixed (it took maybe an hour chatting+pics+explanation)...
[email protected]

p.s. this is a PvE /PvP independant bug... you need ores for wars as well!

Server-Tip:
disable respawn
increase size
increase ore vien
increase probability
to help mitigate this bug... and mitigate "Megalithic-Mass-Salvagers"™
(once respawn has been enabled on MP-servers it is only a matter of time until it brings the server down, turning it off and keep playing is not an option, it will eventually force a reset).

 

[MrGrey1]

Good info, one thing I'd add to keep planets as small as possible;

PLANET_SIZE_MEAN_VALUE = 50.0
PLANET_SIZE_DEVIATION_VALUE = 0.0

 

[Tsnonak]

Good info, one thing I'd add to keep planets as small as possible;

PLANET_SIZE_MEAN_VALUE = 50.0
PLANET_SIZE_DEVIATION_VALUE = 0.0

considering the fact: Planets have been known to cause serious problems in the past...
I will generally agree... updated thread to include it!

Flip side is that planet-interaction has also improved...
main problems are still ship collisions and/or small-ships/cores clipping into the planet-core and getting "Stuck In Space"™... and they reset sometimes, (don't build on them unless you want to risk loosing it*).
...personally I also get the heebeegeebees setting values to 0.0... (once upon a time a small bird told me to not multiply with 0, strange things happen... call it superstition ).

Moreso, I would advise, under no circumstances: do not make them bigger!
(it has been done, seen it, could not understand it, Admin before me set PLANET_SIZE_MEAN_VALUE = 300.0*).

If I recall correctly for the server I used:
PLANET_SIZE_MEAN_VALUE = 75.0
PLANET_SIZE_DEVIATION_VALUE = 10.0

... and never had any problems with planets on the server... even though some players dismatled /salvaged them and built new stations "around" them.
Anyone building on planets was left alone, completely, to fend for themselves...(generally it's just too much controlling for Admins/Mods).
If it causes problems: It would be deleted and you loose everything there.
If it resets: you loose everything there.

the general "Rule" was a good deterrent itself, but it did not stop them all.
I even did it myself for test purposes.