How to avoid server lag?

[Panpiper]

Last night my server had a serious issue of some sort with massive server lag that went on for hours. Exactly what caused the issue is still not clear. This morning, a few grumbles were heard insinuating that my drone fleet of 24 drones and some other very peaceful person with a similar fleet, was the issue.

I take umbrage with this, as that very drone fleet has been present for weeks without presenting any issue to the server. It is still present this morning with no server lag. Moreover, there is one griefer who has at least one much larger drone fleet of 70 drones that are each over 20K mass. Mine are just over 5K. Then there is the griefer who flies around a titan measured in kilometers. How is it that "I" am the culprit while these others get to bully everyone without challenge?

Anyway, enough of my wining.

Do fleets of drones doing nothing, just sitting in space, cause server lag? Does a titan, doing nothing, cause server lag? If a drone fleet is moving, say been given a move order to go to a new sector, does that cause server lag? If a titan starts to move, does that? I am pretty sure that if a titan and a drone fleet start to fight, that will probably be the end of the server, but in the mean time...

What causes server lag? What can be done to mitigate it? I absolutely refuse to disassemble my drone fleet (and fully intend to increase it) while griefers are left to bully and attack people using their own fleets and titans. Will my drone fleet be less of an issue if it is docked?

(I strongly suspect that undocking and redocking a drone fleet would cause FAR more server lag than letting it just sit there with fleet orders.)

 

[Edymnion]

No, totally still objects cause no server lag. That quietly docked titan doesn't hurt the server in any way... until it undocks.

The biggest sources of lag that I've seen are:

1) Collision Checks
Any time the server has to check to see if two things hit each other, especially two really big things, it grinds to a halt as it starts having to check millions of blocks against millions of other blocks. This is why docking and undocking is so hard on the server, once you start getting inside of bounding boxes, constant collision checks occur until its done.

2) Mass Block Changes
Any time more blocks change status than the code can handle, such as massive salvage arrays or huge missile explosions that vaporize extensive tracts of land.

Turret lag appears to be mostly #1, as is blowing up a planet (because its having to calculate the planetary plates and their close initial proximity to each other for collision purposes).

 

[Batavium]

How to avoid server lag?

A StarMade server lags because it's incapable of transferring of all environmental changes by every player online to every player online in real time.

A StarMade map is unlike any map in a multiplayer game because nothing is static. The whole map is made up of chunks. Every player online loads up a minimal of 9 sectors full of chunks just by being online. Every player exists in a 3 by 3 by 3 sector bubble. Each chunk inside the sector that has even one block in it must be loaded. Change a block and the chunk has to be saved/loaded again. The server cpu is working overtime to communicate all these chunks to the players.

The network is stressed by every entity in a sector. Entities are players, ai, missiles, cores, asteroids and planets. They all move at random through the sectors. All of that needs path finding calculation. It also needs to be communicated to all other players online in real time. Some entities have movement of there own and need extra calculation like planets or when you push an asteroid. Even a static object can induce lag when it's big. Because it is made up of so much chunks that it takes to long to feed them all to you in real time.

Logic also creates a lot of lag. Because it can make things move on there own. Even a big factory with many connections can be a major drain on a servers cpu. One major concern is that logic often runs all the time in a loop without ever being shut down. Resulting in a never ending cpu calculation for it whenever it's sector get loaded. Think weapon clocks, rotation, jump drives that use logic.

Add multiple players to a sector and the resulting path finding calculations of all the ships, missiles and just all the stuff there go through the roof.

Sometimes what seems like simple things can completely overpower the system. Try doing a multi mirror copy with a 50 by 50 by 50 buildbox. Computers can have to many systems attached to them. Like salvage beams or missile tubes.


So what to do?


Use the minimal configuration that is possible. Use the smallest sectors. The smallest planets. Disable all movement like asteroids. Limit the amount of systems you can add to a computer. Limit the maximum amount of players online. Very few server can handle over 30 players it just scales badly. 29x29 connections or 30x30 connections is not the same. Every added player adds more drain on the system then the one before him.

In my opinion most servers out there have a configuration that is to much to handle for the server cpu and network upload that is given to them. StarMade in it’s current state is a game that can not be brute forced by adding ever more cpu power. The network part of it the upload to all players won’t keep up even on datacentre capable lines.

Find out what your server cpu is capable of and use a configuration that matches that! Then you should have a StarMade game that plays well in most cases.

 

[Gasboy]

There could be many different reasons for server lag, and many of them are not the game itself. Edymnion has outlined the biggest sources of lag in StarMade, and as you have surmised, docking and undocking is a major source of lag.

But there are external forces that can cause lag, and if you do not physically own and operate the server yourself, you never really know what else the server could be doing.

Your drones, by themselves, shouldn't cause a modest server any trouble. IF everyone on the server was using that many, well, that's a possibility. If the griefer has a large ship, and spends his time launching and recovering all those drones, there's a good possibility that they are the cause.

Did you get any interesting server messages while the lag hit? Sometimes you'll see errors or issues coming from specific sectors, and if you go there, you can probably determine the cause.

 

[aRottenKomquat]

The #1 cause of server lag on my server was apparently asteroid physics. Even if nobody was doing anything near asteroids, they would cause the server to go completely unresponsive until it was reset. I had to edit the server config to disable asteroid physics and since then I've only had minor lag issues.

 

[Panpiper]

Well, I had an interesting and perhaps educational moment. I mentioned to my server admin that I wanted to boost my drone fleet to 80 vessels (still just over 5K each). He gave me his blessings and said if nothing else, it would make for interesting data for the developers. This was however, none the less during a time when the server was experiencing a lot of constant lag. Up until that moment, I had had a fleet of 24 such drones.

I built up the numbers the next day, setting up the drones in a few lines just north of my base. The server lag problems we were having continued, despite restarts and other efforts. They did not appear to get worse however as my fleet grew. At an inopportune moment for an experiment however, a mere few minutes before the server was to shut down for another restart that would disable MOB spawning, I ordered my fleet of 80 drones to move to a sector three distant from my home base. That was far enough for them to appear in my fleet screen as 'unloaded'.

When the server rebooted, the lag issues appeared resolved. The FPS at my station was back to it's normal 100+ with the absence of my fleet. With the admin on line, for a test I asked him to watch as I jumped into the sector containing my drones. Pretty much the instant I jumped, the server lag spiked massively, ping hitting 4000+ and stayed there for the twelve seconds it took for me to cycle my jump drive. I did not give myself time to look around, the server had ground to a halt, and something was firing at me?! I jumped out and the lag disappeared.

I have no clue what happened or why I was being fired upon. The sector was sufficiently distant from any pirate base. I did not stick around to look as that drone fleet would have obliterated me in seconds had I tried (if that was what was firing on me). I can say without a doubt, that the presence of a fleet of 80 drones was enough to bring a robust server to it's knees.

(Those drones were nothing particularly weird. They are basically flying guns. They do have 16 PD turrets each but no larger offensive turrets. They each have a two block logic clock forcing a sensor to check shield status so as to activate jump inhibitors if it detects a 2% drop.)

 

[Malacodor]

I can say without a doubt, that the presence of a fleet of 80 drones was enough to bring a robust server to it's knees.

They do have 16 PD turrets each but no larger offensive turrets.

80 drones with 16 turrets each equals 2640 entities. That's enough for a 100+ ping (1000-2000 shortly after loading the sector) in single player, according to my test with 8 test ships with 300 docked entities. That's 2408 entites with a combined mass of only 600. FPS is ~12 compared to 180-190 when looking at a planet.

 

[Groovrider]

So thats like 2560 docked entities? Wow.

 

[Panpiper]

So, total docked entities is the problem? Do you think if there were substantially less docked entities, that there would be substantially less lag? How much less lag? How few PD need I put on them in order to ensure that the server can survive such a mass?

I cannot imagine myself giving them any less than two, but at that level they will be sitting ducks for swarmers. Or perhaps while that would be the case in fewer numbers, maybe if fielded as a fleet of 80, swarmers would have so many targets to choose from, they would disperse sufficiently for a much smaller number of PD to adequately deal with them.

 

[Malacodor]

I cannot imagine myself giving them any less than two, but at that level they will be sitting ducks for swarmers.

The ships can protect each other, 80 * 2 is still 160.

 

[Scypio]

What's your drones' size if they got 16 PD turrets ? 100m long drones ? Like Malacodor said drones will cover themselves between them. Your fleet of 80 drones got 1280 PD turrets... I think that's more than enough, even against swarm. When you build a fleet think about the mass and not for one ship alone, for example you could get specialized drones with plenty of PD turrets and some with just 1 or 2.

Also, swarmer tend to go for the same entity.

The ships can protect each other, 80 * 2 is still 160.

80x4, you need the turret base and barrel. So with 2 PD turrets it's 4 entities. Do not forget the ship entity too. :p
So with 2 PD turrets on each drones it's 80x5=400 entities. Be careful that goes up easily. Also your drones could have been gone crazy and with 80 drones that bump between themselves...

 

[Malacodor]

80x4, you need the turret base and barrel. So with 2 PD turrets it's 4 entities. Do not forget the ship entity too. :p
So with 2 PD turrets on each drones it's 80x5=400 entities. Be careful that goes up easily. Also your drones could have been gone crazy and with 80 drones that bump between themselves...

I was referring to the defense capability of 160 PD turrets.

 

[Panpiper]

Is there an admin command to despawn a sector? Some way to with one command rid the server of this pestilence? I am hoping that my being chummy with the admin and proactive with dealing with this, that he'll be willing to just delete this bunch and then 'maybe' (pretty please) spawn in 80 replacements for me. Individually each of those drones aren't a lot, but together, that's nearly enough materials for a titan!

 

[BDLS]

Is there an admin command to despawn a sector? Some way to with one command rid the server of this pestilence? I am hoping that my being chummy with the admin and proactive with dealing with this, that he'll be willing to just delete this bunch and then 'maybe' (pretty please) spawn in 80 replacements for me. Individually each of those drones aren't a lot, but together, that's nearly enough materials for a titan!

Admin Commands

 

[Panpiper]

Does anyone have any advise on how to create a minimum server load method of AI controlled ships activating a jump inhibitor when in combat? Right now I have each of these drones with a 75K power inhibition system that is activated if the shields go below 98%. This is 100 clocks constantly setting off sensors. I suppose I could have a handful of dedicated ships instead whose only real job is to jump inhibit and remove that from the rest, but then I am really banking on some of that handful getting shot at. Is there a better way?

 

[Malacodor]

Does anyone have any advise on how to create a minimum server load method of AI controlled ships activating a jump inhibitor when in combat? Right now I have each of these drones with a 75K power inhibition system that is activated if the shields go below 98%. This is 100 clocks constantly setting off sensors. I suppose I could have a handful of dedicated ships instead whose only real job is to jump inhibit and remove that from the rest, but then I am really banking on some of that handful getting shot at. Is there a better way?

You have a system to detect shields in steps of 1%? Throw out half of it and you can still detect a 2% drop. Better throw out most of it and set the detection threshold to 90%. Depending on your maximum tolerable response time you could use a circuit, that can sequentially check for varying thresholds.

 

[Crashmaster]

...100 clocks...

D8

You could have semi-dedicated inhibit ships that activate the inhibitor when their power goes down when they fire their weapons instead of being hit. A significant delay on the off signal would keep them on as power fluctuates during battle.

You could also divide the drone fleet into squadrons with a semi-dedicated inhibit ship also wirelessly activating the inhibit systems of all of your current drones in it's squadron.

 

[Malacodor]

This is 100 clocks constantly setting off sensors.

Did you mean 100 clocks per ship or per fleet?

 

[Panpiper]

Did you mean 100 clocks per ship or per fleet?

1 clock per ship, though each ship 'is' using 50 activation modules to detect the shields at the 98% point. They are relatively strong shields for the ship size, and I don't expect many of them to drop particularly fast, so I wanted them to be sensitive.

I do think though that detecting a power drop from say a large cannon/beam weapon on a dedicated inhibitor ship should do the trick. I was planning on putting some sort of ion weapon on them anyway. Fleet support and all that. I just have to figure out that "significant delay on the off signal" thing, as logic circuits are hard for my aging brain.

 

[Malacodor]

1 clock per ship, though each ship 'is' using 50 activation modules to detect the shields at the 98% point. They are relatively strong shields for the ship size, and I don't expect many of them to drop particularly fast, so I wanted them to be sensitive.

Then I completely misinterpreted your post. But if a 99.9% detector would also work, you could use a single active activator with a NOT gate.

I do think though that detecting a power drop from say a large cannon/beam weapon on a dedicated inhibitor ship should do the trick. I was planning on putting some sort of ion weapon on them anyway. Fleet support and all that. I just have to figure out that "significant delay on the off signal" thing, as logic circuits are hard for my aging brain.

Make a row of delays, each connected to the next one and each one connected to an OR gate. With 10 delays the OR gate will stay on when the signal going into the first delay is off for less than 5 seconds.