What's with the insane lag during autosaves?

[Lecic]

If you've played multiplayer, you've probably noticed this- when the game autosaves, servers freeze up. I've seen servers just completely stop for over 30 seconds during autosaves, which is... fucking ridiculous. Something needs to be done about this.

 

[StormWing0]

Never had the issue but then again it might be because in some cases the game is trying to save a huge amount of things. If the server has enough area that people have expanded into the saving of it might be locking things up due to the huge amount to save.

 

[Keptick]

Because it was "optimized" and save times went up like crazy on servers with big databases. I guess the optimization slightly optimized saving small databases but messed up big servers.

 

[TheBlueThunder]

If you've played multiplayer, you've probably noticed this- when the game autosaves, servers freeze up. I've seen servers just completely stop for over 30 seconds during autosaves, which is... fucking ridiculous. Something needs to be done about this.

Maybe because Java and threading happens Java telling the server multiple tasks at the same time which in turns makes client lag because client is also sending data while server is writing data. But for top dollar high end computers with more cores in processor and gpu threading and sending data while writing data is no problem with Java or SM. In turn if the development of a game is using high ended parts its devs won't know about lag that's where they have tester using different hardware.

Does this answer your question?

 

[Lecic]

Maybe because Java and threading happens Java telling the server multiple tasks at the same time which in turns makes client lag because client is also sending data while server is writing data. But for top dollar high end computers with more cores in processor and gpu threading and sending data while writing data is no problem with Java or SM. In turn if the development of a game is using high ended parts its devs won't know about lag that's where they have tester using different hardware.

Does this answer your question?

I'm pretty sure the problem with the devs not catching this is more to do with the devs using small servers while testing updates than devs having high end parts to test the updates on. It doesn't happen on small servers with a small number of players and small-ish ships. It happens on big servers with lots of players building huge ships and stations.

 

[TheBlueThunder]

I'm pretty sure the problem with the devs not catching this is more to do with the devs using small servers while testing updates than devs having high end parts to test the updates on. It doesn't happen on small servers with a small number of players and small-ish ships. It happens on big servers with lots of players building huge ships and stations.

Doesn't happen on my end when I play on a big server with lots of players with big ships. Everyone on TS would just tell me to stop moving my ship when its auto-saving. So I guess they might have fix for it on the next patch.

 

[Lecic]

Doesn't happen on my end when I play on a big server with lots of players with big ships. Everyone on TS would just tell me to stop moving my ship when its auto-saving. So I guess they might have fix for it on the next patch.

I have extreme doubts that the lag that appears during autosaves is client-sided, if that's what you're implying here.

 

[TheBlueThunder]

I have extreme doubts that the lag that appears during autosaves is client-sided, if that's what you're implying here.

It should be on both sides. But I don't notice the lag because I play different games on different screens while playing SM. I'm not implying anything.

 

[AndyP]

Most of the increases in save time are a tribute to the sector size.

When sectors were 2,5km boxes, there was a room of X objects to save, with doubling the size, we have round about 4 times the objects in the sectors, and the average station and ship size increased.
Some servers even increased their sector size far beyond that.

So, if we had a former sector size of 2.5km (1.25km from center to each sector marker), you had a loaded area of round about
7.5km x 7.5km x 7.5km (422km³)

With a sector size of 10km for example, you end up with
30km x 30km x 30km (27000km³)

And players usually worship that free space and never build anything in it that would not fit in those smaller sectors.
:P

No, really,
the longer autosaves are usually something on servers with large sectors.
Or limitations in disk-IO by writing large portions to disk, as the optimization that was blamed here, even skips writing anything to disk that was not changed, so if you have a server with those long autosaves, send me a copy of one log that covers an autosave (serverside!) and the servers settings.cfg, will look into them, and check if there is something unusual happening there.

- Andy

 

[Matt_Bradock]

As a legacy player who started the game when the basic Isanth-VI was considered a threat, I can only agree with AndyP that sector sizes and average ship size (not to mention station size!) have only increased with time. Add the new planets to that, the rails with new collision detection, the added blocks, and it becomes quite obvious that the amount of data needed to be written and overwritten every single autosave has become exponentially larger. Especially if we add into this mix, the influx of new players, and the rapidly improving building skills of people. Just remember what was considered a complex and big build 2 years ago, one year ago, and now. The game is getting more complex, thus the save files and folders get bigger, no optimization will change that, unless Schema by some wonder implements realtime fractal compression into Java (which would probably be worth a couple million bucks as a patent, heh.)

And while I understand why many people would make space more spacious, as bigger sectors have their own charms in gameplay too (for example, battles across sectors can cause some real issues regarding server load, while bigger sectors give a better chance that the battle will stay in one sector) they do come with things like this. I remember the old times, where the autosave message even contained a warning along the lines of "expect a lag spike". So, yea. With so larger sectors loaded, and due to so many players on, so many change before every single autosave, this can't really be helped unless you impose and enforce strict limits to sector, ship and station sizes on the server.

 

[TheBlueThunder]

Most of the increases in save time are a tribute to the sector size.

When sectors were 2,5km boxes, there was a room of X objects to save, with doubling the size, we have round about 4 times the objects in the sectors, and the average station and ship size increased.
Some servers even increased their sector size far beyond that.

So, if we had a former sector size of 2.5km (1.25km from center to each sector marker), you had a loaded area of round about
7.5km x 7.5km x 7.5km (422km³)

With a sector size of 10km for example, you end up with
30km x 30km x 30km (27000km³)

And players usually worship that free space and never build anything in it that would not fit in those smaller sectors.
:p

No, really,
the longer autosaves are usually something on servers with large sectors.
Or limitations in disk-IO by writing large portions to disk, as the optimization that was blamed here, even skips writing anything to disk that was not changed, so if you have a server with those long autosaves, send me a copy of one log that covers an autosave (serverside!) and the servers settings.cfg, will look into them, and check if there is something unusual happening there.

- Andy

So if the sectors were the standard normal size we wouldn't get lag from autosave.
Friendly question:
Disk-IO would it matter if the server would use SSD instead of HDD, would the write and read be a lot faster to save large sectors?

 

[Tunk]

Silly question,

Are all sectors saved together, or is there a save queue or scheduler?

Might be worth while to add a save queue or scheduler so the server can work through saving rather than batch processing the whole thing at once.
Sector level locking or copy for threaded save would also come in useful here.

 

[Matt_Bradock]

So if the sectors were the standard normal size we wouldn't get lag from autosave.
Friendly question:
Disk-IO would it matter if the server would use SSD instead of HDD, would the write and read be a lot faster to save large sectors?

Isn't the main thing with SSDs that they can be read lightning-fast but writing data onto them is actually slower than a conventional SATA2 hard drive? That'd mean it would not help a bit to save onto an SSD.

 

[TheBlueThunder]

Isn't the main thing with SSDs that they can be read lightning-fast but writing data onto them is actually slower than a conventional SATA2 hard drive? That'd mean it would not help a bit to save onto an SSD.

SATA II and SATA III is the connection for motherboard and storage device. SSD reads and writes faster than HDDs.
If you open up your desktop computer you'll see red standard cables sometimes black colored that are connected to your HDD or SSD and to your mobo and not your power supply. Its part of Sata which others would call Sata Cables or Ata Cables.

Edit: for more information about SSD and HDD here's a link http://asia.pcmag.com/storage-devices-reviews/1526/feature/ssd-vs-hdd-whats-the-difference

If your going to say that the storage capacity isn't good with SSD, they already made a 16TB SSD by Samsung, the only down side is the price.

 

[Megacrafter127]

SATA II and SATA III is the connection for motherboard and storage device. SSD reads and writes faster than HDDs.
If you open up your desktop computer you'll see red standard cables sometimes black colored that are connected to your HDD or SSD and to your mobo and not your power supply. Its part of Sata which others would call Sata Cables or Ata Cables.

Edit: for more information about SSD and HDD here's a link http://asia.pcmag.com/storage-devices-reviews/1526/feature/ssd-vs-hdd-whats-the-difference

If your going to say that the storage capacity isn't good with SSD, they already made a 16TB SSD by Samsung, the only down side is the price.

While true, depending on the size of the database relative to the total SSD-size, memory wear may become a serious issue, depending on how often the server autosaves.

 

[TheBlueThunder]

While true, depending on the size of the database relative to the total SSD-size, memory wear may become a serious issue, depending on how often the server autosaves.

Indeed, nothing stays forever hardware will break that's why we buy new ones. But due to the amount of time the game will autosave maybe it will break hardware much faster.

Edit: I'm not saying it will just saying maybe. If it does break buy a new HDD or SSD.

 

[Megacrafter127]

Indeed, nothing stays forever hardware will break that's why we buy new ones. But due to the amount of time the game will autosave maybe it will break hardware much faster.

Edit: I'm not saying it will just saying maybe. If it does break buy a new HDD or SSD.

Also, related: Yes, the SSD write-speed can still be higher than the HDD write-speed, but in case of the SSD it is at the additional cost of P/E cycles.

 

[TheBlueThunder]

Also, related: Yes, the SSD write-speed can still be higher than the HDD write-speed, but in case of the SSD it is at the additional cost of P/E cycles.

Thank you for educating the community.