Bug Surely 90+GB is overkill?

[Lidren]

Many servers clean mobs out via scheduled commands. So does mine. Many servers reset regularly, transferring sectors that need to be preserved. Mine does not. In fact, the whole point of the server I run is to stay running, indefinitely, and barring catastrophic meltdowns.

Tell me please, how I can realize this with a database that only increases in size exponentially as it runs? On an ssd with limited space?

Also, how does this even happen? An entire single galaxy of blocks should not be much over a few GB? I would have to do the math, but I fail to see how its even possible to get this big.

I am totally ok with resets, its alpha, it happens. The players are not though, just about every player I know would love to have a truly persistent server to play on. I want that to be possible, and not just for my server. We've reset twice now due to this bloat over two years, once it got over 100gb and irreversibly corrupted itself. The second time I reset intentionally as a preventative measure.

Things we've done that only mitigate or do not work at all:
Restrict travel to a single galaxy
Clean mobs daily
Disable mobs
Restrict ship and station size
Re-implement linear dampening so mobs don't just drift off

I have to give credit where it is due. The Schine engine is an amazing piece of work. This has been the only real problem I have had with it as both a player and server owner for two years.

 

[AndyP]

A lot of those issues are caused by players exploring straight forward into one direction.

There is quite a bit of information discarded and regenerated when not modified. (Planets for example)

There may files that were not used for a while and sort of useless,
but the main problem is:
Once you visit an area, and possibly consider it your future home it would be bad if coming back at a later time changed it totally.
So any change to the sector that is not regenerating the same way, will have to be saved.

There are approaches to zip the blockdata, this could save a lot of space, but will also cause a higher CPU load on loading entities.
(Compare: blueprint directory size vs sment file of same entity)
However this puts one limitation to overall entities.
For now, you could load an entity with a size of 25 km length, and when your computer could only handle loading (not display!) a small part of it, it could just pick the few chunks it can handle and create some function from them, with a zip stream you cant "chunk" data without loosing a lot of compression efficiency, so any entity has to be loaded "in one pass".

However, we are aware of that problem, and its one of the things that will be addressed, and so far we do not want to touch the "persistence" status for almost everything. (Compare to morrowind with "persistent containers" and "etheral ones" that make you loose items when not really watching out for the automatic garbage collection that cleans up small changes from the world to save some space.)

Its a two-bladed sword, its for example almost save to delete cores that are in space without any blocks attached and that were not entered for let me assume 3 months. For most multiplayerservers those core-masses around are more a nav-clutter than DB-problem, but what if someone used them in single player to mark paths between stations, cleaning the markers up will be a quite bad surprise.
So there are some cleanup scripts, they work great, but they may delete stuff some people remember.
Example:
Find a planet and put a faction module on each plate to mark your territory and make someone wanting to take over that planet at least declare war. If you forget about it for a while, or dont have time, you could assume in some cleanup that a modification of a single block per plate does not justitfy the saving of several million blocks into the database. However, resetting the plates would unfaction the planet and possibly cause a takeover of it. (with people complaining that planet got hacked somehow.)

Long post, short story:
Zip compressing the raw-block data is something we consider but this needs some benchmarking and extensive testing once implemented, as it wont be a "one-pass" compression of the DB, more likely a background process that checks loaded sectors and saves everything in new format.
But those details are not done yet, nor implemented.

- Andy