Database Compression

[levest28]

I would like to make a suggestion that would allow server database compression.

Just as the way smsec files are, a compressed copy of the sector, I would like to have the option for compressed databases. I understand how having a raw database as we do now can simplify the process of debugging, but not all servers would like to have such debugging, as my server certainly replies more on performance.

I've been running our server since September 2013, and we've reset our server a few times now, usually when we're "forced to" due to major StarMade updates. I've has servers as big as 170GB, and it's not always fun. The thing is, such raw files takes up unneeded space, that 170GB universe can be compressed with standard zip into about 4GB archive. (In fact, all of my compressed weekly server backups since September 2013 combined are less than 200GB.)

I've done everything I can considering the size to improve performance, such as giving the server files itself a dedicated drive formatted to have 64kb unit allocations. This helps reduce a lot of seek times, allowing the majority of files to be read from a single hard drive sector, rather than scanning multiple sectors. This also reduces the majority if any possible drive fragmentation, as files are less likely to be split.

Our server has dual Xeon's and 40GB of ECC ram. The ideal solution to improve performance would be to have StarMade compress it's database to be much smaller, allowing me to easily run even a very large server in a ramdisk (hardware has a 1 hour UPS to make sure any activity should a power issue occur be no issue, as well as ramdisk to hard drive synchronization).

Even if StarMade compressed even into many files, say into 5x5 or 10x10 sector chunks (much of it would be empty sectors anyways, right?), the saved space would be huge, and any "performance loss" caused by the fact it's compressed would be totally outweighed due to the fact it is already stored in memory, doesn't need to seek or transfer from disk, and can be instantly accessed.

While I fully understand the game is in Alpha, we've been running a StarMade server for about 16 months now, and playing for over 20 [months], and believe the ability to reduce the game's database size would significantly improve server performance for server's configured with it, and would be extremely beneficial to the game as a whole. Please not I am not asking for this to be put in place everywhere, but the ability to have an option to enable such database compression.

As always, thank you for reading and considering this suggestion.

 

[Mophise]

Options that could make a game run smoother with the right setup.... I'm all for that.

 

[Megacrafter127]

Compressing a file and still using it is easy, compressing a folder and still using it isn't, unless only read access is required.
You see, when compressing an amount of data, the size of the compressed data cannot be exactly determined. A long trail of zeros results in a negligable size of compressed data, while a random irregular sequence of bits can hardly be compressed that effectively at all, unless one is very lucky. This makes compressed folders hard to overwrite, unless you want to recompress the entire folder again every time you want to change a single file in it and the length of the compressed data changes. This cannot be avoided, unless one adds a padding to each entry, which allows even the worst case to still fit in, but that in turn wastes space. Also, if one is only after a specific set of bytes within the file, and those bytes aren't at the beginning of the file, compression requires one to still decompress from the beginning on, generating MB of decompressed data which is just skipped over, or just recompressed shortly after. And I doubt schema will take the long time it requires to design a compression algorithm which does not suffer from these problems, if such an algorithm is even mathematically possible while still being efficient enough at compressing data.

Edit: I forgot my main point: You may try to reduce the size of the .smd2 files within the server database using this tool: http://starmadedock.net/content/smd2shrinker.1771/
I still need someone to send me some test-results on this one, as all others who did download it were completely silent.

Quoting oneself from other threads is awesome :3

 

[jayman38]

Running as a RAMdisk sounds like a great way to overcome the delay from compressing and decompressing data on-the-fly. I agree that this should be an option for server admins.

 

[levest28]

Thanks Mega, will test it out, but still have a feeling it will leave us rather large worlds to run in a ram disk.
I have an 81GB world on hand I can test on (a copy of course). As you mentioned, zipping up a server and compressing the database, is not the same, but my chunking it into 5x5 or 10x10, it would still be a number of files with less data to work with at a time. Even if it loaded the data from archive and worked with it (loaded sectors) in a raw temp folder, only saving back into the archive on game saves, etc.

 

[Achriel]

Windows servers have an option for 'compressed folders' where the OS compresses data IN/OUT at the OS level. I wonder if this could be used or if there is a UNIX equivalent? The advantage would be Schema doesn't have to make something that already exists and doesn't have to support it.. and if it does exist it means you can test it out pretty fast using the symbolic link to the compressed folder/drive on a RAM Drive. In windows it's mklink (don't know the UNIX equivalent).

 

[levest28]

Sadly folder compression by the OS doesn't seem to be the solution. Just for reference, my 66GB server copy was about 12GB by OS compression. Still pretty good, but noticbly more than the 4GB of a zip archive. However, none of that seems to matter. Even with symlinks or such, for a ramdisk to operate, it needs to load the files, and although compressed on disk, as it loads from the disk to ramdisk, it it uncompressed while in use, and then compressed again when saved. This means doing so still would require the full original size of the files, regardless of OS folder compression (this is on based on my own attempts and testing) I do have one other method to attempt, which would be to create a dynamically sized VHD, allocate it into 512 byte units and then enable compressed OS folders, load the VHD into the ramdisk, then mount the VHD, but it still seems like a lot of extra work, increased risk of files, etc.

The real solution still seems to be database compression by StarMade itself, in which it would be compressed into 5x5 chunks or such. The other thing being, it not only reduces the file size allowing for ramdisk, but when you think about it this would also decrease the over all number/volume of files, which should help improve seek times. Then when a sector from that chunk is loaded, it can be loaded into a temp working folder, and then on game saves archived back into the sector chunk file or such. There is no need to be working with the compressed file in a live state all the time.

 

[BillGHero]

The savings for back-up time cannot be overstated. When running server maintenance every second counts. This kind of option would actually improve the games marketability. So it goes beyond mere convenience.

 

[LordXaosa]

Actually technologies allows us to use zip-on-fly, so I think it should be an option at server software. There should be an ability to choose less disk usage, but more CPU, or vice-versa.
For now, I launched server at the beginning of december (near 45 days ago) and for now DB is 60GB... At my host I'v got only 100GB, and also there should be backups, site and other things... So schema, please, make an ability to use zipped DB...

 

[levest28]

Yes, LordXaosa I do agree, and that would likely be best, but I was trying to make a point that it is not an absolute requirement to do this right off, as a simplistic version in the method I previously explained would likely quench the main issue for now, until things settle and the game's development as a whole evolves.

Cheers for the input!

 

[aidan1982]

Yes, I'm all for some form of compression, cause in all seriousness less than 10 People on a server constantly building and expanding there base's can chew up HDD space like crazy. Devs need to really look into compressing world file. I understand fully the reason behind why they want some of the stuff in raw format, but still the world folder shouldnt be filled with nothing but files upon files of every sectory, of every object, of all ships, of all turrets, planets etc. This stuff really needs to be compressed into a single world file, and just have a txt file output. I mean it would be better to just compress each system in its own file to save lots of space. Even the logs 4mb isn't large by itself but after you have 30+ or 100+, that adds fast.