Support for forcefully resaving/rewriting entity blockdata, via admin command: /touch_uid

[Erth Paradine]

Request/Goal:
Enable admins/operators to programmatically and forcefully save an entity's blockdata to disk.

Currently:
The server's entity/sector unload routine does not re-write an entity's blockdata to disk if entity's "touched" attribute is false. The only known way to currently set an entity's touched attribute to true, is in-game, by making actual changes to an entity's blockdata (e.g. adding/removing blocks, or activating/deactivating logic, lights, AI etc).

Result of proposed:
While evaluating an entity for removal from memory, the server would remove existing blockdata stored on disk, and then write a new/fresh datafile using blockdata from memory.

Suggested Method:
The tool "touch", is common throughout the Linux/Unix world; it's purpose is primarily to adjust a file or folder's access or modification date without requiring a file be opened, saved, and closed.

The equivalent here, would be implementing a way for the game engine to know if a file needs to be re-written, regardless of current in-memory blockdata/metadata attributes, without requiring that a player manually make a blockdata level change to the entity.

A way to support this function programmatically, new command:

/touch_uid [UID of entity to set "touched=true"]​

Rationale:

• StarMade makes extensive use of sparse files, however this is still a relatively new concept for both experienced, and inexperienced, developers, engineers, and users.
• Sparse files support is not ubiquitous nor usually automatic. Therefore it's inevitable for players and/or admins to transfer datafiles using tools unable to support sparse files (e.g. FTP, SFTP, SCP, PkZIP, etc). The result of this can balloon datafiles to many times their original "sparse" size.
  • For a good illustration of this, compare output from the following two commands on any StarMade folder structure:

  • du -d0 -h --apparent-size StarMade/server-database/
    du -d0 -h StarMade/server-database/

• Some of StarMade's own functions lack sparse file support, for instance when backing a server up on Windows (e.g. PkZip lacks sparse support). This means that if a windows server operator depended upon SM's backups to restore a server database, they're risking some profound disk space consumption issues upon extraction of a .zip backup.
• An extreme example of removing sparseness from an entire server's database:
  • With sparseness intact, this server requires only ~26GB of disk space:

    du -d0 -h StarMade/server-database/
    26G    StarMade/server-database/

  • With no sparseness intact, this server would require ~189GB of disk space:

    du -d0 -h --apparent-size StarMade/server-database/
    189G    StarMade/server-database/

• As a "workaround" for the very real scenario of accidentally removing a file's sparseness, it would be very useful if players could load entities (e.g. /sector_load_range), and then force the server to re-write each entity's data into a new datafile on disk.

 

[NeonSturm]

So, sparse files "don't save empty chunks", but by outside tools rather than having SM use a proper file structure?

 

[Erth Paradine]

Sparse files are a proper file structure, and in-fact part of a standard java API. Sparse files are just fairly new for many people.

What sparse files do, in a nutshell...when there's a long string of identical/repeating data (e.g. null, or binary-zero, characters):
- Non-sparse files would write a long string of zeros to the file, therefore consuming space on disk for what's basically nothing. This also has the effect of slowing disk reads, as an entire file (and all its zeros) would need to be read from disk.
- Sparse files do not write a long string of zeros, they instead leverage metadata to indicate where a long string of zeros starts and stops within the file, and then the actual zeros are not written to disk. Programs that can support sparse files, look for and transparently represent this metadata when reading/writing the file. This also has the effect of greatly speeding up disk reads as well: smaller files, faster reads, and the server can leverage metadata to represent (in-memory) where there should be large blocks of null characters.

The issue, is that some common tools just don't support sparse files. For instance, FTP, SCP, SFTP, and PkZip. Tools that do support sparse files are standard on Linux, but not usually found on the Microsoft platforms: rsync, tar, cp (GNU version), etc.

My suggestion, seeks a middle-ground here: enable admins to recover space lost when shuffling files around by tools unable to support sparse files.

I certainly don't want my server DB ballooning from ~30GB to ~200GB, just to support a bunch of empty space; hence sparse files have distinct value, but they need to be accommodated more intelligently.