Read/Write to Zip files for Blueprints

[Benevolent27]

I ran into a bit of a problem on windows. When trying to move blueprints to a specific folder, windows told me the path was too long. This folder was something like: "c:\folder\starmade\blueprints\11-29-16". This is 4 folders deep. First I wasn't able to move some of the blueprints over. Then when trying to delete some of the blueprints, the recycle bin wasn't capable of holding them because the depth of the folder structure was too great.

Example:

Right now, the format used for blueprints is to use subfolders for attached entities. This creates a problem in windows when a ship entities docked to other docked entities, in a chain. So, my suggestion is to simply use a zipped format to read and write to blueprints. This would take care of the directory depth problem. Another reason this would be good would be simply to allow drag and drop of .sment files directly into the blueprint folder. Copy and backup operations would be safer and simpler too. And this is important for some players who would like to easily share blueprints with each other via dock or other methods. Having to sit a computer illterate person down and explain how zip files work, and renaming sment files, ect., doesn't have to be a reality here. And in addition, I don't know what is causing the problem with bringing up the list of blueprints and saving new ones in single player and on servers (when there are many blueprints), but perhaps the implementation here can also seek to fix that issue.

 

[Snapems]

I do not believe the depth of the of the folder structure is the cause of that error. - I'm not sure what is, but that certainly is an issue that should be addressed - under the category of bugs.

Objects in the game can get very large, and adding a compression step E.G. Zip can very easily cause lag. Single files might be nice - but as you said, there's ships within ships... Soo, that complicates things.

Edit: Strike-out- I was wrong, see other's comments on why. Thanks.

 

[Crusade]

Individual components of a filename (i.e. each subdirectory along the path, and the final filename) are limited to 255 characters in windows. This is a hard limitation, and the chaindocking depth option is to prevent you hitting that limit. Move your starmade install to something like c:/starmade

 

[Lone_Puppy]

I found this a problem when docking first came out. I went crazy chain docking loads of entities to build a modular station. Discovered I couldn't move or copy the folder structure anywhere due to the path length restrictions in Windows.

Windows has always had a limit on path length and folder names.
Naming Files, Paths, and Namespaces (Windows)
I think storing chains of folders/subfolders etc in a zip would be a good thing.

 

[Benevolent27]

I do not believe the depth of the of the folder structure is the cause of that error. - I'm not sure what is, but that certainly is an issue that should be addressed - under the category of bugs.

Objects in the game can get very large, and adding a compression step E.G. Zip can very easily cause lag. Single files might be nice - but as you said, there's ships within ships... Soo, that complicates things.

It is most certainly the depth of the subfolders, whether due to how many subfolders there are or the character length is irrelevant. It is caused by the path being too long. This isn't NTFS that is the problem, it's an artificial limitation imposed by windows of 260 characters.

I can see why you might think this about zip files, but I also think it's not really very important. Heck, I used to use compressed drives with my old 486 computer which was a mere 66 mhz, and it worked fine (and this was 20 years ago). You wouldn't necessarily want to run windows off of a compressed drive, but it was ok for running most programs and storage. And computers are a heck of a lot faster nowadays. The overhead of read/write operations to a zip file is minimal. In fact, there are actually ways to speed up disk read/writes using zip files because it offloads the work from the hard drive to the processor. If you've ever tried to copy a folder with thousands of files/directories in it and then compared to copying a single zip file from one location to another, you'd see a humongous difference in speed. I've also seen benchmark tests of compressed file read/write operations compared to regular read/write operations and have seen many circumstances where you actually get a boost in speed when the files were compressed.

But, in any case, zipping the files will do away with the directory limitations, will make the blueprints much more portable and easy to manage, and will reduce disk IO lag when the blueprints are copied, especially if there are a lot of entities attached to the parent ship. Another thing to take into consideration is the size difference. I have a blueprint that is 57.8mb large, but zipped it's a mere 3.68mb. I've noticed that when uploading to the server, it lists it as 4058kb, so it seems to be compressing it when uploading it, so I know StarMade already has this capability built in. Of course the devs would want to do some speed tests to see how it affects performance, but the folder limitation is a real problem for windows and should be addressed.

 

[Batavium]

As far as i understand StarMade uses it's own zip files in the form of sment. When you export a blueprint the whole folder structure gets compressed into the .sment file. When you import the .sment file the content and folder structure is restored into Starmades Blueprint folder.

So basically if Starmade would use .sment on the fly then this path length problem would be gone. However Starmade seems to load all blueprints when it starts the game. If this uncompression process would have to be done at every game start then loading could take a long time. Depending on how many Blueprints you have in your blueprint folder.

Somehow StarMade needs to be made able. To only load blueprints when needed on the fly. Even when the game and server are already running. And not everything that is in it's blueprint folder just because they exist there. Your blueprint folder would then only contain .sment files.

 

[Benevolent27]

As far as i understand StarMade uses it's own zip files in the form of sment. When you export a blueprint the whole folder structure gets compressed into the .sment file. When you import the .sment file the content and folder structure is restored into Starmades Blueprint folder.

So basically if Starmade would use .sment on the fly then this path length problem would be gone. However Starmade seems to load all blueprints when it starts the game. If this uncompression process would have to be done at every game start then loading could take a long time. Depending on how many Blueprints you have in your blueprint folder.

Somehow StarMade needs to be made able. To only load blueprints when needed on the fly. Even when the game and server are already running. And not everything that is in it's blueprint folder just because they exist there. Your blueprint folder would then only contain .sment files.

I honestly don't think StarMade loads every blueprint into active memory upon the game starting. I believe it only loads the blueprint configuration file, which contains the data it needs to provide the ship size, owner, ect. in the blueprint list. I believe it only loads the actual entity files into active memory when loading a ship into the physical world. Sure it is true that loading uncompressed data is probably going to be faster than loading compressed data, but it could actually be faster loading compressed data, especially if there are a lot of disk reads of hundreds of files, because it can load the compressed file into memory and then do all the processing in memory, rather than having to access hundreds of individual files, loading them into memory separately and then processing that data. It's the same reason why copying a folder that has thousands of files and subfolders can take a long time, but copying the zip file is extremely fast. I think benchmarks would need to be done to see what the impact is, but ultimately the directory structure limitations of windows is the biggest problem here. Even if a file format is used, which does NO compression to circumvent this limitation, it'd be better than storing these entity files in long directory chains. The last thing anyone wants is blueprint corruption due to windows directory limits.

 

[Snapems]

Welp, yes please.

I'm not convinced about using Zip to solve the problem. But this is clearly a problem that deserves attention by the developers.

 

[Benevolent27]

Welp, yes please.

I'm not convinced about using Zip to solve the problem. But this is clearly a problem that deserves attention by the developers.

I suggest using a zip file because that's what .sment files are. They're just zip files. It's already partially implemented, being the standard format used for sharing StarMade blueprints on the dock. Why not just allow the .sment file to be dropped into the blueprint folder? And when a player wants to upload the blueprint, just upload the .sment file.

 

[Ithirahad]

The limitations imposed by file-based docking trees are annoying in the first place. (e.g. you can't have extendable docking tubes on your ship, as it's impossible to have a singular file with two folder locations)

Changing that system would fix this issue by default, and also open up new possibilities for the game...

 

[Benevolent27]

The limitations imposed by file-based docking trees are annoying in the first place. (e.g. you can't have extendable docking tubes on your ship, as it's impossible to have a singular file with two folder locations)

Changing that system would fix this issue by default, and also open up new possibilities for the game...

That is an interesting point! Perhaps the blueprint could be structured as a database instead, thus allowing more than simple linear connections. You could then "inherit docking," allowing for an extendable docking arm.

I don't think the files "have to" be compressed, they just need to be in a file. But if they were compressed, then that'd be ideal so they'd also be easily portable too. HSQL databases actually support internal compression. If they could somehow offload this load from the main thread, then I think the impact on in-game performance would be minimal and would only occur when a ship is being initially loaded in from a blueprint. (Think maybe a fraction of a second longer to load a large ship from a blueprint - which then works exactly as it normally would because it is now using an .ent file, not the blueprint to load)

 

[jayman38]

I've looked into Zip-file speed-vs-compression ratios and efficiencies before. Zip Compression is designed to be a relatively fast and tiny operation for the benefit you get. Today's modern processors make short work of zip operations, to the point where it will barely generate a blip on the CPU load chart.

In other words, you don't get an immense improvement in speed by reducing or eliminating compression. (You can create an uncompresses ZIP file which is just a full-sized package of multiple files, if you really want that.) On the other hand, just a tiny bit better compression requires significantly more processing than normal compression.

My point is that regular, moderate, efficient compression of a whole blueprint will not be the lag bomb generated by some of the more intensive Starmade operations (think: planet interactions, heavy logic, and intersection calculations). Zip compression was relatively speedy on computers built two decades ago. The speed increase of reduced disk I/O will greatly outstrip any zip-operation processing. Do not worry about lag caused by zipping. Seriously, it's tiny.