Dev Blog : July 28th 2016

[Bench]

Surprise! Dev blog time! Last time we were talking about asking for your input on what you wanted as part of our in-game video tutorials. Well we've taken those on board and have released the first set of in-game video tutorials to cover the basics of the game. We'll continue to add to them as time goes on and to assist in teaching new major features planned for the game.

This dev blog however is more focused on what's currently being worked on, as it changes a lot of core code for the game and has some ramifications that you as the StarMade community should be aware of.

The Chunk Update

If you're unfamiliar with how our blocky universe is comprised, blocks are grouped into chunks. 16 x 16 x 16 cubes that are used to manage data in a more practical way. It influences everything from performance to lighting and everything in between and is one of the pieces that make up the backbone of the game.

What's currently being worked on is an increase to the chunk size in order to improve performance. Chunk sizes will be changed from 16 x 16 x 16 to 32 x 32 x 32. This increases the number of blocks per chunk from approx 4k to 32k. This primarily will help reduce the number of requests and operations on large objects but will influence many other aspects of the game as well, improving everything from loading and even compressing. We estimate up to 50% increase in performance in some areas due to this change.

What this will mean however is a major conversion period, and when rolled out could cause some significant downtime for servers. The converting is mostly for moving all blocks to the new chunk grid.

For individual ship entities, the ship core was originally placed at coordinate 8 8 8 placing the ship core in the middle of a chunk. This meant that for smaller ships it could be possible to only require the loading of one chunk, rather than if the core was located in the corner of a chunk, it had the potential for 8 chunks to instead be required for loading. All blocks in a blueprint are saved "relative to the core" so a block located at 0 1 0 would actually be located at 8 9 8 in the chunk. This relative locating is one of the ways StarMade is able to support quite large ships.

Shifting to a 32 x 32 x 32 chunk format means that every ship entity needs to have the ship core shifted to 16 16 16 instead of 8 8 8. This is why the conversion process will take quite a while, as not only are all the existing entities in the universe needing to be updated per server, but also all the saved blueprints. In addition, not only do the raw blocks have to be shifted, but associated with them such as their connections, metadata (e.g. texts, names, inventories) etc. The processing time on making this shift on a single entity is also the reason why a simple core moving system for build mode has been difficult to implement.

This conversion could result in servers taking at most a couple hours to convert. This change however is going to present a number of major improvements as well as improving processing for some future core code changes planned later on in the development timeline. It also significantly reduces the database size for servers and reduces the amount of files required. For example, converting an old server saw the blueprints folder drop from 11Gb to 3.2Gb, and the server database go from
19Gb to 4.6Gb.

Why we're telling you this now is so you know what's ahead. We're going to do our best to ensure that everything is as prepared as possible before we release this update. However we do recommend you back up any blueprints, ships, servers, sectors, etc locally that you want to ensure you save just in case anything unforeseen happens during the conversion process.

Have any questions, feel free to ask them, otherwise watch the news space for all the details going ahead in anticipation for this update.

- The Schine Team


ABOVE UPDATED
• Included how metadata etc plays a part in conversion times
• Added in some raw numbers that previously we didn't have

 

[kiddan]

Thanks for the heads-up! It's great to see all the performance-enhancing updates we've been receiving lately. Keep up the good work, Schine!

 

[Xacktar]

How will it handle the spawning of older blueprints after conversion?

Will we have a tool to convert blueprints to the new chunk format to prevent problems?

 

[Zekester81]

My giant, unfinished station approves of performance increases.

 

[AtraUnam]

Any ideas on how this is going to effect collisions?

 

[jstenholt]

How will it handle the spawning of older blueprints after conversion?

Will we have a tool to convert blueprints to the new chunk format to prevent problems?

I too have a similar question.

 

[waski35]

I also subscribe to this question. Would it be offline tool, or just addition do build mode, which automatically convert blueprint on editing ?
On my server theres over 120 blueprints, and i suppose it will not be major problem to convert them on server (with planets, stations, and floating ships), once, but what with players, who will try tu upload blueprints with old chunk size ? Will it work on upload, or just as I written in time of first edit (build mode) ?

 

[Bench]

To answer some of the questions above...

• Uploading older blueprints is not yet implemented and may fail. On the first start of the new version any blueprints in your folders will be converted to the new chunk format.
• Regarding conversion speed we did a test conversion of 4720 blueprints, with a total size of 11.2Gb, and that took 11 minutes to convert.
• Collisions will be faster and less CPU intensive, as the amount of chunks to compare is drastically reduced. If you consider that 8 old chunks fit into 1 new chunk, that's a significant reduction in the number of chunks involved in calculating collisions (1 chunk instead of 8). So in theory the performance for collisions should increase by roughly 50-60%.

 

[Lone_Puppy]

Will default sectors be changing or increasing in size?

 

[AndyP]

Will default sectors be changing or increasing in size?

Not yet. (If at all)

Its mostly to help the loading speed, and we can already confirm a great increase in performance for large stations and structures.

 

[StormWing0]

hmm so that's what that tear in reality was the last time I popped into the dev build. XD

 

[AndyP]

yeah, we moved a copy of the last chunk16 state to play.star-made.org:4848
the :4242 one will be reset and wiped like every day until we get through it without problems in a reasonable time.

Light a candle for the HDD in the server to give it the strength to survive this. (Writing and wiping 60.000+ files and 63GB twice a day... ^^)

Some snapshot of the "smaller-db" we run through the conversion test:

Red/left: State before converting
Green/right: After conversion (and deleting the 2 backup zip files)

Time spent on blueprints: 12 minutes
Time spent on database: 37 minutes
Total conversion time: 49 minutes

So the gain in compression by using larger data chunks give the zip-compression more room to show its advantages. =)

- Andy

 

[BJammin]

Huzzah! Performance improvements! I'll take those kinds of updates before new feature implementations any day (though the new stuff is fun too )

 

[StormWing0]

lol every time someone says performance improvements I say computer meltdown waiting for a place to happen. >_<

 

[Croquelune]

Does that'll affect AI aiming system ?

 

[reptillianstew]

I feel like i'm playing with fire every time I load up a new dev build. I got messages about blueprints being corrupted when I tried to revert to stable (understandable) and all the ships in my vicinity are missing chunks in the most recent build.

 

[Phocian]

This appears to be a massive update way beyond anything recent.I would leave the Dev builds alone unless you want to bug report.

Btw through the Starmade report phabricator.

Phabricator

 

[Doomsider]

As a guy who plays around with servers a lot this is a big deal especially the size reduction. Keep up the good work!

 

[schema]

Using the dev build at the moment is dangerous, but not fatal. Every conversion does an automatic backup in your StarMade dir.

Uploaded blueprints are also already converted on the spot (when it attempts to load and reads the header it will know if it needs to do migration).

We're aiming that this update will be fully compatible with every previous version of data.

In terms of collision, there might possibly be some slight slowdown due to the chunk margins (borders around chunks to start checking for collision) being too big causing extra work. Theoretically however, there will be no slowdown at all, because the new system uses octrees just as the old system, so basically only one layer of complexity shifted from 16 to 32. The difference would be that the sweep which was done on chunk16 is now performed on chunk32. but the direct octree comparison is now starting on 8 times 16x16x16 instead of 8 times 8x8x8. If there should be problems, I can have the physics shift the layer itself without performance determinants.

However it is also highly likely that using chunk32 might even speed up physics performance after follow up optimizations in further updates have been done.

 

[AndyP]

I feel like i'm playing with fire every time I load up a new dev build. I got messages about blueprints being corrupted when I tried to revert to stable (understandable) and all the ships in my vicinity are missing chunks in the most recent build.

You may want to read this topic.
In general, its recommended to move everything from your dev-build folder in your regular folder (blueprints, exports, templates... but NOT whole worlds) after every release and wipe the directory.
Then start with a fresh copy from your release installation in the next dev-cycle.

Anyway, if you frequently play those, click here.
We should be able to make a contact for this, to give at least some support if something breaks.

- Andy