Dev Blog : July 28th 2016

    Joined
    Feb 27, 2014
    Messages
    1,074
    Reaction score
    504
    • Purchased!
    • Legacy Citizen 4
    • Top Forum Contributor
    Not yet. (If at all)

    Its mostly to help the loading speed, and we can already confirm a great increase in performance for large stations and structures.
    Thats awesome :D!
    Is there any performance increase for ship with lots of docked entities:? That seems to be one of the biggest causes of lag/low fps
     
    Joined
    Dec 2, 2013
    Messages
    232
    Reaction score
    98
    • Legacy Citizen 2
    • Purchased!
    • Competition Winner - Small Fleets
    Surprise! Dev blog time! Last time we were talking about asking for your input on what you wanted as part of our in-game video tutorials. Well we've taken those on board and have released the first set of in-game video tutorials to cover the basics of the game. We'll continue to add to them as time goes on and to assist in teaching new major features planned for the game.

    This dev blog however is more focused on what's currently being worked on, as it changes a lot of core code for the game and has some ramifications that you as the StarMade community should be aware of.

    The Chunk Update

    If you're unfamiliar with how our blocky universe is comprised, blocks are grouped into chunks. 16 x 16 x 16 cubes that are used to manage data in a more practical way. It influences everything from performance to lighting and everything in between and is one of the pieces that make up the backbone of the game.

    What's currently being worked on is an increase to the chunk size in order to improve performance. Chunk sizes will be changed from 16 x 16 x 16 to 32 x 32 x 32. This increases the number of blocks per chunk from approx 4k to 32k. This primarily will help reduce the number of requests and operations on large objects but will influence many other aspects of the game as well, improving everything from loading and even compressing. We estimate up to 50% increase in performance in some areas due to this change.

    What this will mean however is a major conversion period, and when rolled out could cause some significant downtime for servers. The converting is mostly for moving all blocks to the new chunk grid.

    For individual ship entities, the ship core was originally placed at coordinate 8 8 8 placing the ship core in the middle of a chunk. This meant that for smaller ships it could be possible to only require the loading of one chunk, rather than if the core was located in the corner of a chunk, it had the potential for 8 chunks to instead be required for loading. All blocks in a blueprint are saved "relative to the core" so a block located at 0 1 0 would actually be located at 8 9 8 in the chunk. This relative locating is one of the ways StarMade is able to support quite large ships.

    Shifting to a 32 x 32 x 32 chunk format means that every ship entity needs to have the ship core shifted to 16 16 16 instead of 8 8 8. This is why the conversion process will take quite a while, as not only are all the existing entities in the universe needing to be updated per server, but also all the saved blueprints. In addition, not only do the raw blocks have to be shifted, but associated with them such as their connections, metadata (e.g. texts, names, inventories) etc. The processing time on making this shift on a single entity is also the reason why a simple core moving system for build mode has been difficult to implement.

    This conversion could result in servers taking at most a couple hours to convert. This change however is going to present a number of major improvements as well as improving processing for some future core code changes planned later on in the development timeline. It also significantly reduces the database size for servers and reduces the amount of files required. For example, converting an old server saw the blueprints folder drop from 11Gb to 3.2Gb, and the server database go from
    19Gb to 4.6Gb.

    Why we're telling you this now is so you know what's ahead. We're going to do our best to ensure that everything is as prepared as possible before we release this update. However we do recommend you back up any blueprints, ships, servers, sectors, etc locally that you want to ensure you save just in case anything unforeseen happens during the conversion process.

    Have any questions, feel free to ask them, otherwise watch the news space for all the details going ahead in anticipation for this update.

    - The Schine Team


    ABOVE UPDATED
    • Included how metadata etc plays a part in conversion times
    • Added in some raw numbers that previously we didn't have
    Cool! Servers running on SSDs will now be way more cost effective.

    With better saving and loading can we get asteroid fields that extend from sector edge to sector edge and scale with sector size? Ya know an actual asteroid "belt"? Opposed to the lonely little groups of rocks we have now.
     
    Joined
    Apr 25, 2016
    Messages
    19
    Reaction score
    16
    will this update give us more fps when looking at big things like astroids and stations?
     

    AndyP

    Customer Experience Manager
    Joined
    Aug 15, 2013
    Messages
    1,199
    Reaction score
    264
    • Schine
    • Wired for Logic
    Thats awesome :D!
    Is there any performance increase for ship with lots of docked entities:? That seems to be one of the biggest causes of lag/low fps
    The actual ships with high counts of entities will not gain performance from this.
    They will load fast, but high counts of overlapping boundary boxes will stay the same.
    But as schema stated already, we will put some care in that afterwards.

    will this update give us more fps when looking at big things like astroids and stations?
    Yes, the 'render calls' are actually per chunk, so every chunk gets rendered on its own, and by putting 8 chunks into one, we reduce a lot of those draw calls, with quite noticeable performance increases.

    - Andy
     
    Joined
    Jul 27, 2014
    Messages
    70
    Reaction score
    11
    • Legacy Citizen 6
    This is actually a really good step in the right direction for the game. In my opinion you have now added enough features into the game especially helped by the fleets and trade system to actually keep new players busy for more than one week. The only real plauge of the game nowdays is the lag. The bugs are reducing ever more quickly with help of the surprisingly effective tester team so the only real task left to do before you get a more professional and polished game is to reduct the performance for both client and server which this update is doing.

    well and gameplay balance needs looking at but......
     
    Joined
    Oct 11, 2015
    Messages
    21
    Reaction score
    9
    • Purchased!
    Thanks to the dev team for their answers. Optimisation, is very welcome, since I, and many of players visiting my server like capital ships. Performance already is visibly better due to recent patches, I'm looking forward for more.
     

    lupoCani

    First Citizen
    Joined
    Jun 23, 2013
    Messages
    504
    Reaction score
    127
    • Purchased!
    • Legacy Citizen 10
    With all the promised benefits of this update, I sort of have to wonder... why now?

    Overhauling the engine in this fashion may of course have been impractical, but with all these benefits, why were chunks even 16*16*16 in the first place? What reasons were there for choosing that number when the Schine engine was built, that have since ceased to apply?

    Also, why 32*32*32? If enlarging chunks increases performance, what is that makes even larger ones, say, 64*64*64 impractical? Are there diminishing returns for this enlargement that would be outweighed by a smaller, but more constant detriment? What is this detriment, in that case?

    Come to think of it, this will increase RAM usage, won't it?
     

    Erth Paradine

    Server Admln & Bug Reporter
    Joined
    Feb 15, 2016
    Messages
    239
    Reaction score
    58
    ...
    Time spent on blueprints: 12 minutes
    Time spent on database: 37 minutes
    Total conversion time: 49 minutes

    So the gain in compression by using larger data chunks give the zip-compression more room to show its advantages. =)

    - Andy
    A test on our blueprint and world DBs (which are roughly twice the size of Schine's test DB) shows this takes about twice as long, so roughly speaking I'm seeing a linear relationship between conversion size and time. So happy with the reduction in disk space requirements too.

    ...
    Also, why 32*32*32? If enlarging chunks increases performance, what is that makes even larger ones, say, 64*64*64 impractical? Are there diminishing returns for this enlargement that would be outweighed by a smaller, but more constant detriment? What is this detriment, in that case?

    A system of 16x16x16 = 4096 blocks (voxels) per chunk, which consumes 12,288 bytes per chunk (assuming SM still uses 3 byte voxels). On a 10Mbps connection this requires 10.1ms to transfer. For the desktop/GPU, smaller chunks also pose a significant rendering overhead. There's also AABB (e.g. collision) checks: a collision typically interacts with more chunks when they're smaller, and therefore more calculation work is required when there's more/smaller chunks.

    Upcoming system using 32x32x32 = 32678 blocks per chunk, consumes 98,034 bytes per chunk. This requires 80.3ms of time to transfer, and should help improve mesh generation for the desktop/GPU, while also reducing the quantity of chunks involved in many types of collisions (therefore making AABB checks less painful).

    A jump to 64x64x64 = 262,144 bytes/chunk, requires 214.7ms to transfer...from a server perspective, that's a point of diminishing returns. Primarily because anytime a single block is changed, the entire chunk (+ any colliding/interacting chunks) must be recomputed/resent. That would require a lot of bandwidth, and I assume would introduce too much lag.

    Come to think of it, this will increase RAM usage, won't it?
    I'm kicking tires with w/ a ~50GB copy of our server's DB, and have NOT seen a proportional increase in RAM consumption. YMMV
     
    Joined
    Jul 14, 2013
    Messages
    98
    Reaction score
    27
    • Purchased!
    • Community Content - Bronze 1
    • Legacy Citizen
    The actual ships with high counts of entities will not gain performance from this.
    They will load fast, but high counts of overlapping boundary boxes will stay the same.
    But as schema stated already, we will put some care in that afterwards.



    Yes, the 'render calls' are actually per chunk, so every chunk gets rendered on its own, and by putting 8 chunks into one, we reduce a lot of those draw calls, with quite noticeable performance increases.

    - Andy

    So its clear there are very important improvements made by moving to the new chunk system, but I'm confused why one wouldn't make the chunk size 64x64x64 instead, or any other higher power of 2 cubed? What is the disadvantage of doing that?
     

    DukeofRealms

    Count Duku
    Joined
    Sep 4, 2013
    Messages
    1,477
    Reaction score
    1,617
    • Schine
    So its clear there are very important improvements made by moving to the new chunk system, but I'm confused why one wouldn't make the chunk size 64x64x64 instead, or any other higher power of 2 cubed? What is the disadvantage of doing that?
    A system of 16x16x16 = 4096 blocks (voxels) per chunk, which consumes 12,288 bytes per chunk (assuming SM still uses 3 byte voxels). On a 10Mbps connection this requires 10.1ms to transfer. For the desktop/GPU, smaller chunks also pose a significant rendering overhead. There's also AABB (e.g. collision) checks: a collision typically interacts with more chunks when they're smaller, and therefore more calculation work is required when there's more/smaller chunks.

    Upcoming system using 32x32x32 = 32678 blocks per chunk, consumes 98,034 bytes per chunk. This requires 80.3ms of time to transfer, and should help improve mesh generation for the desktop/GPU, while also reducing the quantity of chunks involved in many types of collisions (therefore making AABB checks less painful).

    A jump to 64x64x64 = 262,144 bytes/chunk, requires 214.7ms to transfer...from a server perspective, that's a point of diminishing returns. Primarily because anytime a single block is changed, the entire chunk (+ any colliding/interacting chunks) must be recomputed/resent. That would require a lot of bandwidth, and I assume would introduce too much lag.
    Erth Paradine pretty much got all the reasoning for moving from 16x16x16 to 32x32x32 instead of 64x64x64.

    There's also less calls to draw for the GPU, instead of having to call to draw 8 meshes, it's just one. "The size of the chunk is more like a sweetspot where you want to do most operations on, e.g. like lighting calculations. You could theoretically do lighting on parts of a chunk, but then you would introduce more complexity" - schema

    Basically, there are tradeoffs for increasing the chunk size, these tradeoffs eventually outweigh the benefits they provide when increased. It just so happens that 32x32x32 provides enough benefits to outweigh both 16x16x16 and 64x64x64 chunk sizes.
     
    • Like
    Reactions: Erth Paradine
    Joined
    Apr 25, 2016
    Messages
    19
    Reaction score
    16
    The actual ships with high counts of entities will not gain performance from this.
    They will load fast, but high counts of overlapping boundary boxes will stay the same.
    But as schema stated already, we will put some care in that afterwards.



    Yes, the 'render calls' are actually per chunk, so every chunk gets rendered on its own, and by putting 8 chunks into one, we reduce a lot of those draw calls, with quite noticeable performance increases.

    - Andy
    Awsome! more FPS for players with low intergrated graphics and cpu like me! :p
     

    Thalanor

    CEO Snataris Colonial Fleetyards
    Joined
    Sep 10, 2013
    Messages
    818
    Reaction score
    708
    • Master Builder Bronze
    • Thinking Positive
    • Legacy Citizen 3
    So on average 8 times less chunks per entity. I always assumed structures were octtrees but thanks for the technical info :) Looking forward to building large entities that can actually be used on a server.
     

    Matt_Bradock

    The Shrink
    Joined
    Aug 4, 2013
    Messages
    798
    Reaction score
    464
    • Purchased!
    • Thinking Positive
    • Legacy Citizen 5
    SO MUCH YES.

    Finally my craptop will be able to better handle Starmade! Huzzah!
    I always wholeheartedly welcome any updates that improve general game performance, ESPECIALLY collision detection and loading time. So good to know you never stop looking at how to make it run faster :)
    Does this also improve on the well-known issue with large stations (with a lot of docked entities like turrets, rail doors, other ships) generating insane FPS drops and ping spikes when a ship with docked entities enters their collision detection bounding box?
     
    Joined
    Jul 15, 2014
    Messages
    506
    Reaction score
    111
    Already been said that this won't affect collision checks for multiple entities, but something is in the pipeline for them.
     

    AndyP

    Customer Experience Manager
    Joined
    Aug 15, 2013
    Messages
    1,199
    Reaction score
    264
    • Schine
    • Wired for Logic
    We already have an octree system in place.
    (Looks like that was misunderstood at some point.)

    Lancake created a picture that shows the current problem with small turrets and larger chunks being close together:
    upload_2016-7-29_18-14-36.png

    So currently the performance is rather worse than better, if stuff moves close to each other.
    But schema is already working on this, while we test the conversion steps.

    - Andy
     

    AndyP

    Customer Experience Manager
    Joined
    Aug 15, 2013
    Messages
    1,199
    Reaction score
    264
    • Schine
    • Wired for Logic
    Another conversion:

    upload_2016-7-30_0-40-52.png

    Conversion time: 62minutes

    To speed it up, I added this to the server.cfg:

    Code:
    BACKUP_WORLD_ON_MIGRATION = false //Back up world when migrating to a new file format
    BACKUP_BLUEPRINTS_ON_MIGRATION = false //Back up blueprints when migrating to a new file format
    (New option)

    This was added in case a server has its own backup and does not need a second copy.

    - Andy
     
    • Like
    Reactions: Xacktar

    Erth Paradine

    Server Admln & Bug Reporter
    Joined
    Feb 15, 2016
    Messages
    239
    Reaction score
    58
    ...

    Code:
    BACKUP_WORLD_ON_MIGRATION = false //Back up world when migrating to a new file format
    BACKUP_BLUEPRINTS_ON_MIGRATION = false //Back up blueprints when migrating to a new file format
    (New option)

    This was added in case a server has its own backup and does not need a second copy.

    - Andy
    Love this. Any chance of adding a BACKUP_WORLD_ON_UPGRADE option, for the same reasons. Or does the launcher not look at this file?
     

    AndyP

    Customer Experience Manager
    Joined
    Aug 15, 2013
    Messages
    1,199
    Reaction score
    264
    • Schine
    • Wired for Logic
    Nope, only the game cares for that.
    Having opt-in backups is something for calani but I am sure its already planned.

    I only found this one:
    T1502 - but thats a different use case.

    - Andy
     
    • Like
    Reactions: Erth Paradine
    Joined
    Jun 22, 2013
    Messages
    115
    Reaction score
    171
    • Community Content - Bronze 1
    • Purchased!
    • Legacy Citizen 4
    I have all my blueprint backed up in a separate folder. Would it be possible to choose which folders to convert?