Hey, I usually don't do this, but I have to clarify some things.
There are some things assumed in this thread that are simply not true. The chunk system is just one part of the engine and there are a lot of other systems (including group systems) layered for the game to work.
First of all: The chunk system. Why is it a good system and why do 90% of voxel games use some form of it:
- Clear Segmentation of data: A chunk is a clear part of memory. That makes for a lot faster access because you can reference it with static indices in an array. You can make assumptions to it's size and apply several data structures to it for optimization I'll mention below.
- Fast and memory efficient region systems by just indexing empty chunks, and not actually give them any real data.
- Fast referencing of single blocks needed by several physics systems as well as damage and state changing systems like logic.
- It allows for a unit of requesting and sending so that networking can be throttled to not stall.
- It is necessary for drawing, i.e. frustum culling. Else you end up iterating through stuff that is not in camera view. That is one of the biggest pitfalls in draw performance
- Lighting functions can be calculated threaded per chunk, which would be impossible with a group system.
- Fast generation and modification of octrees for rendering and physics.
- Saving and loading can be done per changed chunk.
- Physics: without the chunk system, several optimizations would be impossible. The engine makes use of octrees and Axis sweeps to minimize the problem of comparing millions of cubes to each other in milliseconds.
I'm not entirely sure which alternative is proposed in this thread, but from what I read, it's some kind of grouping system where you would combine equal blocks with equal orientation into groups. If that is the case there would be several game breaking problems with that:
- Logic: activation states change for individual blocks, and it happens often. That would trigger a group change and would need major CPU load to recalculate those groups.
- Ship battles: Again, groups are only changing when blocks are added/removed. In ship battles that happens thousands of times of a longer time period. Keep in mind that a damaged block already counts as a changed state and would invoke a group update.
- Building Style / Generation / Ship Battles: A grouping system's advantage would depend on a player's style to build. It would be strongest when doing big boxes of the same blocks. While some players do that for their system, others prefer other shapes rendering a possible grouping system slower and more convoluted. Worst case would be small system blocks of a few cubes that would be speckled across a ship. Ship battles will generate these kind of worst cases very very quickly. And using a group system would also affect the generation, because it too can generate worst case patterns.
- Comparing the rendering system to an engine that doesn't have complete editable voxels is a fallacy. Add back all the thousands of light sources of a ship into such an engine and then we will see how well that performs. The lighting is stored in the vertices and if you actually removed the vertices for rendering all lighting information in the vertices is lost. It can be solved by deferred lighting but that also has a limit of 50-100 lights which isn't enough. Deferred lighting will be added on top of the current rendering system for the closest lights to the camera in the future
If you can convince me of a better alternative that would have major performance boosts, I will use it no doubt. But I can tell you already, that it will be hard since you would probably need to know all requirements that are needed to process the game, and that is kind of impossible without looking at the code. There are a lot of things at work in this game you wouldn't assume are necessary, and all of this has to work for several situations.
But now to what it seems is being talked about: Connections (C+V) and Logic:
First of all, if a chunk isn't drawn because some kind of bug it doesn't mean it isn't loaded. It could be loaded on server and still function perfectly fine. It could be that just the lighting engine had problems processing it, so it would be on client as well as server. It's im possible for me to tell without looking at the ship itself reproducing the error.
Now, it is assumed that connections are tied into the chunk system. That is actually
NOT the case. Connections is a separate data structure
which actually uses a grouping system in some respect! Reason being, because it makes sense here, since there is only one state per connected block.
Logic not working can have different causes, but I'm working on improving it. I could just wait for all connected blocks to be loaded. Then you have the same effect as any other system which would ensure logic to fully work. The grouping also comes in handy when sending the C+V data to clients. The grouping system here also has several modes to minimize any worst case by also for example being able identify lots of long "sticks" and optimizing for those. There can still be some things done to decrease their size in memory, but they actually don't consume almost any processing power as is.
There is also a misconception on how power/weapons/shield/etc works. Believe it or not, but
this system ALSO uses a group system. The nice thing about this system is that it can have any algorithm for calculating data. The most common one is just detecting touching groups, there are complex ones that have several requirements like shipyard, but there are also simple ones that just count the connected blocks, which goes extremely fast obviously.
It also only changes when a block gets added/removed. The process is being done by multiple threads, so you hardly notice it. It can still be notices when two capital ships with huge blocks of systems destroy each other for the reason I mentioned above on why I use the chunk system for the "basic" data of blocks.
Basically when you add or remove a system block that has any meta-value like power, an update is scheduled for the worker threads to pick up. Then the groups for the power system are changed according to that. The only thing that could be done to speed that up in the beginning is to actually flat save those groups, and load them as is, instead of doing a full reconstruction of all groups at load time.
The downside would be larger file size, and problems when unloading/saving a ship that still has ongoing threads that would have to finish first (but that's solvable). The real reason why I don't do that already is because I want the base system to be as fast as efficient as possible, so I kind of want it to do a lot to identify bottlenecks, and over the years I found plenty and made the systems at least a thousand times faster already. These problems wouldn't be obvious if I would save and load it.
There is no "iteration over every block" per tick when processing any system. After having the groups the system just accumulates and calculates all the values like power, powercap, shield-regen, weapon power, combinations, etc. These values are then used for the updates per tick making the time taken for it O(1) or in other terms, constant!
Iterating over each block of a system per frame would be insane and end up in a slideshow with not even one frame per second, so I'm not sure why anyone that played the game would assume that I would do that...
The bugs of this system are still present when sometimes a signal gets fired while these groups are still in the process for first generating, it is possible for it to be "eaten". This was the case for the rail speed controller for example, where even if you put a rail to 0 speed, loading the structure up would move the docked ship that was placed on that 0 speed rail. This is because when checking for that rail's speed, the attached group would not be done calculating and report that the rail actually had speed (the default). This bug is however solved in the last version by just waiting for the group before making any decision.
There may still be similar bugs left, but those are solvable in the same way.
By they way, the reasons why planets are currently loaded slow is because of some missing optimizations on it, but I can't get into detail too much about it, because that would reveal too much about planned stuff in progress.
The reason why shift+v looks like it does is purely to calculate the touching blocks, and it is done on client. It has no impact whatsoever on how the game works.
If you have a bug, please report it. I'm not sure why your previous report was deleted, but usually there is a reason behind it.
I hope this clears up things a little bit.