I think a good question to ask of this premise is "which engines (if there are any) would adequately handle such large block/voxel structures?" Since that is the core of the game, i think that would be the best starting place when looking for an engine. This is not FPS engine territory, since most FPS games deal with static maps and are more concerned about individual player mobility/graphics.
The only engines i can think of in use by apparently similar games i've played are unity (robocraft, kinetic void) and custom (minecraft).
I haven't played robocraft in some time, but last i did it was extremely buggy in terms of block creation/destruction. As in - creating worked fine in the garage/studio interface, but once you were out playing (and again, on a static map) any kind of collision or damage would quickly make your vehicle unplayable. Being "repaired" in the field didn't work as intended at all, and often your only option would be to self-destruct and respawn, since the game could handle correctly spawning a new copy of your original design. Even the concept of a "respawn" in a MOBA like that is only somewhat similar to the deconstruct/rebuild mechanics of this game.
Kinetic Void hasn't seen an update in almost 3 years i think, but it has probably the most similar feel in terms of space exploration and modular ship design. It was also extremely buggy / borderline unplayable, and didn't have the option of connecting players on servers or traveling in direct paths between star systems. Each star system was a zone with finite borders, and you needed a gate to travel between systems/zones.
Minecraft is sort of the gold standard in terms of block games, and it is driven by a custom engine. Part of the necessity was that there was nothing like it before, which i think is part of the same necessity here. Minecraft also deals with dynamic, zoneless environments, which necessitates saving/unloading/reloading chunks of map in a way that you don't see with a static arena, and it has its own internal logic/mechanics for handling environment to environment interaction that can alter the landscape in customizable and unpredictable ways. Sound familiar?
I think for a game like starmade with no clear precursor, you're not going to be able to find a readymade engine that can handle what the game demands, because you can't make an engine for something that doesn't exist. The engine is most likely being made as the game is being made, with alterations and customizations coded as needed to realize design goals.
The only engines i can think of in use by apparently similar games i've played are unity (robocraft, kinetic void) and custom (minecraft).
I haven't played robocraft in some time, but last i did it was extremely buggy in terms of block creation/destruction. As in - creating worked fine in the garage/studio interface, but once you were out playing (and again, on a static map) any kind of collision or damage would quickly make your vehicle unplayable. Being "repaired" in the field didn't work as intended at all, and often your only option would be to self-destruct and respawn, since the game could handle correctly spawning a new copy of your original design. Even the concept of a "respawn" in a MOBA like that is only somewhat similar to the deconstruct/rebuild mechanics of this game.
Kinetic Void hasn't seen an update in almost 3 years i think, but it has probably the most similar feel in terms of space exploration and modular ship design. It was also extremely buggy / borderline unplayable, and didn't have the option of connecting players on servers or traveling in direct paths between star systems. Each star system was a zone with finite borders, and you needed a gate to travel between systems/zones.
Minecraft is sort of the gold standard in terms of block games, and it is driven by a custom engine. Part of the necessity was that there was nothing like it before, which i think is part of the same necessity here. Minecraft also deals with dynamic, zoneless environments, which necessitates saving/unloading/reloading chunks of map in a way that you don't see with a static arena, and it has its own internal logic/mechanics for handling environment to environment interaction that can alter the landscape in customizable and unpredictable ways. Sound familiar?
I think for a game like starmade with no clear precursor, you're not going to be able to find a readymade engine that can handle what the game demands, because you can't make an engine for something that doesn't exist. The engine is most likely being made as the game is being made, with alterations and customizations coded as needed to realize design goals.