(UPDATE 2 - 2/14/2016) Coding an external ship editor

[Splamy]

I've stumbled into the same(?) problem, that the header file contains some data blob after the block entry array.
I've done some decompiling and it seems like they are serializing something in there

Spoiler

    Object localObject1 = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(this.jdField_a_of_type_JavaIoFile)));
    Object localObject4 = localObject1;Object localObject3 = paramSegmentController;Object localObject2 = this;((DataOutputStream)localObject4).writeInt(1);((DataOutputStream)localObject4).writeInt(apX.a(localObject3.getClass()).ordinal());Vector3i localVector3i1 = new Vector3i(((SegmentController)localObject3).getBoundingBox().jdField_a_of_type_JavaxVecmathVector3f);Vector3i localVector3i2 = new Vector3i(((SegmentController)localObject3).getBoundingBox().jdField_b_of_type_JavaxVecmathVector3f);((DataOutputStream)localObject4).writeFloat(localVector3i1.jdField_a_of_type_Int);((DataOutputStream)localObject4).writeFloat(localVector3i1.b);((DataOutputStream)localObject4).writeFloat(localVector3i1.c);((DataOutputStream)localObject4).writeFloat(localVector3i2.jdField_a_of_type_Int);((DataOutputStream)localObject4).writeFloat(localVector3i2.b);((DataOutputStream)localObject4).writeFloat(localVector3i2.c);((SegmentController)localObject3).getElementClassCountMap().a((DataOutputStream)localObject4);
    if (((localObject3 instanceof Td)) && ((localObject3 = ((Td)localObject3).a()).getStatisticsManager() != null)) {
      try
      {
        ((apS)localObject2).jdField_a_of_type_OrgSchemaGameCommonControllerElementsEntityIndexScore = ((ManagerContainer)localObject3).getStatisticsManager().calculateIndex();
      }
      catch (Exception localException)
      {
        localException;
      }
    }
    ((DataOutputStream)localObject4).writeBoolean(((apS)localObject2).jdField_a_of_type_OrgSchemaGameCommonControllerElementsEntityIndexScore != null);
    if (((apS)localObject2).jdField_a_of_type_OrgSchemaGameCommonControllerElementsEntityIndexScore != null) {
      ((apS)localObject2).jdField_a_of_type_OrgSchemaGameCommonControllerElementsEntityIndexScore.serialize((DataOutputStream)localObject4, true);
    }

In one of the last lines they write a boolean (((DataOutputStream)localObject4).writeBoolean) wether or not the blob data is present. So you might try to add a 0 instead of 1 after it...
If there is any interest i can dig deeper into it; for now i'll do a few experiments before continuing...

 

[Cheapskate]

I've stumbled into the same(?) problem, that the header file contains some data blob after the block entry array.
I've done some decompiling and it seems like they are serializing something in there

Spoiler

    Object localObject1 = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(this.jdField_a_of_type_JavaIoFile)));
    Object localObject4 = localObject1;Object localObject3 = paramSegmentController;Object localObject2 = this;((DataOutputStream)localObject4).writeInt(1);((DataOutputStream)localObject4).writeInt(apX.a(localObject3.getClass()).ordinal());Vector3i localVector3i1 = new Vector3i(((SegmentController)localObject3).getBoundingBox().jdField_a_of_type_JavaxVecmathVector3f);Vector3i localVector3i2 = new Vector3i(((SegmentController)localObject3).getBoundingBox().jdField_b_of_type_JavaxVecmathVector3f);((DataOutputStream)localObject4).writeFloat(localVector3i1.jdField_a_of_type_Int);((DataOutputStream)localObject4).writeFloat(localVector3i1.b);((DataOutputStream)localObject4).writeFloat(localVector3i1.c);((DataOutputStream)localObject4).writeFloat(localVector3i2.jdField_a_of_type_Int);((DataOutputStream)localObject4).writeFloat(localVector3i2.b);((DataOutputStream)localObject4).writeFloat(localVector3i2.c);((SegmentController)localObject3).getElementClassCountMap().a((DataOutputStream)localObject4);
    if (((localObject3 instanceof Td)) && ((localObject3 = ((Td)localObject3).a()).getStatisticsManager() != null)) {
      try
      {
        ((apS)localObject2).jdField_a_of_type_OrgSchemaGameCommonControllerElementsEntityIndexScore = ((ManagerContainer)localObject3).getStatisticsManager().calculateIndex();
      }
      catch (Exception localException)
      {
        localException;
      }
    }
    ((DataOutputStream)localObject4).writeBoolean(((apS)localObject2).jdField_a_of_type_OrgSchemaGameCommonControllerElementsEntityIndexScore != null);
    if (((apS)localObject2).jdField_a_of_type_OrgSchemaGameCommonControllerElementsEntityIndexScore != null) {
      ((apS)localObject2).jdField_a_of_type_OrgSchemaGameCommonControllerElementsEntityIndexScore.serialize((DataOutputStream)localObject4, true);
    }

In one of the last lines they write a boolean (((DataOutputStream)localObject4).writeBoolean) wether or not the blob data is present.
If there is any interest i can dig deeper into it; for now i'll do a few experiments before continuing...

I am interested! I want my editor to be as accurate as possible. However, seeing as it works for the time being, deciphering the data blob is not really a high priority. If you're already doing that I'd love to see what you find, but don't break your back over it if you have other more important things to do.

 

[Splamy]

I am interested! I want my editor to be as accurate as possible. However, seeing as it works for the time being, deciphering the data blob is not really a high priority. If you're already doing that I'd love to see what you find, but don't break your back over it if you have other more important things to do.

As soon as i have the time, I'll will see what can be found.
Btw, as i'm writing my own file interpreter right now, it seems like we are doing this work twice.
Is there maybe a way we could combine our effords? Iirc you are writing in GML, unfortunately I'm not very familiar with that language. My own project is in C++ right now. If there's a way maybe we could team up on github or sthm like that.

 

[Cheapskate]

As soon as i have the time, I'll will see what can be found.
Btw, as i'm writing my own file interpreter right now, it seems like we are doing this work twice.
Is there maybe a way we could combine our effords? Iirc you are writing in GML, unfortunately I'm not very familiar with that language. My own project is in C++ right now. If there's a way maybe we could team up on github or sthm like that.

This could work! GML has a very similar syntax to C++ and I actually want to learn it one day. There are, however, quite a few differences with commands, functionality, etc. I don't think we would be able to swap code, but I'd like the opportunity to help and be helped by someone else. At the very least we could swap pseudocode and code theories. There is, however, a slight problem. I took up this project about a week ago because I became sick and had to take off work. With wellness fast approaching, I'm going to be going back to work shortly! D: I won't be able to consistently provide time to the coding or the github. [I'm REALLY hoping to at least get the converter part of my editor done before fleet controls become a thing] Though, after I figure out all the file formatting and whatnot I did want to write a layman's guide to current StarMade file formats. That way, anyone from any coding language would be able to read, understand, and build their own program.

 

[Megacrafter127]

As soon as i have the time, I'll will see what can be found.
Btw, as i'm writing my own file interpreter right now, it seems like we are doing this work twice.
Is there maybe a way we could combine our effords? Iirc you are writing in GML, unfortunately I'm not very familiar with that language. My own project is in C++ right now. If there's a way maybe we could team up on github or sthm like that.

Coincidentally, I am also doing the same, and am also already done with an SMD2 reader.

 

[Cheapskate]

Coincidentally, I am also doing the same, and am also already done with an SMD2 reader.

Awesome! Any chance I can see it?

 

[Megacrafter127]

Awesome! Any chance I can see it?

Once I upload it to github and get the cross-compiler set up for every architecture I can think of.

 

[Megacrafter127]

Preliminary repository: GitHub - Megacrafter127/C-StarMadeFileParser: C++ library for reading StarMade's blueprint files
Code isn't tested yet, and may not compile on all systems[namely not on systems without <arpa/inet.h> or <zlib.h>]

 

[Cheapskate]

Preliminary repository: GitHub - Megacrafter127/C-StarMadeFileParser: C++ library for reading StarMade's blueprint files
Code isn't tested yet, and may not compile on all systems[namely not on systems without <arpa/inet.h> or <zlib.h>]

aah thank you very much. I don't need to compile the code. If I can just take a look at it, it will probably help me to better understand it's applications in my own language.

 

[Splamy]

I am interested! I want my editor to be as accurate as possible. However, seeing as it works for the time being, deciphering the data blob is not really a high priority. If you're already doing that I'd love to see what you find, but don't break your back over it if you have other more important things to do.

Ok, I got it, it wasn't actually that difficult.
The few extra bytes were as I suspected ship statistics. The numbers seem to match mostly, although they somehow have some multiplication condition by x1000. Sometimes the stored value is already correct, sometimes it has to be divided, by 1000 - don't ask me xD - I haven't found anything in the source.

Spoiler

(here all values seem to be x1000, but this is just coincidence)

Something like this could be the new .bt file. If everyone agrees, we can update the wiki entry with it.

Spoiler

//--------------------------------------
//--- 010 Editor v6.0 Binary Template
//
// File: smbph.bt
// Author: Cliff Hudson
// Revision: 0.1867
// Purpose: StarMade Blueprint Header file
//--------------------------------------

struct Vector3f
{
float x;
float y;
float z;
};

enum <unsigned int> EntityType
{
Ship = 0,
Shop = 1,
SpaceStatiion = 2,
Asteroid = 3,
Planet = 4
};

typedef struct
{
Vector3f min;
Vector3f max;
} BoundingBox;

typedef struct
{
short blockId;
int blockCount;
} Element;

typedef struct
{
int numElements;
Element elements[numElements];
} ElementCountMap;

typedef struct
{
int version;
EntityType entityType;
BoundingBox boundingBox;
ElementCountMap elementCountMap;
} Header;

Header header;

//--------------------------------------
// *Addendum*
// Author: Splamy
// Revision: 0.19556
//--------------------------------------

typedef struct
{
byte hasStatistics;
short version;
double offensiveIndex;
double defensiveIndex;
double powerIndex;
double mobilityIndex;
double dangerIndex;
double survivabilityIndex;
double offensiveIndex;
double supportIndex;
} Statistics;

if( !FEof() )
Statistics stats; // Optional: only if 'hasStatistics' is present and is '1'

- Oh, and before I forget, no, this is not my mistake that I added the 'offensiveIndex' twice, I think the devs failed that.

 

[navy3000]

can you rotate the core in a blueprint using this editer?

 

[Cheapskate]

Ok, I got it, it wasn't actually that difficult.
The few extra bytes were as I suspected ship statistics. The numbers seem to match mostly, although they somehow have some multiplication condition by x1000. Sometimes the stored value is already correct, sometimes it has to be divided, by 1000 - don't ask me xD - I haven't found anything in the source.

Spoiler

(here all values seem to be x1000, but this is just coincidence)

Something like this could be the new .bt file. If everyone agrees, we can update the wiki entry with it.

Spoiler

//--------------------------------------
//--- 010 Editor v6.0 Binary Template
//
// File: smbph.bt
// Author: Cliff Hudson
// Revision: 0.1867
// Purpose: StarMade Blueprint Header file
//--------------------------------------

struct Vector3f
{
float x;
float y;
float z;
};

enum <unsigned int> EntityType
{
Ship = 0,
Shop = 1,
SpaceStatiion = 2,
Asteroid = 3,
Planet = 4
};

typedef struct
{
Vector3f min;
Vector3f max;
} BoundingBox;

typedef struct
{
short blockId;
int blockCount;
} Element;

typedef struct
{
int numElements;
Element elements[numElements];
} ElementCountMap;

typedef struct
{
int version;
EntityType entityType;
BoundingBox boundingBox;
ElementCountMap elementCountMap;
} Header;

Header header;

//--------------------------------------
// *Addendum*
// Author: Splamy
// Revision: 0.19556
//--------------------------------------

typedef struct
{
byte hasStatistics;
short version;
double offensiveIndex;
double defensiveIndex;
double powerIndex;
double mobilityIndex;
double dangerIndex;
double survivabilityIndex;
double offensiveIndex;
double supportIndex;
} Statistics;

if( !FEof() )
Statistics stats; // Optional: only if 'hasStatistics' is present and is '1'

- Oh, and before I forget, no, this is not my mistake that I added the 'offensiveIndex' twice, I think the devs failed that.

Awesome! Thanks for getting that straightened out.

can you rotate the core in a blueprint using this editer?

Yep! But probably not initially. At first launch, the program will really only be able to export ship to SM. I need to add a lot more code and features before I can successfully import pre-constructed ships. At the current time, if you were to try to import a pre-constructed ship, you would loose all logic connections and system-module connections.

 

[navy3000]

so whats the progress?

 

[RabidBat]

Cheapskate
Just wondering if this is still alive. Seems to have been no updates in a very very long time.

 

[Cheapskate]

Cheapskate
Just wondering if this is still alive. Seems to have been no updates in a very very long time.

Wow, topic back from the dead!

It is still alive, just on hold... until I can learn Java. I hope to at the very least code a translator in Java that will compress the ship files into their proper format, acting as a bridge between my program and Starmade. The coder I use doesn't have much in the way of compression/decompression I'm afraid ._ .

If I can get my hands on that, coding everything else really won't be that much of a problem. I'm aiming to have my program basically output text files in a very simple format for conversion into SM's file type. Though, I think if I can write this converter/translator, it would open up a realm of possibilities for other people to also make their own unique ship editing software in their own coding languages.

So, all in all, it's a long, slow process... for which I apologize. Once I get a handle on Java and make the converter, things will suddenly go a lot faster.

 

[RabidBat]

Thanks for replying, just thought I would check and not assume, SMedit has a habit of not being updated except once a year if we're lucky so I don't like to jump the gun and assume long time = dead.

SMedit is great for making a basic hull from a model and little else these days sadly, and I for one am the kind of person who could make a ship much faster using your method than in game.

Good luck and hope you manage to work it out. I think this is something the community would embrace wholeheartedly if you can pull it off and actually needs it.

And yeah, bloody Java. 'nuff said lol

 

[Cheapskate]

Thanks for replying, just thought I would check and not assume, SMedit has a habit of not being updated except once a year if we're lucky so I don't like to jump the gun and assume long time = dead.

SMedit is great for making a basic hull from a model and little else these days sadly, and I for one am the kind of person who could make a ship much faster using your method than in game.

Good luck and hope you manage to work it out. I think this is something the community would embrace wholeheartedly if you can pull it off and actually needs it.

And yeah, bloody Java. 'nuff said lol

Yeah I know know how that is. I'm still waiting for certain Minecraft mods to update even after they've been silent for years {I'm looking at you, Millénaire.} It's always better to check. I haven't been posting, but I've still been scheming up some potential updates for future versions... once I get over the java hump. Updates such as better optimizations and features for 3D mode, what I like to call the Curve system, automatic ship performance calculators that work based on your specific server settings, and easy to read weak spot detectors to find out where you should add more armor to defend vital areas and systems. I've been keeping a file on all the things I'd like to do with my editor in the future after the initial release.

Me too! I originally started this program because the in-game editor was... well, not up-to-par. It is good for precision building, but not precision building at speed. I felt the best way to do megabuilds (or even just regular builds for that matter) would be to have a system like this. I just recently started a new universe with the faction update, and all the factions overrun the universe before I even finished my first cruiser design! Recent updates with SMedit have also taken away the ability to convert models... so even that won't do anymore.

(P.S. In case you're curious, The Curve system is basically a subdivisable polygon that will try to smoothly fill in blocks across it's surface. This will allow people to very easily create curved hull segments. It will be mirror-able and strive to be symmetrically accurate. I know that was a big problem with SMedit in the past.)

 

[Malacodor]

Did you notice that the blueprint format was changed from smd2 to smd3?

 

[Cheapskate]

Did you notice that the blueprint format was changed from smd2 to smd3?

I did, unfortunately. I noticed it a little while ago. Do you know if there's any documentation on the new format anywhere? If you could post links that would be appreciated.

Also, was the SMD3 created just because they upped the size of each chunk?

 

[Malacodor]

Do you know if there's any documentation on the new format anywhere?

Not sure, but I guess not.

Also, was the SMD3 created just because they upped the size of each chunk?

I don't know.