Ship File Format

    Joined
    Jun 25, 2013
    Messages
    403
    Reaction score
    11
    • Legacy Citizen 2
    • Legacy Citizen
    Does anyone know the ship file format? I had assumed it would be XML, but I see these nasty binary blobs. I don't suppose they are something as simple as GZipped XML?

    [I was thinking of writing a simple "ship multipler" too. That would take a ship design and, say, double all dimensions. With a bit of logic to handle controllers, the result should be functional, but eight times more massive.

    Other ideas too, but they all start with being able to read and write ship files.]
     
    Joined
    Jun 25, 2013
    Messages
    403
    Reaction score
    11
    • Legacy Citizen 2
    • Legacy Citizen
    I\'ve decompiled the source and found where the .smd2 file is written. Looks like it\'s just a Java DataOutputStream dump. Not a lot of help that.

    http://www.ocean-of-storms.com/vote4joe/kq_JAD.java (JAD decompiled)

    http://www.ocean-of-storms.com/vote4joe/kq_JD.java (JD decompiled)
     
    Joined
    Aug 13, 2012
    Messages
    163
    Reaction score
    0
    I was thinking about making an out-of-game ship designer, and being able to convert it to/from readable XML would make that very useful.

    I\'ve done Java before, but never encountered a DataOutputStream, that I can remember. Would the files be easily importable by another Java application?
     
    Joined
    Jun 25, 2013
    Messages
    403
    Reaction score
    11
    • Legacy Citizen 2
    • Legacy Citizen
    It\'s not a barrier like, say, something that encrypted it as it wrote it. A DataOutputStream basically lets you output raw data elements. In this case an array of bytes, then several ints. You can read them back in if you know what to expect. That\'s the catch. It\'s not self describing the way XML or the Minecraft format is. You have to know what you\'re reading.
     
    Joined
    Jul 3, 2013
    Messages
    16
    Reaction score
    0
    • Legacy Citizen 2
    • Legacy Citizen
    package org.schema.game.server.data.blueprintnw; class BlueprintEntry

    ^I assume we all have a decompiler by this point ;) but I\'d recomend takeing a look at that file... remember it\'s a lot eassier to understand a file format via reads than writes...

    So far I have the format to consist of:
    File name: Basicaly random so not much need as ship name is folder name.

    Data: Basicaly An array of \"blocks\" (As expected)

    Each contains:

    ID? type q(3 int\'s)

    Vector3(3 Floats) [Position?]

    a short[Posibly faceing? seems large for this use thou...]

    a byte [HP value?]

    ~~~~Note: These are my best guess\'s from a very limited view and could be way off from the actaul data format. Also I haven\'t looked much at the header, logic or meta yet...
     
    Joined
    Jun 25, 2013
    Messages
    403
    Reaction score
    11
    • Legacy Citizen 2
    • Legacy Citizen
    Yeah, this is a real pain. I started top down and deciphered much of BlueprintEntry. It\'s a bit of a pain. I few things I found:

    the tH class appears to hold the root location of a ship directory. The static fields a_tH_static_fld and b_tH_static_fld pointing to blueprints and blueprints-default. (Not sure which is which.)

    A BlueprintEntry is constructed passing in the ship\'s name, and optionally the tH root.

    I\'ve found the functions to read the header, meta and logic. Header and Logic quickly descend into other classes which I haven\'t looked into. Meta mostly reads an Entity, which I already have code for. I can\'t find where the .smb2 file is read, but from looking at the binary, it seems right.

    Oh, a \"q\" is basically an int vector. Ported that in for the Entity stuff. Probably the location within the ship of the block.



    What decompiler are you using? The class names will be the same, but we\'re not going to be able to compare field names if we\'re using different ones. I switch between JD and JAD.
     
    Joined
    Jun 22, 2013
    Messages
    14
    Reaction score
    0
    • Legacy Citizen 2
    • Legacy Citizen
    Looking into this as well to make some kind of blueprint editor/viewer. So far fouund out these things about the files you were looking into. Although seems all I care about (info about where/what is each block in the ship) is all in the DATA subfolder. Maybe someone can start a documentation somewhere to help (or schema can help us ? :D)

    header.smbph - Seemingly containing a summary of # of blocks used in the blueprint
    int ?? (version?)
    int unknown??
    a pair of 3d float vectors (bounding box/size for the blueprint?)
    int tableSize
    then tableSize # of entries containing:
    short blockId (from ElementKeyMap.class)
    int numBlocks (number of blockId blocks in the ship)

    logic.smbpl - Pretty sure this is related to linking blocks with the controller (e.g. salvage/weapons computer, docking?)
    int ?? (version?)
    int numberOfControllerBlocks (??)
    then numberOfControllerBlock entries:
    3 shorts (48 bit value ?)
    int k ??
    then k entries of:
    short tag
    int numberOfControlledBlocks
    then q vectors for each controlled blocks
     
    Joined
    Jun 25, 2013
    Messages
    403
    Reaction score
    11
    • Legacy Citizen 2
    • Legacy Citizen
    Tembre set up a GitHub site for the modding work. I see that you can have a wiki there. I was thinking of documenting the .ent file format there. I didn\'t see a file format section on any of the other wikis, and there seems to be a lack of clarity over which is the best. So I was thinking of documenting there.
     
    Joined
    Jul 3, 2013
    Messages
    16
    Reaction score
    0
    • Legacy Citizen 2
    • Legacy Citizen
    Sent this to jjaquinta via Msg a couple days back, I haven\'t had time to relook at it again until today.

    ~\"Hi,



    Butchered some of your code to try and read a smd2 data file with only a ship core.

    Sadly this file is 70kb while a 444block ship I had is only 110kb

    Reading the files as byte\'s I recived an alternating pateren of

    -1, -1, -1, -1, 0, 0, 0, 0

    The only difference was in the 4th LAST line where the larger ship had a value of 82

    This might indicate a bug in my code... but it\'s preaty simple to read from a DataInputStream

    FYI seems they use \"1\" to indicate the end of the data file...

    Will work more on this later but it seems that the data file is just a grid :( or I\'m completly missing something)
     
    Joined
    Jun 22, 2013
    Messages
    14
    Reaction score
    0
    • Legacy Citizen 2
    • Legacy Citizen
    It looks like to me that the .smd2 file consist the following:

    * an int (for file version)?
    * a fixed length portion 32,768 (0x8000) bytes long (consist of mostly alternating 0\'s and -1 int\'s, but there are a few other values here as well).
    * Then there is another fixed length portion of 32768 bytes for some purpose, in my blueprint\'s it\'s all 0\'s except for some data in the middle
    * followed by variable length portion that grows in chunks of 5120 bytes (of which you need at least one for a ship with only 1 core block, meaning a min filesize is 70,660 bytes). As to the format the individual chunks, I\'m not sure, it seems each chunk starts with a value 0x0000013F followed by data then 0 padded to the 5120 byte boundary.

    I only checked small ship blueprints (up to around 3000 blocks of mass) so far, and I\'m having trouble finding the code about loading/saving these smd2 files, so this info is based around looking at the datafile itself.
     
    Joined
    Jun 23, 2013
    Messages
    25
    Reaction score
    0
    • Legacy Citizen 2
    • Legacy Citizen
    I would have your babies if this was done. I\'ve been looking for an out-of-game ship editor for a while.
     
    Joined
    Jun 22, 2013
    Messages
    14
    Reaction score
    0
    • Legacy Citizen 2
    • Legacy Citizen
    well I found out some cool things about the format for the variable portion of the smd2 file. Basically with this info I think it\'s enough to at least be able to read a blueprint for the purposes of reconstructing overall look of a ship. As I mentioned in the last post smd2 file contains 5120 byte chunks that follow the beginning of the file which so far looks like fixed size (1 32bit int followed by a pair of 32KB sections). The 5120 byte chunks describe a 16x16x16 block area of the object and have following format:

    64 bit integer - timestamp from System.currentTimeMillis() call when blueprint was created
    q vector (3 32-bit ints) - relative position of this chunk. In the case of a ship the ship core is located in chunk (0,0,0) and each adjacent chunk is at an offset of for example (0,0,16) being the next nearby chunk to the core.
    32-bit int (type?) - seems to be a 0x0001 in most cases
    32-bit bytelen - This is indicates how many data bytes in the remaining part of this chunk. Actually the chunks in the smd2 file are zlib compressed and aligned to 5120 byte boundary. So after the bytelen you will see 0x78 0x9C which is the zlib header followed by the compressed data then the checksum. So it\'s necessary to decompress the data to then retrieve the 16x16x16 block data.

    The 16x16x16 block data is a 12288 byte array after decompression. In other words each block occupy a 3 byte value in the array, having this following format:

    - The lower 3 bits - the orientation of the block (one of 8 faces)
    - The next 8 bits - the HP value of the block
    - the next bit - was the block activated
    - The upper 12 bits - the block ID

    For a ship core will always occupy the (8,8,8) coordinate of chunk (0,0,0).
     
    Joined
    Jun 25, 2013
    Messages
    403
    Reaction score
    11
    • Legacy Citizen 2
    • Legacy Citizen
    Have you got code to read this yet? If you do, consider checking it into the GitHub project.
     
    Joined
    Jun 22, 2013
    Messages
    14
    Reaction score
    0
    • Legacy Citizen 2
    • Legacy Citizen
    Yes, just made some rough code to read through the file and print out stuff to the window, it needs more work to actually be useable in a script/program...



    https://github.com/StarMade/StarMade/pull/2
     
    Joined
    Jun 22, 2013
    Messages
    14
    Reaction score
    0
    • Legacy Citizen 2
    • Legacy Citizen
    Here is a rough example of what I was able to make parsing one of my ship blueprints for a small salvager ship. Using this script to help visually verify tha tthe parsing was being done properly which is mostly true although there is a bulge in the starboard engine which I don\'t recall being there, so it may be a bug in my script possibly happening at a chunk boundary.
     
    Joined
    Jun 22, 2013
    Messages
    14
    Reaction score
    0
    • Legacy Citizen 2
    • Legacy Citizen
    I just added some more info about the file structures in my last commit of an example script to the repo. At this point the logic/header structure is known (meta seems straightforward, but I didn\'t look much into it), along with the chunk part of the datafile. So now I will focus on trying to understand the first part of the datafile, which I think will be needed to make ourselves some kind of blueprint editor :)
     
    Joined
    Jun 25, 2013
    Messages
    403
    Reaction score
    11
    • Legacy Citizen 2
    • Legacy Citizen
    Are you working solely in python? Do you mind if I do a Java port?

    (Once I can work out how to check-in >_
     
    Joined
    Jun 22, 2013
    Messages
    14
    Reaction score
    0
    • Legacy Citizen 2
    • Legacy Citizen
    Do what you want, I wrote it mainly as an example and Python is generally simple to understand

    Definitely you\'ll want to implement in some other language if you want to make something efficient and that most people can download & use

    Edit: For github I think you\'ll either need to get permission to push directly to the repo otherwise you make a fork of it and follow these directions and tambre can approve your commits

    http://media.pragprog.com/titles/tsgit/chap-005-extract.html
    https://help.github.com/articles/using-pull-requests
     
    Joined
    Jun 25, 2013
    Messages
    403
    Reaction score
    11
    • Legacy Citizen 2
    • Legacy Citizen
    Thanks for the tutorials. I think I finally have GitHub working. There\'s a pull request for the work I did tonight.

    So, you know that \"unknown\" first int that\'s always zero? I think I can shed some light on that for you.

    In the entity code, it reads the first two bytes, if they are zero, it reads two more and continues on. That\'s what you are seeing. However, if they are \"PK\" (31, -117), then it assumes the following data is zip compressed. It backs up, wraps the stream in a GZIPInputStream and carries on. I haven\'t found a file yet that does this. I don\'t know if it is a hangover or future planning.

    (If you look in my push for the JoFileLibrary project, jo.sm.ent.logic.EntityLogic class, readFile() method, you can see this at the top.)