Obfustication, why?

[jjaquinta]

With all that experience, Kal, I\'m sure you are quite aware that there are many ways to be open without becoming a \"patchwork quilt\". At no point have I suggested that StarMade be made open source. I specifically call that out in my original post. I think that would be completley inappropriate for a project like this.

All I\'ve been suggestion was that there is no value in obfusticating the code. Ceasing to obfusticate it does not make it open source. It just makes it readable. The model I pointed to was the client for Second Life by Linden Labs. Their source code is available, but they retain all rights to it and people can only propose changes to it. They cannot check into the code base. Discovered bugs and exploits are usually quickly accepted. Feature changes are taken more slowly, depending on developer load. It also saves them a lot of time answering questions about how it works for people modifying it.

It is a model that I think makes a lot of sense here and would benefit the developers and the community.

 

[KalEl]

Hi jjaquinta!

I\'m sorry if you misunderstood my post. I don\'t think I stated that you wanted to make Starmade open source, just merely referenced that I had worked within a similar environment.

If you then continue to read the points I made afterwards you will find that they still hold true with what you are proposing.

Being a fellow programmer you probably understand that a bigger team would be needed to avoid hitting a decrease in development speed. If they had a team the size of early Mojang even then a person could be diverted to managing submissions and that aspect of the community in general.

However, do you develop games? I think you should, your ideas are exciting and I would love to see a community driven development / benevolent ruler setup - would even contribute myself!

Best,

Kal

 

[jjaquinta]

Hey Kal,

I\'m afraid I do not see how your points hold. Ceasing to obfusticate the source shouldn\'t make a difference at all to the developers. If they chose to take suggestions from the community, clearly it would only be if they were of value. If they were of value then it means that the time they save by adopting fixes from the community would be greater than the time spent reviewing them. If the community has some sort of self-review before passing them on, that will help keep the quality of suggestions up. Overall I would expect the pace of development to increase. We all want that.

Professionally, I\'m a software engineer. It\'s a different philosophy from a programmer. But I think for my non-prefessional stuff it\'s fair to say I mostly approach it like a programmer. (Although I kid myself otherwise!) I\'ve done games, but nothing really worth writing home about. I\'ve been doing a lot of Android work recently. Most recently I was trying to do the Android port of Traveller-AR. That died and I\'d been looking to use some of the near-earth stellar data from my solstation project in a modular edutainment game. I\'ve got a baseline which is an interesting mix of known data and procedural generation for stellar data. I.e. known stars come up with what we know about them, what we don\'t know gets filled in via generation.

The last thing is what lead me to StarMade. I was curious if it could be driven by the baseline I made. I\'m less sure now. The actual stars don\'t figure that highly in the game and the worlds are too plentiful (and 2-dimensional) to really meaningfully align with real-world data or systems.

That would need a modding structure set up. A more recent idea was a \"ship magnifier\", which I posted about in this thread. That\'s an example where ceasing to obfusticate the source would be of great benefit to people. Instead of waiting for them to find the time to document the ship file format, we could do it ourselves. And they aren\'t likely to change it, given that they are encouraging people to create ships all over.

Last night I got an idea and started a Mural Designer. Basically, you give it an image, and it tells you how to draw it with hull blocks. The idea is to use some classic dithering methods to deal with the limited palette available to do the best job. I got a barely functional Alpha working before I hit the sack (threshold only). If you are interested, post in this thread and I\'ll pass it on. (Works for wool murals in Minecraft and lego too!)

But as far as this thread goes... I\'d like to see the devs cease to obfusticate the code, as I think it would benefit everyone. If we need to set up a user community to convince them that it can be managed in a way sensitive to their time, then we should do that. I\'d rather not run it. (Two small kids at home, I don\'t have quality time to do a good job!) But I\'d happily participate.

Jo

 

[jjaquinta]

So after about three hours of playing I\'ve got code to read entity files. Below is a dump of the PlayerState file. Almost got write to work, but not quite.

So the only things obfusticating the code has achieved is wasting a few hours of my life.

Shrug.

PlayerState (STRUCT) = {
  credits (INT) = 376472
  spawn (VECTOR3f) = (0.0, -6.5, -8.0)
  inventory (STRUCT) = {
    slots (LIST) = [
      (INT) = 3
      (INT) = 6
      (INT) = 4
      (INT) = 0
    ]
    types (LIST) = [
      (SHORT) = 78
      (SHORT) = 2
      (SHORT) = 77
      (SHORT) = 75
    ]
    values (LIST) = [
      (INT) = 6
      (INT) = 143
      (INT) = 2
      (INT) = 1
    ]
  }
  sector (VECTOR3i) = (2, 2, 2)
  lspawn (VECTOR3f) = (8.00286, 55.510994, 174.78363)
  lsector (VECTOR3i) = (10, -5, 7)
  pFac-v0 (STRUCT) = {
    null (INT) = 0
  }
  null (LONG) = 1372465329229
  null (LONG) = 1372470226541
  ips (STRUCT) = {
    null (STRING) = /127.0.0.1
  }
}

 

[tambre]

Never seen such Java code...Well what I mean is that it doesn\'t seem like Java to me...But yeah obfuscation will just waste our time...Minecraft - Notch promised modding supports 1.7.3. Next promise was 1.3 from Jeb, then it was 1.5...Well don\'t know if it\'s gonna be next update even :S

 

[jjaquinta]

Sorry for the confusion. That\'s the output of my program, not the code of the program. I guess I probably should have dumped XML or something.

It\'s nearly lunchtime. I\'ll tickle it to dump XML, stick it on a web site and post a link. Write isn\'t working yet, but people can start messing with it to see what files it can and cannot read.

 

[Charlie_]

Glad to see you\'ve actually taken up arms and tackled your objective yourself (to put it dramatically).

It kind of amuses me though how you say it only wasted a few hours of your life (the de-obfs-er. Let\'s call it that), after all this time spent on the forum. It seems you could\'ve had it done and finished earlier.. But eh. On one note, I don\'t think any time spent exchanging opinions and points with someone is wasted time, unless it\'s not beyond the \" \"wat ut tink?\" \"luks good. gg.\" \"yeh. bit short doe\" \"yh\" \"unf unf\" \".

..And now I have Wasted Years playing in my head.
Also, what site did you have in mind to dump these?

 

[jjaquinta]

Ha ha, Charlie. Not as much time as you think. I type fast.

Note: the job is not complete or well done. Right now the \"edit\" util corrupts any file it touches. I got bugs to fix. And, this doesn\'t even get close to supporting mods. (Although technically I\'ve got the info now to do a \"mod\" that would change the .ent file format to xml.)

The source code I\'ve done is included in my jar. But it\'s still pretty unlikely that many will be able to build upon my work. Also it is nigh impossible for me to constructively comment on the dev\'s code. It\'s been through too much tourture for me to really be able to definitively spot any but the largest of logical bugs. If the devs stopped obfusticating their code, we would all benefit.

The code is posted here. There\'s a .txt file in the zip with blah-blah-blah and instructions on how to use.

I have no problems with people using this, or creating derived works. I have my own source control at home, but that\'s probably not a good solution for a community project. I have enough \"recreational programming\" time to contribute, but it wouldn\'t be fair for me to try to run a community. If someone wants to set up a Source Forge or GitHub repository, I\'m happy to post the stuff there.

I looked for a spot on the wikis to start documenting the file format, but didn\'t see anywhere obvious. I expect someone, someday will want to be abel to read/write these files in another language. Documenting it now, while fresh in my mind, will make it easier for them. If someone can find a spot I can start entering the data.

 

[Charlie_]

I wouldn\'t have the arrogance of complaining about an incomplete or not best-made tool when I myself haven\'t made any efforts to accomplish anything. If anyone complains about using a tool someone else made and is distributing free of charges, then they\'re complete dumbstrucks to me.

And the wikis are pretty much in their infacy right now. I just went and did a page myself the other day. That one\'s still only a start. Will check for the documenting page, though.

Gives me something interesting to read/do during lunch.

 

[Uristqwerty]

One very useful feature of obfuscation is to clearly define the boundary between somewhat stable API, and the parts of the code that may change at any time.

Almost every Minecraft mod that directly interacts with Mojang code needs to be updated to work with each new Minecraft version (sometimes mods continue to work with bugfix releases, but just about any time they add new content, every mod breaks). Once they finally have enough of an API to release it, I expect that their API will be the only part that isn\'t obfuscated, since it\'s the only part that people can rely on after the next update.

Especially important to consider is how Minecraft 1.3 completely changed the internal structure, and many mods took months to finally adjust, while many others simply never updated. Something like that is just about guaranteed to happen once in a while in any large game.



Encapsulation can be a good thing, because it defines what part of a class is intended for public use, while the implementation can change at any time. Obfuscation can be used the same way on the entire game codebase.

Also, on data formats: If there is any magic number or version information, then they can always decide to use a new format, only writing an importer for the old ones. This is a problem for any tool that you want to continue to use on later game versions.

 

[tambre]

Well I like updating! Don\'t wanna update your mod? Move to Minecraftforge what from 1.4 started including runtime deobfuscation. What means if your mod doesn\'t use any deprecated stuff it will work on 1.5. Forge needs to be updated what normally takes 1day. If we, the community are able to estabilish such API for modding then I will like it.

About other stuff, I just today got tablet, gonna learn to code for it tomorrow. Will wait for new Starmade and create Github repository with newest code and hope contributions to there. Also will try to port Starmade to Android in future after Github repository has progressed a little. (Should be pretty straight forward port considering Android is written in Java...)

 

[jjaquinta]

Obfustication is a very heavy handed way to do encapsulation. No sane person would want to go the Minecraft route where mods are deployed as changes to core classes. Rather you want to build in extensions into the code base so additions can be explicitly managed. Doing so also explicitly defines what and how to extend things.

So, assuming they plan to adopt an extension model down the road, obfustication does not help. With clear code, people can be reading and learning the code base now. People who are happy to have their code blown out of the water might start making minecraft style core-source hacks now. But they get what they deserve. The rest can build their skills, and even contribute towards the construction of those extension interfaces.

As for their data format, they aren\'t currently including a version #. I think that\'s a bit short sighted. However, their code does peak at the first two bytes of the file. If it\'s a PK then it assums it is a GZIP stream and wraps it accordingly. Otherwise they assume it is their binary format.

I had thoughts of hijacking this. I think an XML format is much more sensible. If I wanted to do things the Minecraft way, I\'d re-code their eF.class file so that it wrote XML, and on read, if the first two bytes were

 

[jjaquinta]

I\'m guessing you\'ve never written an Android app before? Or were you being sarcastic.

User interface elements aside, Android\'s OpenGL is... quite different, from the libraries that StarMade are using. OpenGL is a \"standard\" in a very loose sense. I\'ve been trying to write a few apps that use rich graphics that interoperate between Desktop and Android, but the \"common denominator\" between them is so small that it\'s hard to avoid a re-write.

I think the bigger problem is that a very large part of the code is in the client rather than the server. That makes things extrordinarily difficut to port to different systems. That\'s part of the reason why Pocket Minecraft will probably never be able to interplay with the normal version. (Much to my daughter\'s dismay.)

One of my first thoughts in looking at StarMade was of doing an Android port. But if you need to re-write the client side, then you need to know the line protocol (bits and bytes that go up and down the network). Just like the file formats, it\'s a long painful process to work that format out when the code is obfusticated.

 

[Charlie_]

Here\'s an intermede to discussion.
I imagine Schema reading this thread, all laid back into his chair and sipping something like a beer with the strongest and most disinterested possible expression he can manage.
Of course, that\'s all my imagination, but the thought amuses me. Especially because of that cat avatar.

 

[WiskeyEcho]

Been focused on the blueprint .smd2 format myself (If anyone is hiding an editor/conveter hook us up plox Currently my goals are:

1. Converting smd2 to an open file format for piping into pre existing editors/converters.

2. ????

3. Perhaps creating a stand alone editor that allows for copy/paste, undo/redo etc

^Yeah I know I did that meme wrong...

 

[jjaquinta]

@Charlie. If schema is the professional I think he is, I\'m guessing he would see the merits in my arguments. I think the obfustication thing is a knee-jerk reaction, rather than one that was thought out.

@WiskeyEcho. I looked at the ship file code first. It went into a maze pretty quickly. I posted the decompiled sources in another thread. But now that I know more, I should look at it again. I just updated the tool (finally got write to work). Files that I\'ve been able to dump:

ENTITY_PLAYERSTATE

ENTITY_PLAYERCHARACTER

FACTIONS.fac

SIMULATION_STATE.sim

CATALOG.cat

I\'m having problems with ENTITY_SHOP and ENTITY_SPACESTATION. But I think that\'s bugs in how I parse SERIALIZED data elements. Those elements aren\'t in the other files.

 

[Verox]

Consoles games aren\'t sofwtare, they\'re hardware.

There are so many things wrong with that statement I can\'t even.

 

[tambre]

About that I finally got up nice StarMade resources, missing server though. (and crashreport?) Still need to add licenses and create a proper readme what I will do soon.

Link: https://github.com/StarMade/StarMade

 

[jjaquinta]

Good work, tambre. I found a StackOverflow thread here on deobfustication. JDO seems to be the tool of choice. Someone further down the commentary has a patch for it that allows it to handle case sensitivity, which we\'ll need. If the patch hasn\'t been incorporated, you\'ll probably need to download the source for JDO, apply the patch, and recompile. If we can automate that process, I would suggest startgin to do weekly imports. Are you up for that? What can I help with?

I\'m updating my Eclipse right now to include Git. I\'ll push the source I\'ve massaged in as soon as I have that going and everyone can have a look/fix my bugs/etc.

If we are to give up on the devs ever publishing code in the clear, the next step would be to create a reversible process. I.e. that we can deobfusticate and decompile the code into a legible form, AND we can recompile and obfusticate it back into the exact same unintelligible form. Along the way we\'ll want to ability to give better names to the deobfustication process, and have it remember them.

This is what I think they did for Minecraft. If they\'re willing to share their tools, that would be good. Otherwise we\'ll have to waste even more time replicating them.

 

[tambre]

They pretty much just search for example field_7234 in client.csv (or something like that) for something like: field_7234 timeFromLastHit The time that the entity last received hit.

Bad example. but will do. I currently used JDGui, what was more recently update. I think we should use that not JDO. What you think about that?

Will push server sources now and update readme.

EDIT: Just realized StarMade.jar includes server code too...