- Joined
- Aug 21, 2015
- Messages
- 585
- Reaction score
- 327
IN SHORT:
Alright, so! This idea is for the devs to open up database access to modders. Basically have a version of StarNet that communicates with the StarMade server, performing a SQL query on a selected open database (such as the world file). There is a TON of useful information available, for which there are currently no commands, and this kind of access is a lot, LOT faster than using StarNet to run individual commands. Also the scope of what a SQL query can do is incredibly more capable than the available console commands.
WHY THIS IS NEEDED:
StarMade has such great potential to be modded. I know the devs want to wait for modding till they have stable codebase and all that.. But are the databases really going to change that much? And if they do, would it really be that big of a deal for scripters to update their scripts to reflect these changes? There are players that want to create really cool content for StarMade RIGHT NOW, and these players are quitting their efforts out of frustration and many of them will not be returning later. This is a missed opportunity and a loss to our community as a whole.
Let me tell you a bit about my journey to produce some interesting content for the server I admin on (LvD).
I'm building what I call an "event engine." This will create spontaneous instances around the universe to make it more alive and random. There may be distress calls, more unique pirate attacks, asteroid clusters, and ways to spend cash and faction points.. I spent a lot of time working on the back-end, then finished a script that that checks the star type of a system and also returns the location. This is a necessary back-bone to the whole event engine. These scripts are about as fast as I know how to make them BUT I ran into a giant problem. They aren't nearly fast enough.
The PROBLEM:
When testing a system for a star, I have my script load the sector and then use StarNet.jar to run a "/sector_info" command on the center of the system. It then parses the result. If no star is found, it keeps searching. If a "DOUBLE_STAR" is found, it keeps searching till two stars are found. When a "SUN" is found, it stops and returns the result (same thing with "GIANT" stars). So, the script is not doing any work it doesn't need to do. However if the system has no star, then it HAS TO continue searching till ALL possible locations return "VOID." This is a 6x6x6 grid of sectors in the center of the system (216 sectors in total). So to further reduce the time it takes, it builds a database that it then uses for future checks. It takes a split second to use the database.
The plan here is to use the set of coordinates for the star(s) returned to determine a safe distance from them, effectively mapping out the safe sectors in a system for the event engine to use. This is important because I DON'T want to make my event engine to require players to go out to the middle of nowhere to find anything, like the old larimar asteroids. I also don't want players being lured to an unsafe sector where they burn up in a star! (Unless I do this intentionally in the scripting, of course)
So, I did some testing on the speed of my script on different kinds of systems:
1. Centered SUN/GIANT - ~0.5 seconds (acceptable)
2. Double Star - 8-12 seconds (not acceptable for real-time events)
3. Off-center SUN - 8-12 seconds (not acceptable for real-time events)
4. Void space - 30-36 seconds (definitely NOT acceptable)
5. Previously checked system - less than 0.1 seconds (definitely acceptable for all purposes)
I then created a script that generated a list of all the possible systems in the main galaxy. It would then check the star type for each system, thus building up the database to make future checks lightning fast. I used a big box of the systems by basically drawing a box around the main galaxy. This returned over 40,000 possible systems! I did some rough calculations and determined that would take over 11 days of the script running continuously for it to create this database, lagging the server the whole time! In addition to this, while I was running the script, some players were having trouble connecting to the server because it was getting spammed with loading sectors and then "/sector_info" commands.
I decided it was time to learn a little SQL and see what I could do with loading a copy of the world file as a database, doing direct queiries (since the STAR sectors do not change). Turns out there is an ENORMOUS amount of stuff I can do! It's also MUCH FASTER to query. Here's an example of a query for any star type in a 6x6x6 block of sectors (216 in total). There was no need to optimize the search or anything, I just offload all that to the database and do a brute-force check of the sectors.
This took about a second.
For anyone curious, I went ahead and mapped out what each "type" number meant:
0 = SPACE_STATION
1 = ASTEROID
2 = PLANET
3 = MAIN
4 = SUN
5 = BLACK_HOLE
6 = VOID
7 = LOW_ASTEROID
8 = GIANT
9 = DOUBLE_STAR
So, I then created a script to run the search and I timed it.
Look at that! Such speed, so wow!
So then I created a loop to run the command 100 times in a screen that I detached from.. And here are the results of that:
That is 0.67275 seconds per query. A brute force check with my script takes 30-36 seconds but here it only takes half a second! That's a GIANT DIFFERENCE. And, in addition to that, I don't need to bog the server down at all. AND I can simplify my scripting quite a bit, cutting out 90% of it. I also don't need to make any sort of duplicate database either.. This is an acceptable speed for all intents and purposes.
So what else might the database be used for?
Right now I've only explored the world database, which contains a plethora of information. It has info about players, entities, sectors, fleets, trade history, jumps, and fog of war. Imagine if my script had real-time access to all this information rather than an old copy of the world file.. The possibilities of what I could do with this information are nearly endless.
So, now I wonder, what other kinds of databases does the StarMade have open in active memory? What if I could query bits of data for individual players..like that player's inventory.. or entities.. ect. There are so many things I could do with this information. And if everyone had access.. It would attract a lot of modding activity to StarMade.
How do modders currently access information about the server and why is this not good enough?
Right now the methodology is for the scripters to use a console command. The devs have to program this into the game code and then have it handle all the queries for relevant info. One example is "/player_info". Behind the scenes, the server does all the database queries when it is given this command and then returns the results in a format that is human readable. It includes a TON of information. IP addresses, sector location, faction stuff, ect. Then a script needs to parse all this information, throwing out what it doesn't need. Maybe the script only wants the sector location of that player. This is EXTREMELY inefficient. Allowing direct access would give scripts the capability to only poll for the information they need. And not only that, they can rely on the power of the database to return many results at once instead of having to do several individual queries, with separate instances of StarNet for every single little bit of data the script wants. <-- this is incredibly slow when doing batch processing.
So how might this be implemented?
From what I've read about HSQL, it is a JSQL database. Parts of it remain in memory and it does writes to the actual database on the hard drive periodically. I'm guessing this is part of what an auto-save does. So, create a StarNet.jar type tool which sends a query to the StarMade server, maybe call it "StarNetDB.jar."
example of usage: java -jar StarNetDB.jar select:world sql: SELECT "UID", "X", "Y", "Z", "NAME", "FACTION" FROM "PUBLIC"."ENTITIES" WHERE "X" = 4 AND "Y" = 4 AND "Z" = 4;
Result:
(it would be a command line result, of course to be parseable by scripts, I just used my GUI console to return results easily )
Just imagine what we could do with this. I could create a command that easily removes all pirate entities from that sector. Or maybe a "!repel" command, that moves all entities from all factions but the player's current faction out of the sector. Or maybe this is used to get a list of all entities for a faction within a system. There are all kinds of things we can do here if I had real-time access to the database. The possibilities are only limited by your imagination.
So, what happens on the back end? Well when this command is issued by the "StarNetDB.jar", it would initiate a connection to the StarMade server and send the request along. The server then issues the SQL commands to whatever database is specified and returns the results. I'd recommend to AT FIRST make this access read-only. This would allow modders to become familiar with the database. Then later on, allow these mods to actually change values and add new values in real-time <-- This is where modding would get even MORE interesting.
In Conclusion
Having the ability to do SQL queries to the StarMade databases would be an incredible resource for modders. I cannot over-state the importance that modding has for a sandbox game like StarMade. We need the diversity that the community can provide. No sandbox game will survive long without modding. This is simply a fact of this kind of game.
One of the reasons MineCraft exploded (and kept exploding) the way it did was because of server-side and client-side mods. These kept the game interesting for players for years and years and people are STILL playing it as a result. When MineCraft was being developed.. How long did it take for the devs at Mojang to release an update for MineCraft? It was a mind-numbingly slow process. Much slower than the StarMade development cycle. I remember waiting for months and months just to release horses! HORSES. And in the same time-frame, hundreds of mods were released. If everyone was stuck with vanilla minecraft.. I don't think people would have stuck around for nearly as long. But nobody cared how slow the devs were because they had all this interesting new content from modders to explore.
I give credit to the StarMade devs for all their hard work, but they really gotta start looking at the bigger picture here. They need to determine what ways they can give power to modders to expand on their base gameplay. For every new feature the developers are capable of implementing, modders could have implemented the 20 other features if only they had the right capabilities. Opening up the database to modders would have a minimal impact on the other projects of Schine, but would give such a great amount of access to modders. I just don't see why this already isn't a thing.
So (constructive) thoughts are welcome here. Thank you for your time!
Alright, so! This idea is for the devs to open up database access to modders. Basically have a version of StarNet that communicates with the StarMade server, performing a SQL query on a selected open database (such as the world file). There is a TON of useful information available, for which there are currently no commands, and this kind of access is a lot, LOT faster than using StarNet to run individual commands. Also the scope of what a SQL query can do is incredibly more capable than the available console commands.
WHY THIS IS NEEDED:
StarMade has such great potential to be modded. I know the devs want to wait for modding till they have stable codebase and all that.. But are the databases really going to change that much? And if they do, would it really be that big of a deal for scripters to update their scripts to reflect these changes? There are players that want to create really cool content for StarMade RIGHT NOW, and these players are quitting their efforts out of frustration and many of them will not be returning later. This is a missed opportunity and a loss to our community as a whole.
Let me tell you a bit about my journey to produce some interesting content for the server I admin on (LvD).
I'm building what I call an "event engine." This will create spontaneous instances around the universe to make it more alive and random. There may be distress calls, more unique pirate attacks, asteroid clusters, and ways to spend cash and faction points.. I spent a lot of time working on the back-end, then finished a script that that checks the star type of a system and also returns the location. This is a necessary back-bone to the whole event engine. These scripts are about as fast as I know how to make them BUT I ran into a giant problem. They aren't nearly fast enough.
The PROBLEM:
When testing a system for a star, I have my script load the sector and then use StarNet.jar to run a "/sector_info" command on the center of the system. It then parses the result. If no star is found, it keeps searching. If a "DOUBLE_STAR" is found, it keeps searching till two stars are found. When a "SUN" is found, it stops and returns the result (same thing with "GIANT" stars). So, the script is not doing any work it doesn't need to do. However if the system has no star, then it HAS TO continue searching till ALL possible locations return "VOID." This is a 6x6x6 grid of sectors in the center of the system (216 sectors in total). So to further reduce the time it takes, it builds a database that it then uses for future checks. It takes a split second to use the database.
The plan here is to use the set of coordinates for the star(s) returned to determine a safe distance from them, effectively mapping out the safe sectors in a system for the event engine to use. This is important because I DON'T want to make my event engine to require players to go out to the middle of nowhere to find anything, like the old larimar asteroids. I also don't want players being lured to an unsafe sector where they burn up in a star! (Unless I do this intentionally in the scripting, of course)
So, I did some testing on the speed of my script on different kinds of systems:
1. Centered SUN/GIANT - ~0.5 seconds (acceptable)
2. Double Star - 8-12 seconds (not acceptable for real-time events)
3. Off-center SUN - 8-12 seconds (not acceptable for real-time events)
4. Void space - 30-36 seconds (definitely NOT acceptable)
5. Previously checked system - less than 0.1 seconds (definitely acceptable for all purposes)
I then created a script that generated a list of all the possible systems in the main galaxy. It would then check the star type for each system, thus building up the database to make future checks lightning fast. I used a big box of the systems by basically drawing a box around the main galaxy. This returned over 40,000 possible systems! I did some rough calculations and determined that would take over 11 days of the script running continuously for it to create this database, lagging the server the whole time! In addition to this, while I was running the script, some players were having trouble connecting to the server because it was getting spammed with loading sectors and then "/sector_info" commands.
I decided it was time to learn a little SQL and see what I could do with loading a copy of the world file as a database, doing direct queiries (since the STAR sectors do not change). Turns out there is an ENORMOUS amount of stuff I can do! It's also MUCH FASTER to query. Here's an example of a query for any star type in a 6x6x6 block of sectors (216 in total). There was no need to optimize the search or anything, I just offload all that to the database and do a brute-force check of the sectors.
This took about a second.
For anyone curious, I went ahead and mapped out what each "type" number meant:
0 = SPACE_STATION
1 = ASTEROID
2 = PLANET
3 = MAIN
4 = SUN
5 = BLACK_HOLE
6 = VOID
7 = LOW_ASTEROID
8 = GIANT
9 = DOUBLE_STAR
So, I then created a script to run the search and I timed it.
Look at that! Such speed, so wow!
So then I created a loop to run the command 100 times in a screen that I detached from.. And here are the results of that:
That is 0.67275 seconds per query. A brute force check with my script takes 30-36 seconds but here it only takes half a second! That's a GIANT DIFFERENCE. And, in addition to that, I don't need to bog the server down at all. AND I can simplify my scripting quite a bit, cutting out 90% of it. I also don't need to make any sort of duplicate database either.. This is an acceptable speed for all intents and purposes.
So what else might the database be used for?
Right now I've only explored the world database, which contains a plethora of information. It has info about players, entities, sectors, fleets, trade history, jumps, and fog of war. Imagine if my script had real-time access to all this information rather than an old copy of the world file.. The possibilities of what I could do with this information are nearly endless.
So, now I wonder, what other kinds of databases does the StarMade have open in active memory? What if I could query bits of data for individual players..like that player's inventory.. or entities.. ect. There are so many things I could do with this information. And if everyone had access.. It would attract a lot of modding activity to StarMade.
How do modders currently access information about the server and why is this not good enough?
Right now the methodology is for the scripters to use a console command. The devs have to program this into the game code and then have it handle all the queries for relevant info. One example is "/player_info". Behind the scenes, the server does all the database queries when it is given this command and then returns the results in a format that is human readable. It includes a TON of information. IP addresses, sector location, faction stuff, ect. Then a script needs to parse all this information, throwing out what it doesn't need. Maybe the script only wants the sector location of that player. This is EXTREMELY inefficient. Allowing direct access would give scripts the capability to only poll for the information they need. And not only that, they can rely on the power of the database to return many results at once instead of having to do several individual queries, with separate instances of StarNet for every single little bit of data the script wants. <-- this is incredibly slow when doing batch processing.
So how might this be implemented?
From what I've read about HSQL, it is a JSQL database. Parts of it remain in memory and it does writes to the actual database on the hard drive periodically. I'm guessing this is part of what an auto-save does. So, create a StarNet.jar type tool which sends a query to the StarMade server, maybe call it "StarNetDB.jar."
example of usage: java -jar StarNetDB.jar select:world sql: SELECT "UID", "X", "Y", "Z", "NAME", "FACTION" FROM "PUBLIC"."ENTITIES" WHERE "X" = 4 AND "Y" = 4 AND "Z" = 4;
Result:
(it would be a command line result, of course to be parseable by scripts, I just used my GUI console to return results easily )
Just imagine what we could do with this. I could create a command that easily removes all pirate entities from that sector. Or maybe a "!repel" command, that moves all entities from all factions but the player's current faction out of the sector. Or maybe this is used to get a list of all entities for a faction within a system. There are all kinds of things we can do here if I had real-time access to the database. The possibilities are only limited by your imagination.
So, what happens on the back end? Well when this command is issued by the "StarNetDB.jar", it would initiate a connection to the StarMade server and send the request along. The server then issues the SQL commands to whatever database is specified and returns the results. I'd recommend to AT FIRST make this access read-only. This would allow modders to become familiar with the database. Then later on, allow these mods to actually change values and add new values in real-time <-- This is where modding would get even MORE interesting.
In Conclusion
Having the ability to do SQL queries to the StarMade databases would be an incredible resource for modders. I cannot over-state the importance that modding has for a sandbox game like StarMade. We need the diversity that the community can provide. No sandbox game will survive long without modding. This is simply a fact of this kind of game.
One of the reasons MineCraft exploded (and kept exploding) the way it did was because of server-side and client-side mods. These kept the game interesting for players for years and years and people are STILL playing it as a result. When MineCraft was being developed.. How long did it take for the devs at Mojang to release an update for MineCraft? It was a mind-numbingly slow process. Much slower than the StarMade development cycle. I remember waiting for months and months just to release horses! HORSES. And in the same time-frame, hundreds of mods were released. If everyone was stuck with vanilla minecraft.. I don't think people would have stuck around for nearly as long. But nobody cared how slow the devs were because they had all this interesting new content from modders to explore.
I give credit to the StarMade devs for all their hard work, but they really gotta start looking at the bigger picture here. They need to determine what ways they can give power to modders to expand on their base gameplay. For every new feature the developers are capable of implementing, modders could have implemented the 20 other features if only they had the right capabilities. Opening up the database to modders would have a minimal impact on the other projects of Schine, but would give such a great amount of access to modders. I just don't see why this already isn't a thing.
So (constructive) thoughts are welcome here. Thank you for your time!
Last edited:
