StarMade Prebuild 0.09425: Too many open files

[schema]

Hello players,

found a big source of bugs, and I have to do another prebuild to test the fix.

http://files.star-made.org/build/pre/starmade-build_20131015_090732.zip

http://files.star-made.org/build/pre/starmade-build_20131015_152625.zip (fixed a long existing bug that caused the game stop displaying new segments)

http://files.star-made.org/build/pre/starmade-build_20131015_162228.zip (fixed GUI bug where in some resolutions the "click-area" would be incorrect. Apparently LWJGL Display resolutions aren't always pixel perfect)

(as always "dev" is pass/user. It will be installed on the test server play.star-made.org:4242. Other admins, please use prebuilds with care as they can contain gamebreaking bugs)

Description: There was a big bug in the file manager, as well as running conditions (unsafe operations when more then one thread has access to the same data). In starmade, a File manager is used to keep a certain amount of files open, so access can be as fast as possible, and each write/read operation doesn't have to open a file again (which causes big overhead). With multiple threads trying to access the open file, there are several unsave operations:

• on the top level the simple read/write operations. A thread may not read while another thread is currently writing. This was fully functional since the last versions
  
• when a file is requested from cache, and does not exist in cache, it is opened and its header is read. Here was a problem, since another thread could come in with the same file request and issue another header read when the first file wasn't finished reading the same header and inserting the file into the cache. Here another lock was needed as well as a forced file synchronization (NIO File channel can defer writing operations) for the header, so all unwritten data is written to disk. This was also the reason for the NoHeaderExceptions, and a possible cause of corruption and chunk errors.
  
• when the limit (set by starmade) of open files is reached, the system will upon file request (and cache miss) close and remove the file from the cache, that was accessed the longest ago. Here was the bug: by keeping a local map for each game entity, the method that removed the file did it globally and then was attempting to remove that from its own cache. Now the file on the other hand was not necessary part of that cache to begin with, so basically the file while closed remained in the local cache of the entity that owned it. This is where the "FileNotFoundException: Having to reopen closed file" came from. The reopened file however was no longer in the global map: the removing removed it from global but tried to remove it from the wrong local map. So the file was no longer part of the "remove file that was accessed the longest ago" operation, and therefore remained open. This is a leak since if it happened more often the amount of open files that are no longer subject of being removed will rise over time, eventually slowing the OS down or causing the "Too many open files"-Exception, which is also dependent on the OS, and crashing the game.
  
• another running condition was a thread(A) reading while another thread(B) requested a whole other file (which is fine), but while doing so issued a "remove file that was accessed the longest ago" operation, because the limit of open files was reached: If in this case the file, that thread(A) is currently reading, was selected, thread(B) would close the file, and thread(A) would throw an exception reporting back to the game that no data was being read.

Sorry for the long text. I had to write it down to confirm the logic for myself.



Thank you for playing StarMade,

- schema

 

[aceface]

What has this update done which can be noticed in the game?

 

[davisev5225]

I think he spelled it out pretty clearly there - it prevents server errors and crashes related to invalid file handles. To dumb it down enough for you, \"server less error, more run\".

 

[davisev5225]

Oh, and Schema, thanks for all your hard work. I\'m looking forward to when these pre-builds are stable enough for release - I\'ve been subject to many of these bugs, which manifest themselves in the form of chunks no longer loading upon logging back in (after a client crash). This normally necessitates a server restart to correct. If I\'ve been understanding these pre-builds correctly, this will drastically reduce or possibly even eliminate that behavior.

Cheers!

 

[Trafalgar]

The block editor seems to be bringing up two IllegalAccessException when looking at any block (so far as I have seen) in it. It\'s because ElementInformation has private members. It\'s also showing \'Cannot parse cubatomRecipe: nf\' and \'Cannot parse cubatomCompound: ne[]\' once the display for a block is open. Possibly related.

This happens whether using the prebuild or the last release build.

 

[aceface]

no he does not spell it out clearly in my opinion, that\'s why I\'m asking

 

[Ejraiel]

Doing more sector boundary testing near 2 2 2; I exited my teeny (7 block) ship near the boundary of four sectors, about 4 blocks away from where I was working, after coming to a full stop.

When I turned around to re-enter, it was 20-30 or so blocks away.

Even more concerning (and confirming) was the behavior when I either entered it once I reached it, or when I exited it again (I forget which); it or I was suddenly at the location where I had parked it originally.

 

[Ejraiel]

This time, I saw my ship move. I was at the 8 sector boundary of 2 2 2 and 1 3 1 (in in the corner of 1 2 2), and watched my ship move into 2 2 1. The ship motion is irregular; it begins after a random interval, and moves smoothly for about 10 seconds.

When I re-entered my ship after flying to it as an astronaut, I found I was suddenly at the opposite corner (horizontally, so near 3 2 1) of 2 2 1, barely able to see my home station.

After flying around and stopping near the same 8 corner area, alt-tabbing to type this, and going back to the game, I found I was teleported again, near the same location. This time, however, none of my station was visible, from the part with the build blocks at the center of the 2 2 2 / 2 2 1 wall, nor the edges I\'ve started building around the spawn borders. It did start loading eventually, however.

Got screenshots this time: https://mega.co.nz/#!TMlQSS5K

 

[Ejraiel]

I flew back over to my work area, and backed up (accidentally, lol; *BUMP*ed) into the vertical structure seen in recent screenshots. (It\'s water.)

And then my client crashed. Gave me the same generic message as when disconnected due to a server /shutdown. Upon logging in, I found myself over 8 sectors away. (Hmm, half a system?)

Flying to my ship, I realized I wasn\'t 1500m from 2 2 2 as it appeared, but the full 8-ish sectors. And then, my older pre-client-crash ship (that I flew to in one I made) moved downward on it\'s own, at a faster rate than previous times.

Also, btw, your server message gets cut off due to lack of wrapping (as seen in the first screenshot).

Screenshots: https://mega.co.nz/#!yIFnwbjA



Also, since it may be related, here\'s a bug I encountered in 0.0942:

\"As far as file operations go, [and syncing,] I believe that may have been the cause of a bug I encountered the other day. In addition to witnessing ships completely clipping through each other, when I collided one of my larger ships with another (both around 200 length, 50 width/height,) I found my ship not only partially clipped into it, but left a chunk around it\'s core.

The 16^3 chunk around my ship\'s core was copied to the same chunk surrounding the other ship\'s core. The blocks only overwrote empty block spaces, and were \"ghosts\", as my shots went through them and hit the original blocks now inside. This was on my localhost server, btw. Logging out and back in did nothing. Didn\'t think to test server shutdown/re-run.\"

Screenshots: https://mega.co.nz/#!bIsxWQaA



And while I\'m reporting bugs, I found a generic overflow and multi-instance ones that *may* be related to the game processes earlier last night (0.09424, I think):

\"This may simply be Windows or Java, not sure:

I placed a large amount of blocks at once, (256x256x55, x8 due to using all three symmetry planes,) causing the client process to freeze as to user input. (When I did this with only enough blocks for about 256x256x3x8, unlike now when I made sure to have more than enough, the client froze for maybe 3-5 minutes.)

Well, the audio continued throughout the freeze, even while not in game (it tried to stay fullscreened behind the taskbar and desktop windows, likely due to the fact that alt+tab and ctrl+shift+esc are nonfuncional when the client is frozen and so I used ctrl+alt+del to \"minimize\" the process). However, even though the audio was still running when I returned hours later, the process was no longer visually accessible. (Could be seen in task manager, using about 1.5 GB of RAM.) Upon running a second instance of the client, I found the window to have entirely blank white contents (just the initial news/update/etc. window, not yet the settings/connect/launch window). Within a second or two of closing the unaccessible still running javaw.exe process, the usual content was created in the second instance\'s window.

So anyway, if that info helps in any way, there it is.\"

 

[Ejraiel]

Found out I needed to be sharing the keys for these to be downloadable.



Ship not synced pt. 2: https://mega.co.nz/#!TMlQSS5K!HSTK1H50zGtnZb89ROZuxk38ww85siuyKTxpg08mvh0

Ship not synced crash: https://mega.co.nz/#!yIFnwbjA!AK-yZguRq6866ZXcPpV1YXOTmDsJNM0QPOjsZAcPHdg

Core chunk ghost blocks: https://mega.co.nz/#!bIsxWQaA!WeP5nlHGW_qZ4-2RqK1Vg1kryScVgrZo-S0DrpUPyZw

Inventory + lag = dupe & loss: https://mega.co.nz/#!eQ93Ea7Z!avQ7bxrgxYW1HkFuZiqKKBtfVj41WOKtibyBZ3sm_Xg

 

[Trafalgar]

Saved a design (6489 blocks) with Create New Entry on a local server (not sandbox, dedicated server running locally, no other players), and the client declared that the server was not responding, for around 45 seconds before the server apparently unfroze and finished saving. Possibly a problem in the file I/O still?

 

[schema]

Hmm disks sometimes have periods where they take long to \"wake up\". I\'ll see that I can move the blueprint saving into another thread

 

[schema]

Thank you

 

[Trafalgar]

I tested saving it again with another name and it did the same thing, so I\'m not sure that\'s the problem?

 

[Ejraiel]

Another bug, still in this new version (build 20131015_152625):

Sometimes, while in build mode in a station, pressing Esc to pull up the menu, then selecting Back, will move the camera to where the player\'s body was before entering the build block. In this state the camera can be rotated, but not moved. Pressing Esc and selecting Back again resets the camera to where you were previously in build mode.

 

[Ejraiel]

while testing this bug more, after exiting the build block, my camera remained where it was in build mode; I could even see my player (far away). Moving over sector boundaries while in this state caused teleportation of current camera location.

After a short while, I was deposited back into astronaut mode as I should have been; though now I was away from the build block, due to the astronaut moving while I pressed movement keys when my camera was still at the build mode location.

Screenshots: https://mega.co.nz/#!SA803K4D!QE38viGkJFwpVkYNBVxiszvVOvUg2ih6HNCczQAapj4

 

[Ejraiel]

http://www.youtube.com/watch?v=SyX_4CtRHPk

 

[Ejraiel]

...in build 20131015_162228.

 

[Darcion]

Gravity-Unit does not work anymore (since 152625) i got the message but nothing chanced. Some times it works or it work after time...



anyone have the same issue?

 

[Darcion]

if i move to an other Gravity-Unit and press \"R\", this works, but this is not helpfull.