Add admin command "/Server_Restart"

[Napther]

TL;DR: Add a new command that shuts down the server from inside the game, BUT, Brings it online again after it has completed the shut down to reduce potential down time.

As the title goes, From my time of being a member of the admin/Moderator staff of the StarSide community, this included assisting in the running of the Starmade server, Shattered Skies. Lets not get into polotics about it, though...

Most admin staff as far as I know (Except the owners and trusted individuals) do not have access to the Server's back-end, and are thus unable to initiate a server start, or update it, etc.

In my time of Moderating, there have been times I felt the server needed to be brought down temporarily in order to deal with an issue after it has restarted, Or, the server is starting to have issues with loading new chunks, or the missile explosives are no longer dealing damage and needs a reboot to get it working again.

Time and Time again, there wasnt anyone who HAD access to the back end of the server online in order to reboot and fix these issues. As the Fix for the issue is just shutting down the server and it never coming back online for another 3 hours...

This allows admin staff "basic" control of the server itself. Of course, when the server is down, they cant activate it since they cant join an offline server, so we are still waiting for a "Super-admin" or Owner to come and reboot the server in the event of a sudden server crash.

This will also reduce the reliance on "Wrappers" by a small fraction. Only a small fraction.

 

[AndyP]

Well, most servers run in their own small loop script.
Usually inside a screen (program).

This allows a /shutdown to be an automatic restart.
And all services you provide on your system should act like this.
Shut them down, and they auto-restart IF the shutdown was not intended.

There is no need for a wrapper to do this.
From technical side, its also hard to impossible to do this.
As a full and useful restart needs to end the virtual machine (process environment for starmade.)
And to my knowledge it cannot execute code to restart it self after ending.

- Andy

 

[Erth Paradine]

A java process cannot "restart" itself; it needs to exit completely, and then be started anew.

To "restart" a server, issue the /shutdown command, while the server instance itself is called within a simple while loop (no wrappers needed). The example script included below will maintain a backup copy of your server instance, and auto-update your install upon each restart.

Launch your server via the following script, using the command:
screen -dmS Prod /path/to/script

Create script:

#!/bin/bash

# Set installed path
SMPath=/opt/prodStarMade

while :
do
    # Create snapshot for later backup archival
    rsync -Hax --stats --delete-after $SMPath/StarMade/ $SMPath/StarMade.backup/
    # Check for and install updates
    cd $SMPath
    /usr/bin/java -jar ./StarMade-Starter.jar -nogui
    # Launch server
    cd $SMPath/StarMade
    /usr/bin/java -Xms128m -Xmx1024m -jar StarMade.jar -server
    sleep 10
done

There are a few caveats to this strategy, most notably that servers do have a tendency to hang on shutdown, there's also threading and deadlock issues that'll kill a server as well, all of which usually requires someone with administrative access to the host OS for cures.

 

[Napther]

Well, most servers run in their own small loop script.
Usually inside a screen (program).

This allows a /shutdown to be an automatic restart.
And all services you provide on your system should act like this.
Shut them down, and they auto-restart IF the shutdown was not intended.

There is no need for a wrapper to do this.
From technical side, its also hard to impossible to do this.
As a full and useful restart needs to end the virtual machine (process environment for starmade.)
And to my knowledge it cannot execute code to restart it self after ending.

- Andy

Ahh right. So it does have to completely end all scripts. I was thinking that the script to restart would be able to continue right to the end, then initiate the reboot. Thank you for the response.

As for the outside program doing the reboot, couldnt this be a standard, or too much work for such a small thing? Or is this just a CMD-box like thing, in which case, there should be some page or tutorial somewhere on big things to run a server, etc that I have no doubt missed because I am me...
[DOUBLEPOST=1466637067,1466636825][/DOUBLEPOST]

Create script:

#!/bin/bash

# Set installed path
SMPath=/opt/prodStarMade

while :
do
    # Create snapshot for later backup archival
    rsync -Hax --stats --delete-after $SMPath/StarMade/ $SMPath/StarMade.backup/
    # Check for and install updates
    cd $SMPath
    /usr/bin/java -jar ./StarMade-Starter.jar -nogui
    # Launch server
    cd $SMPath/StarMade
    /usr/bin/java -Xms128m -Xmx1024m -jar StarMade.jar -server
    sleep 10
done

...I sure hope your server really isnt running off 1024m of ram you suggested... Starmade is most definitely not a Non-Modded, Private Minecraft server, if not, how much are you running?
Also remember some of us are Windows scrublords too, so that specific code wont work...

 

[Nuclear Doughnut]

No matter what, this is/has been a longtime issue about being able to manage the game from within. Not all hardware owners are as willing to give up their passwords or access as others. It also cuts down on the "large teams of active available admins" large servers REQUIRE and must keep on hand 24/7 just to keep it running when some idiot at 3am accidentally causes the server to crash. [Or Purposefully]

Not all servers are running even the same operating system. So a unified rebooter would be very useful.

Another issue coupled with this is there is an extreme lack of Administration hierarchy to allow certain players certain functions within. Right now its an "All or Nothing" approach without having a wrapper.

You can add specific functions to certain admins but this is tedious and theres no simple Operator class.

 

[Erth Paradine]

...
Another issue coupled with this is there is an extreme lack of Administration hierarchy to allow certain players certain functions within. Right now its an "All or Nothing" approach without having a wrapper.

You can add specific functions to certain admins but this is tedious and theres no simple Operator class.

Recognized - /add_admin_allowed_comand
[DOUBLEPOST=1466641965,1466641318][/DOUBLEPOST]

...
...I sure hope your server really isnt running off 1024m of ram you suggested... Starmade is most definitely not a Non-Modded, Private Minecraft server, if not,

For my example in this thread, I stuck with the defaults maintained in SM's vanilla StarMade-dedicated-server-* files.

how much are you running?

Most days our server averages ~8GB of consumed memory after 24 hours (barring the occasional memory leak).

Also remember some of us are Windows scrublords too, so that specific code wont work...

You have my sympathies.

 

[AndyP]

okay, so if you use the file named StarMade-dedicated-server-windows.bat to launch the server,
create a copy of it and open it with a text editor.

If should contain a line like this:

java -Xms512m -Xmx1024m -Xincgc -Xshare:off -jar StarMade.jar -server

Now alter it to this:

:repeat

java -Xms512m -Xmx1024m -Xincgc -Xshare:off -jar StarMade.jar -server

GOTO repeat

Now, ANY ending the server will lead to an immediate restart.
You can extend it with conditions and possible check for updates, but this works.
No additional magic needed.

And every provider to launch your server, should be able to do this, and even add a condition to skip the restart if quitting the server was intended.

For sure, this will not catch deadlocked threads, and other things that crash the engine, but do not lead to an ended java process.

- Andy

 

[Erth Paradine]

okay, so if you use the file named StarMade-dedicated-server-windows.bat to launch the server,
create a copy of it and open it with a text editor.

If should contain a line like this:

java -Xms512m -Xmx1024m -Xincgc -Xshare:off -jar StarMade.jar -server

Now alter it to this:

:repeat

java -Xms512m -Xmx1024m -Xincgc -Xshare:off -jar StarMade.jar -server

GOTO repeat

Now, ANY ending the server will lead to an immediate restart.
You can extend it with conditions and possible check for updates, but this works.
No additional magic needed.

And every provider to launch your server, should be able to do this, and even add a condition to skip the restart if quitting the server was intended.

For sure, this will not catch deadlocked threads, and other things that crash the engine, but do not lead to an ended java process.

- Andy

A brief sleep would be prudent as well, which can be accomplished by adding the following immediately after a java command, before the goto line:

timeout /t 10 /nobreak

To replicate rsync functionality on Microsoft's platform, I suggest looking into DeltaCopy.

 

[Benevolent27]

All this sounds well and good, but what if they don't want the game to be running in a loop? What about when they want to do multiple backups and update checks? What if the server gets stuck in a crash loop? The method I use to auto-restart the game in linux is an until loop which only runs the game if it ends with an error code. If it shuts down normally, it ends the script. For windows, I use errorlevel checking to determine if the game should restart or not.

With this in mind, it not only possible to implement a "/restart" command, but it'd be trivial to do so for the devs. It does not need to rely on java to accomplish the actual restart. It would simply shut down as normal and exit with a specific errorcode, perhaps 5000? The base script for running the server (on linux, mac, and windows) can then be set to check the errorlevel and if it equals 5000 (or whatever it's set to) then it waits a second and starts the server again. Right now, however, we simply don't have the option. All shutdowns look like normal shutdowns and all crashes look like crashes (except sometimes the server seems to crash with no error, such as with the /clear_overheating bug).

I agree with Napther. I think it's important to allow server owners to easily give or take away the ability to restart the server to their admins, and it'd be trivial to implement.

Here are a few script examples which would work easily if a "/restart" command caused the game to end with an errorlevel of 5000:

Linux:

#!/bin/bash
cd "$(dirname "$0")"
while true; do
java -Xms128m -Xmx1024m -jar StarMade.jar -server
if ! [ $? -eq 5000 ]; then
exit
fi
sleep 1
done

Windows:

@echo off
:start
java -Xms512m -Xmx1024m -Xincgc -Xshare:off -jar StarMade.jar -server
If errorlevel 5000 (
timeout /t 1
goto start
)

 

[Nuclear Doughnut]

Thank you Benevolent27. I'm glad others see the point of the need to allow Admins to give control of restarts (especially with how unstable things can be) to Moderators... I just wish there was a dedicated class of moderators with preset admin commands

 

[Benevolent27]

TL;DR: Add a new command that shuts down the server from inside the game, BUT, Brings it online again after it has completed the shut down to reduce potential down time.

As the title goes, From my time of being a member of the admin/Moderator staff of the StarSide community, this included assisting in the running of the Starmade server, Shattered Skies. Lets not get into polotics about it, though...

Most admin staff as far as I know (Except the owners and trusted individuals) do not have access to the Server's back-end, and are thus unable to initiate a server start, or update it, etc.

In my time of Moderating, there have been times I felt the server needed to be brought down temporarily in order to deal with an issue after it has restarted, Or, the server is starting to have issues with loading new chunks, or the missile explosives are no longer dealing damage and needs a reboot to get it working again.

Time and Time again, there wasnt anyone who HAD access to the back end of the server online in order to reboot and fix these issues. As the Fix for the issue is just shutting down the server and it never coming back online for another 3 hours...

This allows admin staff "basic" control of the server itself. Of course, when the server is down, they cant activate it since they cant join an offline server, so we are still waiting for a "Super-admin" or Owner to come and reboot the server in the event of a sudden server crash.

This will also reduce the reliance on "Wrappers" by a small fraction. Only a small fraction.

To add to my response, there are creative solutions the owners of Shattered Skies could use to enable limited access to control the server. Bear with me though, as it might look rather wonky, but it would totally work. An example would be to create an FTP server on the host machine (using minimal system resources). A script on the server machine would then run in a loop (perhaps every 30 seconds) and check to see if a certain file exists in the FTP folder. If it does, then it performs the appropriate command. There are plenty of free FTP servers for windows out there which would allow separate usernames and passwords for admins as well, so access could be given or restricted easily without compromising the security of the server machine, since these FTP users would ONLY have access to one folder and their access could be revoked immediately. Then a limited access admin could be given a "control" script that allows the admin to run a batch file with a menu and select options for what command they'd like to send. This command menu would then use a program like ftp.exe to upload an empty file with the correct name for the command. On the next loop check, the command is run.

I would agree, however, that a "/restart" command natively being included in the game would be preferable.
[DOUBLEPOST=1466842106,1466842017][/DOUBLEPOST]

Thank you Benevolent27. I'm glad others see the point of the need to allow Admins to give control of restarts (especially with how unstable things can be) to Moderators... I just wish there was a dedicated class of moderators with preset admin commands

I'd like to see permission groups myself. Create a permission group, then assign or unassign a person to it. Then you can have whatever levels of access you want to those permission groups, thus changing all the access of each member. It'd also be easy to add or remove people to these groups.
[DOUBLEPOST=1466842543][/DOUBLEPOST]

A brief sleep would be prudent as well, which can be accomplished by adding the following immediately after a java command, before the goto line:

timeout /t 10 /nobreak

To replicate rsync functionality on Microsoft's platform, I suggest looking into DeltaCopy.

Why not use robocopy?

 

[Erth Paradine]

Why not use robocopy?

Depends upon what you're after at that point. As robocopy lacks many of the features found with rsync, but robocopy could fulfill a basic folder mirror function. Native rsync is significantly faster than robocopy though - so at that point the answer depends upon how large your world is, and how long you want it offline for backup syncs.

 

[DrTarDIS]

All this sounds well and good, but what if they don't want the game to be running in a loop? What about when they want to do multiple backups and update checks? What if the server gets stuck in a crash loop? The method I use to auto-restart the game in linux is an until loop which only runs the game if it ends with an error code. If it shuts down normally, it ends the script. For windows, I use errorlevel checking to determine if the game should restart or not.

With this in mind, it not only possible to implement a "/restart" command, but it'd be trivial to do so for the devs. It does not need to rely on java to accomplish the actual restart. It would simply shut down as normal and exit with a specific errorcode, perhaps 5000? The base script for running the server (on linux, mac, and windows) can then be set to check the errorlevel and if it equals 5000 (or whatever it's set to) then it waits a second and starts the server again. Right now, however, we simply don't have the option. All shutdowns look like normal shutdowns and all crashes look like crashes (except sometimes the server seems to crash with no error, such as with the /clear_overheating bug).

I agree with Napther. I think it's important to allow server owners to easily give or take away the ability to restart the server to their admins, and it'd be trivial to implement.

Here are a few script examples which would work easily if a "/restart" command caused the game to end with an errorlevel of 5000:

Linux:

#!/bin/bash
cd "$(dirname "$0")"
while true; do
java -Xms128m -Xmx1024m -jar StarMade.jar -server
if ! [ $? -eq 5000 ]; then
exit
fi
sleep 1
done

Windows:

@echo off
:start
java -Xms512m -Xmx1024m -Xincgc -Xshare:off -jar StarMade.jar -server
If errorlevel 5000 (
timeout /t 1
goto start
)

For - Looping commands | Windows CMD | SS64.com

Depends upon what you're after at that point. As robocopy lacks many of the features found with rsync, but robocopy could fulfill a basic folder mirror function. Native rsync is significantly faster than robocopy though - so at that point the answer depends upon how large your world is, and how long you want it offline for backup syncs.

msdn.microsoft.com/en-us/sync/bb887623.aspx
Introducing Microsoft Sync Framework: Sync Services for File Systems

 

[Benevolent27]

For - Looping commands | Windows CMD | SS64.com

msdn.microsoft.com/en-us/sync/bb887623.aspx
Introducing Microsoft Sync Framework: Sync Services for File Systems

Could use a for loop, I guess, but why add the complexity? My windows script is easy to understand and only restarts the server if it ends with an errorcode of 5. It can be easily modified to restart the server if any errorcode is thrown, or to loop no matter what. What would be the benefit of a 'for' loop? (Oh and I know that "goto" is so faux pas or whatever, but I really couldn't care less about that, lol)

[DOUBLEPOST=1466932650,1466932453][/DOUBLEPOST]

Depends upon what you're after at that point. As robocopy lacks many of the features found with rsync, but robocopy could fulfill a basic folder mirror function. Native rsync is significantly faster than robocopy though - so at that point the answer depends upon how large your world is, and how long you want it offline for backup syncs.

Could use rsync with Cygwin. lol