Add admin command "/Server_Restart"

    Do you think we need this command?


    • Total voters
      7

    Napther

    Grumpy builder of Kaiju Design Initiative
    Joined
    Feb 7, 2015
    Messages
    192
    Reaction score
    180
    • Competition Winner - Small Fleets
    • Community Content - Bronze 2
    • Legacy Citizen 2
    TL;DR: Add a new command that shuts down the server from inside the game, BUT, Brings it online again after it has completed the shut down to reduce potential down time.

    As the title goes, From my time of being a member of the admin/Moderator staff of the StarSide community, this included assisting in the running of the Starmade server, Shattered Skies. Lets not get into polotics about it, though...

    Most admin staff as far as I know (Except the owners and trusted individuals) do not have access to the Server's back-end, and are thus unable to initiate a server start, or update it, etc.

    In my time of Moderating, there have been times I felt the server needed to be brought down temporarily in order to deal with an issue after it has restarted, Or, the server is starting to have issues with loading new chunks, or the missile explosives are no longer dealing damage and needs a reboot to get it working again.

    Time and Time again, there wasnt anyone who HAD access to the back end of the server online in order to reboot and fix these issues. As the Fix for the issue is just shutting down the server and it never coming back online for another 3 hours...

    This allows admin staff "basic" control of the server itself. Of course, when the server is down, they cant activate it since they cant join an offline server, so we are still waiting for a "Super-admin" or Owner to come and reboot the server in the event of a sudden server crash.

    This will also reduce the reliance on "Wrappers" by a small fraction. Only a small fraction.
     

    AndyP

    Customer Experience Manager
    Joined
    Aug 15, 2013
    Messages
    1,199
    Reaction score
    264
    • Schine
    • Wired for Logic
    Well, most servers run in their own small loop script.
    Usually inside a screen (program).

    This allows a /shutdown to be an automatic restart.
    And all services you provide on your system should act like this.
    Shut them down, and they auto-restart IF the shutdown was not intended.

    There is no need for a wrapper to do this.
    From technical side, its also hard to impossible to do this.
    As a full and useful restart needs to end the virtual machine (process environment for starmade.)
    And to my knowledge it cannot execute code to restart it self after ending.

    - Andy
     

    Erth Paradine

    Server Admln & Bug Reporter
    Joined
    Feb 15, 2016
    Messages
    239
    Reaction score
    58
    A java process cannot "restart" itself; it needs to exit completely, and then be started anew.

    To "restart" a server, issue the /shutdown command, while the server instance itself is called within a simple while loop (no wrappers needed). The example script included below will maintain a backup copy of your server instance, and auto-update your install upon each restart.

    Launch your server via the following script, using the command:
    screen -dmS Prod /path/to/script

    Create script:
    Code:
    #!/bin/bash
    
    # Set installed path
    SMPath=/opt/prodStarMade
    
    while :
    do
        # Create snapshot for later backup archival
        rsync -Hax --stats --delete-after $SMPath/StarMade/ $SMPath/StarMade.backup/
        # Check for and install updates
        cd $SMPath
        /usr/bin/java -jar ./StarMade-Starter.jar -nogui
        # Launch server
        cd $SMPath/StarMade
        /usr/bin/java -Xms128m -Xmx1024m -jar StarMade.jar -server
        sleep 10
    done
    There are a few caveats to this strategy, most notably that servers do have a tendency to hang on shutdown, there's also threading and deadlock issues that'll kill a server as well, all of which usually requires someone with administrative access to the host OS for cures.
     

    Napther

    Grumpy builder of Kaiju Design Initiative
    Joined
    Feb 7, 2015
    Messages
    192
    Reaction score
    180
    • Competition Winner - Small Fleets
    • Community Content - Bronze 2
    • Legacy Citizen 2
    Well, most servers run in their own small loop script.
    Usually inside a screen (program).

    This allows a /shutdown to be an automatic restart.
    And all services you provide on your system should act like this.
    Shut them down, and they auto-restart IF the shutdown was not intended.

    There is no need for a wrapper to do this.
    From technical side, its also hard to impossible to do this.
    As a full and useful restart needs to end the virtual machine (process environment for starmade.)
    And to my knowledge it cannot execute code to restart it self after ending.

    - Andy
    Ahh right. So it does have to completely end all scripts. I was thinking that the script to restart would be able to continue right to the end, then initiate the reboot. Thank you for the response.

    As for the outside program doing the reboot, couldnt this be a standard, or too much work for such a small thing? Or is this just a CMD-box like thing, in which case, there should be some page or tutorial somewhere on big things to run a server, etc that I have no doubt missed because I am me...
    [DOUBLEPOST=1466637067,1466636825][/DOUBLEPOST]
    Create script:
    Code:
    #!/bin/bash
    
    # Set installed path
    SMPath=/opt/prodStarMade
    
    while :
    do
        # Create snapshot for later backup archival
        rsync -Hax --stats --delete-after $SMPath/StarMade/ $SMPath/StarMade.backup/
        # Check for and install updates
        cd $SMPath
        /usr/bin/java -jar ./StarMade-Starter.jar -nogui
        # Launch server
        cd $SMPath/StarMade
        /usr/bin/java -Xms128m -Xmx1024m -jar StarMade.jar -server
        sleep 10
    done
    ...I sure hope your server really isnt running off 1024m of ram you suggested... Starmade is most definitely not a Non-Modded, Private Minecraft server, if not, how much are you running?
    Also remember some of us are Windows scrublords too, so that specific code wont work...
     

    Nuclear Doughnut

    A Radioactive Pastry
    Joined
    Jul 2, 2013
    Messages
    157
    Reaction score
    135
    • Purchased!
    • Legacy Citizen 10
    No matter what, this is/has been a longtime issue about being able to manage the game from within. Not all hardware owners are as willing to give up their passwords or access as others. It also cuts down on the "large teams of active available admins" large servers REQUIRE and must keep on hand 24/7 just to keep it running when some idiot at 3am accidentally causes the server to crash. [Or Purposefully]

    Not all servers are running even the same operating system. So a unified rebooter would be very useful.

    Another issue coupled with this is there is an extreme lack of Administration hierarchy to allow certain players certain functions within. Right now its an "All or Nothing" approach without having a wrapper.

    You can add specific functions to certain admins but this is tedious and theres no simple Operator class.
     

    Erth Paradine

    Server Admln & Bug Reporter
    Joined
    Feb 15, 2016
    Messages
    239
    Reaction score
    58
    ...
    Another issue coupled with this is there is an extreme lack of Administration hierarchy to allow certain players certain functions within. Right now its an "All or Nothing" approach without having a wrapper.

    You can add specific functions to certain admins but this is tedious and theres no simple Operator class.
    Recognized - /add_admin_allowed_comand
    [DOUBLEPOST=1466641965,1466641318][/DOUBLEPOST]
    ...
    ...I sure hope your server really isnt running off 1024m of ram you suggested... Starmade is most definitely not a Non-Modded, Private Minecraft server, if not,
    For my example in this thread, I stuck with the defaults maintained in SM's vanilla StarMade-dedicated-server-* files.

    how much are you running?
    Most days our server averages ~8GB of consumed memory after 24 hours (barring the occasional memory leak).

    Also remember some of us are Windows scrublords too, so that specific code wont work...
    You have my sympathies.

    HereIsANickle.png
     
    • Like
    Reactions: Valck

    AndyP

    Customer Experience Manager
    Joined
    Aug 15, 2013
    Messages
    1,199
    Reaction score
    264
    • Schine
    • Wired for Logic
    okay, so if you use the file named StarMade-dedicated-server-windows.bat to launch the server,
    create a copy of it and open it with a text editor.

    If should contain a line like this:
    Code:
    java -Xms512m -Xmx1024m -Xincgc -Xshare:off -jar StarMade.jar -server
    Now alter it to this:

    Code:
    :repeat
    
    java -Xms512m -Xmx1024m -Xincgc -Xshare:off -jar StarMade.jar -server
    
    GOTO repeat
    Now, ANY ending the server will lead to an immediate restart.
    You can extend it with conditions and possible check for updates, but this works.
    No additional magic needed.

    And every provider to launch your server, should be able to do this, and even add a condition to skip the restart if quitting the server was intended.

    For sure, this will not catch deadlocked threads, and other things that crash the engine, but do not lead to an ended java process.

    - Andy
     

    Erth Paradine

    Server Admln & Bug Reporter
    Joined
    Feb 15, 2016
    Messages
    239
    Reaction score
    58
    okay, so if you use the file named StarMade-dedicated-server-windows.bat to launch the server,
    create a copy of it and open it with a text editor.

    If should contain a line like this:
    Code:
    java -Xms512m -Xmx1024m -Xincgc -Xshare:off -jar StarMade.jar -server
    Now alter it to this:

    Code:
    :repeat
    
    java -Xms512m -Xmx1024m -Xincgc -Xshare:off -jar StarMade.jar -server
    
    GOTO repeat
    Now, ANY ending the server will lead to an immediate restart.
    You can extend it with conditions and possible check for updates, but this works.
    No additional magic needed.

    And every provider to launch your server, should be able to do this, and even add a condition to skip the restart if quitting the server was intended.

    For sure, this will not catch deadlocked threads, and other things that crash the engine, but do not lead to an ended java process.

    - Andy
    A brief sleep would be prudent as well, which can be accomplished by adding the following immediately after a java command, before the goto line:
    Code:
    timeout /t 10 /nobreak
    To replicate rsync functionality on Microsoft's platform, I suggest looking into DeltaCopy.
     

    Benevolent27

    Join the Dark Side
    Joined
    Aug 21, 2015
    Messages
    585
    Reaction score
    327
    • Purchased!
    All this sounds well and good, but what if they don't want the game to be running in a loop? What about when they want to do multiple backups and update checks? What if the server gets stuck in a crash loop? The method I use to auto-restart the game in linux is an until loop which only runs the game if it ends with an error code. If it shuts down normally, it ends the script. For windows, I use errorlevel checking to determine if the game should restart or not.

    With this in mind, it not only possible to implement a "/restart" command, but it'd be trivial to do so for the devs. It does not need to rely on java to accomplish the actual restart. It would simply shut down as normal and exit with a specific errorcode, perhaps 5000? The base script for running the server (on linux, mac, and windows) can then be set to check the errorlevel and if it equals 5000 (or whatever it's set to) then it waits a second and starts the server again. Right now, however, we simply don't have the option. All shutdowns look like normal shutdowns and all crashes look like crashes (except sometimes the server seems to crash with no error, such as with the /clear_overheating bug).

    I agree with Napther. I think it's important to allow server owners to easily give or take away the ability to restart the server to their admins, and it'd be trivial to implement.

    Here are a few script examples which would work easily if a "/restart" command caused the game to end with an errorlevel of 5000:

    Linux:
    Code:
    #!/bin/bash
    cd "$(dirname "$0")"
    while true; do
    java -Xms128m -Xmx1024m -jar StarMade.jar -server
    if ! [ $? -eq 5000 ]; then
    exit
    fi
    sleep 1
    done
    Windows:
    Code:
    @echo off
    :start
    java -Xms512m -Xmx1024m -Xincgc -Xshare:off -jar StarMade.jar -server
    If errorlevel 5000 (
    timeout /t 1
    goto start
    )
     
    Last edited:

    Nuclear Doughnut

    A Radioactive Pastry
    Joined
    Jul 2, 2013
    Messages
    157
    Reaction score
    135
    • Purchased!
    • Legacy Citizen 10
    Thank you Benevolent27. I'm glad others see the point of the need to allow Admins to give control of restarts (especially with how unstable things can be) to Moderators... I just wish there was a dedicated class of moderators with preset admin commands
     

    Benevolent27

    Join the Dark Side
    Joined
    Aug 21, 2015
    Messages
    585
    Reaction score
    327
    • Purchased!
    TL;DR: Add a new command that shuts down the server from inside the game, BUT, Brings it online again after it has completed the shut down to reduce potential down time.

    As the title goes, From my time of being a member of the admin/Moderator staff of the StarSide community, this included assisting in the running of the Starmade server, Shattered Skies. Lets not get into polotics about it, though...

    Most admin staff as far as I know (Except the owners and trusted individuals) do not have access to the Server's back-end, and are thus unable to initiate a server start, or update it, etc.

    In my time of Moderating, there have been times I felt the server needed to be brought down temporarily in order to deal with an issue after it has restarted, Or, the server is starting to have issues with loading new chunks, or the missile explosives are no longer dealing damage and needs a reboot to get it working again.

    Time and Time again, there wasnt anyone who HAD access to the back end of the server online in order to reboot and fix these issues. As the Fix for the issue is just shutting down the server and it never coming back online for another 3 hours...

    This allows admin staff "basic" control of the server itself. Of course, when the server is down, they cant activate it since they cant join an offline server, so we are still waiting for a "Super-admin" or Owner to come and reboot the server in the event of a sudden server crash.

    This will also reduce the reliance on "Wrappers" by a small fraction. Only a small fraction.
    To add to my response, there are creative solutions the owners of Shattered Skies could use to enable limited access to control the server. Bear with me though, as it might look rather wonky, but it would totally work. An example would be to create an FTP server on the host machine (using minimal system resources). A script on the server machine would then run in a loop (perhaps every 30 seconds) and check to see if a certain file exists in the FTP folder. If it does, then it performs the appropriate command. There are plenty of free FTP servers for windows out there which would allow separate usernames and passwords for admins as well, so access could be given or restricted easily without compromising the security of the server machine, since these FTP users would ONLY have access to one folder and their access could be revoked immediately. Then a limited access admin could be given a "control" script that allows the admin to run a batch file with a menu and select options for what command they'd like to send. This command menu would then use a program like ftp.exe to upload an empty file with the correct name for the command. On the next loop check, the command is run.

    I would agree, however, that a "/restart" command natively being included in the game would be preferable.
    [DOUBLEPOST=1466842106,1466842017][/DOUBLEPOST]
    Thank you Benevolent27. I'm glad others see the point of the need to allow Admins to give control of restarts (especially with how unstable things can be) to Moderators... I just wish there was a dedicated class of moderators with preset admin commands
    I'd like to see permission groups myself. Create a permission group, then assign or unassign a person to it. Then you can have whatever levels of access you want to those permission groups, thus changing all the access of each member. It'd also be easy to add or remove people to these groups.
    [DOUBLEPOST=1466842543][/DOUBLEPOST]
    A brief sleep would be prudent as well, which can be accomplished by adding the following immediately after a java command, before the goto line:
    Code:
    timeout /t 10 /nobreak
    To replicate rsync functionality on Microsoft's platform, I suggest looking into DeltaCopy.
    Why not use robocopy?
     
    • Like
    Reactions: Nuclear Doughnut

    Erth Paradine

    Server Admln & Bug Reporter
    Joined
    Feb 15, 2016
    Messages
    239
    Reaction score
    58
    Why not use robocopy?
    Depends upon what you're after at that point. As robocopy lacks many of the features found with rsync, but robocopy could fulfill a basic folder mirror function. Native rsync is significantly faster than robocopy though - so at that point the answer depends upon how large your world is, and how long you want it offline for backup syncs.
     

    DrTarDIS

    Eldrich Timelord
    Joined
    Jan 16, 2014
    Messages
    1,116
    Reaction score
    310
    All this sounds well and good, but what if they don't want the game to be running in a loop? What about when they want to do multiple backups and update checks? What if the server gets stuck in a crash loop? The method I use to auto-restart the game in linux is an until loop which only runs the game if it ends with an error code. If it shuts down normally, it ends the script. For windows, I use errorlevel checking to determine if the game should restart or not.

    With this in mind, it not only possible to implement a "/restart" command, but it'd be trivial to do so for the devs. It does not need to rely on java to accomplish the actual restart. It would simply shut down as normal and exit with a specific errorcode, perhaps 5000? The base script for running the server (on linux, mac, and windows) can then be set to check the errorlevel and if it equals 5000 (or whatever it's set to) then it waits a second and starts the server again. Right now, however, we simply don't have the option. All shutdowns look like normal shutdowns and all crashes look like crashes (except sometimes the server seems to crash with no error, such as with the /clear_overheating bug).

    I agree with Napther. I think it's important to allow server owners to easily give or take away the ability to restart the server to their admins, and it'd be trivial to implement.

    Here are a few script examples which would work easily if a "/restart" command caused the game to end with an errorlevel of 5000:

    Linux:
    Code:
    #!/bin/bash
    cd "$(dirname "$0")"
    while true; do
    java -Xms128m -Xmx1024m -jar StarMade.jar -server
    if ! [ $? -eq 5000 ]; then
    exit
    fi
    sleep 1
    done
    Windows:
    Code:
    @echo off
    :start
    java -Xms512m -Xmx1024m -Xincgc -Xshare:off -jar StarMade.jar -server
    If errorlevel 5000 (
    timeout /t 1
    goto start
    )
    For - Looping commands | Windows CMD | SS64.com :)
    Depends upon what you're after at that point. As robocopy lacks many of the features found with rsync, but robocopy could fulfill a basic folder mirror function. Native rsync is significantly faster than robocopy though - so at that point the answer depends upon how large your world is, and how long you want it offline for backup syncs.
    msdn.microsoft.com/en-us/sync/bb887623.aspx
    Introducing Microsoft Sync Framework: Sync Services for File Systems
     

    Benevolent27

    Join the Dark Side
    Joined
    Aug 21, 2015
    Messages
    585
    Reaction score
    327
    • Purchased!
    Could use a for loop, I guess, but why add the complexity? My windows script is easy to understand and only restarts the server if it ends with an errorcode of 5. It can be easily modified to restart the server if any errorcode is thrown, or to loop no matter what. What would be the benefit of a 'for' loop? (Oh and I know that "goto" is so faux pas or whatever, but I really couldn't care less about that, lol)

    [DOUBLEPOST=1466932650,1466932453][/DOUBLEPOST]
    Depends upon what you're after at that point. As robocopy lacks many of the features found with rsync, but robocopy could fulfill a basic folder mirror function. Native rsync is significantly faster than robocopy though - so at that point the answer depends upon how large your world is, and how long you want it offline for backup syncs.
    Could use rsync with Cygwin. lol
     
    • Like
    Reactions: Valck