Read by Council Auto Restart Feature for server side

[Gamel]

Hi all, since 0.18999, my server always got frozen after running
for 18-28 hours. Then a manual restart for the java program is needed.
But I am not at home all time and most ssh connection of android(I tried 6)
can't use Ctrl+A+D which is a must for using "screen" in linux.

So I am looking for a auto restart solution. And I found this two old post
http://oldsite.star-made.org/content/auto-restart
http://oldsite.star-made.org/content/constant-server-crashing
But I have no idea how to set this up, never use any scripts before, I am new to linux...

So I suggest the "StarMade-dedicated-server-linux.sh" should include auto-restart function

 

[RandomEvent]

I use theSmelter for my server home.
They are helpful up to a certain point.
Unfortunately I cannot access the linux server to issue commands or scripts.
I would like the game to have an inbuilt ability to hard reboot.
The game tends to partially hang which does not trigger auto server reboots (at least not theSmelter one)

Also, /shutdown # seems to sometimes shut the server down and sometimes reboot it???????

Not enough admin control in my opinion and hard to find updated admin commands.
Great game, but broken in parts and all the guys at SM want to do is add new content instead of repairing whats there.
"Waits for the flame, sigh"

 

[Gamel]

I use theSmelter for my server home.
They are helpful up to a certain point.
Unfortunately I cannot access the linux server to issue commands or scripts.
I would like the game to have an inbuilt ability to hard reboot.
The game tends to partially hang which does not trigger auto server reboots (at least not theSmelter one)

Also, /shutdown # seems to sometimes shut the server down and sometimes reboot it???????

Not enough admin control in my opinion and hard to find updated admin commands.
Great game, but broken in parts and all the guys at SM want to do is add new content instead of repairing whats there.
"Waits for the flame, sigh"

we need more admin control

 

[nightrune]

This should likely not be in the main game, and adding a watch dog for a process is something an admin usually does.

I'd you have a server wrapper please post here?

 

[nightwalker_007]

Hi all, since 0.18999, my server always got frozen after running
for 18-28 hours. Then a manual restart for the java program is needed.
But I am not at home all time and most ssh connection of android(I tried 6)
can't use Ctrl+A+D which is a must for using "screen" in linux.

So I am looking for a auto restart solution. And I found this two old post
http://oldsite.star-made.org/content/auto-restart
http://oldsite.star-made.org/content/constant-server-crashing
But I have no idea how to set this up, never use any scripts before, I am new to linux...

So I suggest the "StarMade-dedicated-server-linux.sh" should include auto-restart function

You can issue a server command instead:
killall -9 java
killall -9 screen

You could also write a wrapper script around the script you use to launch the game:

#!/bin/bash
export seconds_between_restarts=$((60*60*6))
export restart_warning=$((60*5))
export shutdown_time=$((60*5))
path_to_starmade_script=/home/noobie/StarMade/StarMade-dedicated-server-linux.sh"

restarter(){
sleep $seconds_between_restarts
echo "/shutdown $restart_warning"
sleep $restart_warning
sleep $shutdown_time
}

until false
do
restarter| sh $path_to_starmade_script
done


See? Easy.

 

[jayman38]

It would be nice if the game provided some kind of "heartbeat", so that external systems could monitor it for a freeze or race state. Is the game always writing to a log file? You could always check log file size for differences from time to time as a heartbeat check.

 

[nightwalker_007]

It would be nice if the game provided some kind of "heartbeat", so that external systems could monitor it for a freeze or race state. Is the game always writing to a log file? You could always check log file size for differences from time to time as a heartbeat check.

It sometimes sends messages to stdout, not a log file. Please learn more about the server before posting suggestions.

 

[Valck]

It would be nice if the game provided some kind of "heartbeat", so that external systems could monitor it for a freeze or race state. Is the game always writing to a log file? You could always check log file size for differences from time to time as a heartbeat check.

It sometimes sends messages to stdout, not a log file. Please learn more about the server before posting suggestions.

You could always send a message to the server and see if it responds.
StarMade 0.09378: Fixes, shield balance & new admin tools

 

[Gamel]

You can issue a server command instead:
killall -9 java
killall -9 screen

You could also write a wrapper script around the script you use to launch the game:

#!/bin/bash
export seconds_between_restarts=$((60*60*6))
export restart_warning=$((60*5))
export shutdown_time=$((60*5))
path_to_starmade_script=/home/noobie/StarMade/StarMade-dedicated-server-linux.sh"

restarter(){
sleep $seconds_between_restarts
echo "/shutdown $restart_warning"
sleep $restart_warning
sleep $shutdown_time
}

until false
do
restarter| sh $path_to_starmade_script
done


See? Easy.

Can you further explain how this works? I am not sure what to do at all.

 

[Benevolent27]

Right now some of the issues with auto-restart are..

1. Hard crash: This is easy enough to solve for either windows or linux. You can create a script that simply runs the server, if there's an errorlevel on exit, it restarts it. If not, then the script ends. Boom, simple.

2. Soft Crash: This one is the hard issue to deal with. Sometimes the server will crash or get hung up, but not actually exit. So the process is still running, but the server isn't responding for players. Some existing scripts try to deal with this by reading the output of the log and when it notices certain errors, kills the server and restarts it. However, these scripts tend not to be very complete, so some situations remain.

So, for StarMade to have an auto-restart function, I think they'd need to basically make the server hard crash rather than soft-crash in these situations. And for them to do that, they'll need to identify these situations it's occurring in. Then they could very easily have a batch file or .sh file that handles the hard crashes built into the server as an option for players to use.

 

[nightwalker_007]

Can you further explain how this works? I am not sure what to do at all.

killall -9 java

This issues signal 9 (kill) to all java processes owned by the current user (or all java processes if run as root)​

For the other stuff:

#!/bin/bash

All Unix shell scripts should start with this "#!" character followed by the location of the command interpreter. In this case, bash. You may see #!/usr/bin/perl, and #!/usr/bin/python as well in files. You could do #!/bin/more for a self-opening readme file, but that's kind of silly.​

export something=$((some+math))

export takes a variable and exports it to the environment, making it accessible to sub-shells. The $((some+math)) expands to the result of the math operation. In the case 60 seconds * 60 minutes * n hours.​

function() { ... }

While it's easy to think of this a s a function, it's actually the equivalent of writing a shell script within a shell script. "function" is run and behaves just as if it were a stand-alone program--and can even be launched a s a background process.​

until false; do stuff;done

This creates a loop that will run forever until aborted.​

[DOUBLEPOST=1460702943,1460702347][/DOUBLEPOST]

2. Soft Crash: This one is the hard issue to deal with. Sometimes the server will crash or get hung up, but not actually exit. So the process is still running, but the server isn't responding for players. Some existing scripts try to deal with this by reading the output of the log and when it notices certain errors, kills the server and restarts it. However, these scripts tend not to be very complete, so some situations remain.
.

There are some commands that can be issued to the server console that will produce output. These would have to be read by a program/script with non-blocking stdin/stderr reads.

 

[Benevolent27]

killall -9 java

This issues signal 9 (kill) to all java processes owned by the current user (or all java processes if run as root)​

For the other stuff:

#!/bin/bash

All Unix shell scripts should start with this "#!" character followed by the location of the command interpreter. In this case, bash. You may see #!/usr/bin/perl, and #!/usr/bin/python as well in files. You could do #!/bin/more for a self-opening readme file, but that's kind of silly.​

export something=$((some+math))

export takes a variable and exports it to the environment, making it accessible to sub-shells. The $((some+math)) expands to the result of the math operation. In the case 60 seconds * 60 minutes * n hours.​

function() { ... }

While it's easy to think of this a s a function, it's actually the equivalent of writing a shell script within a shell script. "function" is run and behaves just as if it were a stand-alone program--and can even be launched a s a background process.​

until false; do stuff;done

This creates a loop that will run forever until aborted.​

[DOUBLEPOST=1460702943,1460702347][/DOUBLEPOST]

There are some commands that can be issued to the server console that will produce output. These would have to be read by a program/script with non-blocking stdin/stderr reads.

Yeah, but it's not really a complete solution, right?

Though I don't think it'd be a bad idea for there to be a restart script included with the game though, because why not? I've seen admins for some of the top servers struggling with getting a solution that works well.