Running Your Debian Server in RAM

[MrGrey1]

This was a response to kiwispike in his thread but it got so long I decided to give it a thread of its own. It's a how to for setting up and running a StarMade server in a RAM drive on a Debian system. You will need a decent chunk of RAM, 8 gigs may suffice for a small server with nothing else running but I recommend your server has at least 16 G available.

Righto, it's been a while since I set this all up so lets see if I can cover everything without missing things.

Firstly! Make a backup of your starmade server folder some where safe before implementing and testing this. It's a process that has the potential to wipe out the server directory if the sync goes wrong somewhere. All care taken, no guarantees given and all that jazz.

I run a Debian server, the SM server is run in a program called screen and you'll also need a program called rsync that handles all the file transfers. Both screen and rsync are available in the repos. On my server the user that runs starmade is also called starmade and the SM server is setup and installed as standard in the users home directory... so /home/starmade/starmade/ through all my scripts. You will either need to setup a starmade user and install the server as I have done or modify the scripts to suit your settings.

In Debian there is a file location /dev/shm/ This is the tempfs mount point and is mounted in/on your RAM.
By default it should have access to 50% of your available system RAM. You can check this with

df -h
Filesystem                                         Size  Used Avail Use% Mounted on
tmpfs                                              16G  4.1G   12G  26% /dev/shm

^That's my current readout with starmade running.
If you want more then 50% of RAM allocated you'll need to edit /etc/fstab, I won't go into the details here but there's lots online on how to achieve that. /dev/shm being RAM is volatile so any power loss or restarts and everything in it is wiped out. What we need to do is setup the SM server so the start script copies the SM server folder from it's default location /home/starmade/starmade to /dev/shm/starmade and then runs the server from there. Then we need to make sure we get regular snapshots from the RAM drive back to the default location just in case of unexpected reboot. To achieve this I have 3 script files in /home/starmade/bin/
RAM_start.sh, RAM_Stop.sh and RAM_save.sh
There is also a cron job running for the save script.

I almost forgot, I have a custom start script in the server folder called start.sh. It is a direct copy of StarMade-dedicated-server-linux.sh with the java settings modified as desired. The reason I have done this, when the server is updated the default StarMade-dedicated-server-linux.sh is reset to non-executable meaning your server won't start. You will need to create the start.sh file and set it to executable as well.

cd /home/starmade/starmade/
cp StarMade-dedicated-server-linux.sh start.sh && chmod u+x start.sh

(Be warned, my scripting skills are pretty rough. If anyone wants to chip in with advice on how to improve them I'm all ears. )

Spoiler: RAM_start.sh

I have commented out the MOTD script and the backup script I run every start. They are not essential for the process. If you want to run them as well I'll add those scripts at the end.

#!/bin/bash

## PERMAMENT = the default location of the SM server folder. 
##STARTSCREEN function = starts the SM server inside a screen session called starmade_screen.

VOLATILE=/dev/shm/starmade/
PERMANENT=/home/starmade/starmade/
STARTSCREEN() {
        /usr/bin/screen -DmS starmade_screen "$VOLATILE"start.sh
}

## Run message of the day.

#if ! pgrep -x MOTD.sh > /dev/null
#        then
#                /home/starmade/bin/MOTD.sh &
#fi

## Run Backup

#/home/starmade/bin/backup_starmade_server.sh  2>&1 > /home/starmade/backup_log.txt

##Checks to see if a screen session called starmade_screen is running and abort if true, we don't want to start it twice.

if screen -list | grep -q starmade_screen
then echo StarMade screen running. Aborting!

##Checks if the /dev/shm/starmade (VOLATILE) folder exists, if true it resyncs the default to the volatile. 
##ie For after a manual stop of the server and a settings change then restart.

else
if [ -d "$VOLATILE" ];
then
        rsync -r -t -v "$PERMANENT" "$VOLATILE"
        STARTSCREEN

##If no folder is found in /dev/shm then create the folder and sync, then start the screen session.

else
        mkdir -p "$VOLATILE";
        rsync -r -t -v "$PERMANENT" "$VOLATILE"
        STARTSCREEN
fi
fi

Spoiler: RAM_stop.sh

#!/bin/bash

VOLATILE=/dev/shm/starmade/
PERMANENT=/home/starmade/starmade/

## Send the shutdown 1 command to the SM server running in screen. Wait until the screen is closed and then sync Volatile to Permanent.

/usr/bin/screen -p 0 -S starmade -X eval 'stuff "/shutdown 1"\\015'

until ! screen -list | grep -q starmade_screen  > /dev/null

do
          sleep 5
done

rsync -r -t -v "$VOLATILE" "$PERMANENT"

##If MOTD script is running kill it also.

if  pgrep -x MOTD.sh > /dev/null
then
pkill -x MOTD.sh
fi

Spoiler: RAM_save.sh and crontab entry

The RAM_save.sh script is run through cron. Edit the starmade users cron with 'crontab -e' and add this entry. It will run the RAM_save.sh script every 3 minutes.

*/3 * * * * /home/starmade/bin/RAM_save.sh 2>&1 > /home/starmade/RAM_save_log.txt

The actual RAM_save.sh script.

#!/bin/bash

VOLATILE=/dev/shm/starmade/
PERMANENT=/home/starmade/starmade/

## If the VOLATILE file is present and the SM screen session is running then run rsync to save.

if [ -d "$VOLATILE" ] &&  screen -list | grep -q starmade_screen
then
rsync -r -t -v "$VOLATILE" --delete "$PERMANENT"
echo Backup run "$(date '+%y-%m-%d-%H%M')"
else
echo Backup aborted. Starmade not running.
echo "$(date '+%y-%m-%d-%H%M')"
fi

Spoiler: MOTD.sh

The good thing about running your SM server in screen is it's easy to send messages and commands to the screen session. Put this MOTD.sh in /home/starmade/bin/ along with the RAM scripts and uncomment the relevant MOTD section in the RAM_start.sh script to have this script run when your server starts.

#!/bin/bash
##This script repeats indefinitely and every 600 seconds sends a command to the server. You can substitute any command you like.

while true

do
/usr/bin/screen -x -S starmade_screen -X stuff "/server_message_broadcast plain 'Everything is Awesome!'^M"
sleep 600
/usr/bin/screen -x -S starmade_screen -X stuff "/server_message_broadcast plain 'It's cool to be part of the team!'^M"
sleep 600
/usr/bin/screen -x -S starmade_screen -X stuff "/server_message_broadcast info '+12 Mining Faction Space'^M/server_message_broadcast info '+6 Mining Claimed Space'^M/server_message_broadcast info '+3 Mining Unclaimed Space'^M"
sleep 600

done

Spoiler: backup_starmade_server.sh

Again I commented out the backup section in the RAM_start.sh. Uncomment if you want to use this script. You will also need to create the initial backup directories. The script makes 3 backups and needs 4 files. You can create all those files with a single command like

mkdir -p /home/starmade/backups/{starmade,starmade1,starmade2,starmade_tmp}

If you want to restore a backup, make sure the server is shutdown and that you are in /home/starmade, then do something like this

rsync -avzh backups/starmade1/ --delete starmade/

^This will restore the SECOND last backup.

#!/bin/sh

## For a better understanding of this script see http://www.mikerubel.org/computers/rsync_snapshots/

BACKUPS=/home/starmade/backups

mv  $BACKUPS/starmade2 $BACKUPS/starmade_tmp
mv  $BACKUPS/starmade1 $BACKUPS/starmade2
mv  $BACKUPS/starmade $BACKUPS/starmade1
mv  $BACKUPS/starmade_tmp $BACKUPS/starmade
rsync -rvt --del --link-dest=$BACKUPS/starmade1 /home/starmade/starmade/ $BACKUPS/starmade

echo Backup run "$(date '+%y-%m-%d-%H%M')"

This has taken a lot longer then anticipated and it's now quite late for me. I hope I haven't made any critical errors. Next I get a chance I'll add information on how to setup the server as a process using systemd so that the server starts/stops/restarts on it's own.

 

[kiwispike]

This is awesome. Thanks so much for your time. I googled as much server info as I could find. And it was a fruitless search. Which is why I didn't bother setting up a server till now.

Much appreciation for your time.

 

[kiwispike]

The more I work with these scripts, the more awesome I realise they are.

I'm not using the ramdrive yet. But I am using the backup system.

Thanks again MrGrey1 .

A question, is there a way to read server chat? It would be good if I could read and respond to chat alerts.

Eg (server has issues) player types "restart", script checks if server has been restarted recently, if not sends command "/shutdown 300"

Emails server admin that it has been done.

 

[MrGrey1]

Using tail you can follow the chat logs. Below is the command I use to view all chat logs the server is generating simultaneously. Substitute a specific log such as all.txt for the * if you only want to read just the one.

tail -f /dev/shm/starmade/chatlogs/*

I haven't setup any specific automation for this yet but yes it's quite possible. There are two ways I know of to send commands to the server. You can use the StarNet.jar to send commands or you can use the screen session. I prefer using screen myself but StarNet is probably initially more versatile as you can more easily use it from other locations. For automation I would use grep to monitor the log and execute a command if a certain word was detected.

Spoiler: StarNet Examples

StarNet
These are two scripts using StarNet that I use to send commands from my primary machine.
(Server password is set in your server.cfg SUPER_ADMIN_PASSWORD )

Type the script name followed by the command. Eg StarNet_server_command /shutdown 1
StarNet_server_command

#!/bin/bash

#Location of StarNet.jar
STARNET=/home/USER/.steam/steam/steamapps/common/StarMade/StarMade/StarNet.jar
#Server details
SERVERIP=###.###.###.###:4242
SERVERPW=myserverpassword

java -jar $STARNET $SERVERIP $SERVERPW $@

This is one was another experiment that I made to take more complicated commands but then remodified just for a detailed playerlist. That's what the commented out echo and read are about. I left it as is, it may be of use... Anyway, very useful for logging all the details of who's on your server.

StarNet_player_list

#!/bin/bash

#Location of StarNet.jar
STARNET=/home/USER/.steam/steam/steamapps/common/StarMade/StarMade/StarNet.jar
#Server details
SERVERIP=###.###.###.###:4242
SERVERPW=myserverpassword

#echo Enter Starmade Server Command
#read INPUT

INPUT=/player_list

java -jar $STARNET $SERVERIP $SERVERPW $INPUT |& grep -v 'PERSONAL'| grep 'Name:\|IP:\|SM-NAME:\|SECTOR:\|FACTION:\|CREDITS:'

echo Executed: $INPUT

Spoiler: Commands Using Screen

I prefer a remote shell and using screen. The MOTD.sh script I posted before is essentially how to go about it. If you login to your server with screen, you can view the starmade screen session.
Login with your normal user using ssh.
You can then change credentials to your starmade user `su starmade` and then view the running game server using `screen -x` or `screen -r`. You can then enter commands directly to the console.
To do it via command line without the console it's just like the MOTD script.

/usr/bin/screen -x -S starmade_screen -X stuff "CommandGoesHere^M"

eg 

/usr/bin/screen -x -S starmade_screen -X stuff "/server_message_broadcast info '+12 Mining Faction Space'^M/server_message_broadcast info '+6 Mining Claimed Space'^M/server_message_broadcast info '+3 Mining Unclaimed Space'^M"

Got to run. Would maybe have added more details if I had time but see how you go with that.

 

[MrGrey1]

The more I work with these scripts, the more awesome I realise they are.

I'm not using the ramdrive yet. But I am using the backup system.

Thanks again MrGrey1 .

A question, is there a way to read server chat? It would be good if I could read and respond to chat alerts.

Eg (server has issues) player types "restart", script checks if server has been restarted recently, if not sends command "/shutdown 300"

Emails server admin that it has been done.

Needed a break from building last night and I thought I'd stretch my scripting skills and see if I could do what you wanted. Success, at least in my limited testing and with a couple of caveats.
The below script will restart you server after a set amount of time has passed based on a trigger word entered in a specific chat channel. It can be modified for any trigger word and channel and also to run any command desired. There's a few caveats I've left in the comments. I'd be wary of who I gave the trigger word to of course.
EDIT: you will need to install inotify-tools
EDIT2: as an after thought I'd suggest you set up a specific channel for this script so the trigger words are kept out of the general chat.

Spoiler: Chat channel command script.

#!/bin/bash

## First two functions get the time the screen session was started and the current time and convert them to a decimal value in seconds.
## ***This means the starmade_screen session will need to be restarted every day after midnight or it will break***.
## The MONITORLOG function checks the last line of the all.txt chat log for the variable TRIGGER. Set the specific log with SMLOG.
## The server restart command is commented out for testing.
## Start the script with the server, ie add it to the RAM_start script just like MOTD.sh.
## If the server is terminated by other means the script may still be running.
## Modify the RAM_stop.sh script to check for and kill this script, again just like the MOTD.sh script.

SCREENNAME=starmade_screen
SMLOG=/dev/shm/starmade/chatlogs/all.txt
TSLAST=600
TRIGGER=blahblah

SCREENSTARTED() {
        screen -ls | grep "$SCREENNAME" | awk '{print $3}' | sed 's/)//' | awk -F: '{ print ($1 * 3600) + ($2 * 60) + $3 }'
}

CURRENTTIME() {
        echo "$(date '+%H:%M:%S')" | awk -F: '{ print ($1 * 3600) + ($2 * 60) + $3 }'
}

MONITORLOG() {
         tail -n1 "$SMLOG" | grep -q "$TRIGGER"
}

## While "$SCREENNAME" is running do
until ! screen -ls | grep -q "$SCREENNAME"  > /dev/null

do

#inotifywait checks the log until it is modified.
        inotifywait -qq -e modify "$SMLOG"

X=$( CURRENTTIME )
Y=$( SCREENSTARTED )
#echo Current time is $X seconds.
#echo Screen was started at $Y seconds.
TIMERUN="$(($X-$Y))"
#echo The time Screen has been running is $TIMERUN seconds.


## Set the time since last reboot to allow before another reboot is permitted.
if [ "$TIMERUN" -ge "$TSLAST" ] && MONITORLOG

        then

#       echo Server restarting.
        /usr/bin/screen -x -S "$SCREENNAME" -X stuff "/chat Server is rebooting in 10 seconds^M"
##Uncomment for the actual reboot command to be run.
#       /usr/bin/screen -x -S "$SCREENNAME" -X stuff "/Shutdown 10^M"
        exit
fi

## Set the return comment if less then the specified time has past.
if [ "$TIMERUN" -lt "$TSLAST" ] && MONITORLOG

        then

#       echo less then 10 minutes since last restart
       /usr/bin/screen -x -S "$SCREENNAME" -X stuff "/chat Less then 10 minutes since last restart^M"
fi

done

 

[kiwispike]

"***This means the starmade_screen session will need to be restarted every day after midnight or it will break***"

oh my lord, this reminds me of my first commercial job, I had this exact issue! they never shut down my program and if killed it after midnight.

Way back then, I don't remember a simple way, now, its just a couple of calls, and your done.

I didn't know there was a all chat log, this opens up a lot of possibilities.

Thanks for your time!

I've modified the script a bit, It checks restart.log for timestamp
also checks ops.lst for player permissions
This way I can allow all types via chat commands!

 

[NTIMESc]

Nice job with the shell scripts. Regarding handling running duration longer than 24hours, instead of getting the current H:M:S, you can use the date format '+%s' to get the full date in seconds. Then use grep and sed to get the full date string from the screen session and feed that string into the date command, again formatting as seconds with '+%s'. Then just subtract as usual to get the running time in seconds.

SCREENSTARTED() {
     date -d "$(screen -ls ${SCREENNAME} | grep -o \(.*\) | sed 's/(//' | sed 's/).*//')" +%s
}

CURRENTTIME() {
        date +%s
}

 

[Lone_Puppy]

Wow man! I've been wanting to know this stuff for a while now. Has given me the bug to get back into building my own server again.
Thanks MrGrey1 !

Hey kiwispike, are you running a server in NZ?
[doublepost=1557360350,1557360176][/doublepost]Scratch that kiwispike I followed MrGrey1's link to your original thread. I'll have to jump onto your server for a nosey! ;-)