Script to update Linux based servers with selective file download

[Doomsider]

Recent changes to how the update system works has changed the way many automated scripts updates Linux based servers. AndyP has been extremely helpful pointing out a possible solution to this. I decided to write a example script to update the server using the checksum so that only files that have been changed willl be downloaded. This method was originally written by Schema and then improved upon. It does rely on curl which is not always installed on distros so you may have to install it.

*Edit: 7/14/2015 added check for missing files
*Edit: 7/14/2015 added check for missing directories and fix for curling web pages with spaces
*Edit: 7/28/2015 Moved to Github added some robustness

https://github.com/doomsider/SMUPDATE

To install:

wget https://raw.githubusercontent.com/doomsider/SMUPDATE/master/update.sh

To Run:

chmod +x update.sh
./update.sh

Spoiler

#!/bin/bash
# Starmade Selective upgrade script for BASH. Please run in the parent or root directory of Starmade
# Requires curl to be installed
# Grab the releaseindex as a very long string variable
echo "This script will install/update StarMade from the master repo. Use ./update.sh force to force a check on all files"
read -s -r -p "Press any key to continue..." -n 1 dummy
if command -v curl >/dev/null
then
echo "Curl was found"
releaseindex=$(curl http://files.star-made.org/releasebuildindex)
# Go to the end of the variable at the last .
newversion=${releaseindex##*.}
echo "newversion $newversion"
# Get the first version to compare
cutstring=${newversion#*_}
NEWSMVERSION1=${cutstring%_*}
echo "NEWSMVERSION1 $NEWSMVERSION1"
# Get the second version to compare
NEWSMVERSION2=${cutstring#*_}
echo "NEWSMVERSION2 $NEWSMVERSION2"
# Gather the old version from the version.txt file in StarMade
oldversion=$(cat StarMade/version.txt)
if [ -z "$oldversion" ]
then
echo "No install found for StarMade"
OLDSMVER1=0
OLDSMVER2=0
else
echo "oldversion $oldversion"
# Get the first version to compare
cutstring=${oldversion#*#}
OLDSMVER1=${cutstring%_*}
echo "OLDSMVER1 $OLDSMVER1"
# Get the second version to compare
OLDSMVER2=${cutstring#*_}
echo "OLDSMVER2 $OLDSMVER2"
fi
# If the first or second newversion exceeds the first or second old version
if [ "$NEWSMVERSION1" -gt "$OLDSMVER1" ] || [ "$NEWSMVERSION2" -gt "$OLDSMVER2" ] || [ "$1" = "force" ]
then
echo "Newer Version Detected"
# Set the field seperator to new line and then store the chucksums as an array with each element being a line
OLD_IFS=$IFS
IFS=$'\n'
releaseindex=( $(curl http://files.star-made.org$newversion/checksums) )
IFS=$OLD_IFS
# Set line count to 0 then go through the array line by line until the array index is unset
LINECOUNT=0
while [ -n "${releaseindex[$LINECOUNT]+set}" ]
do
CURRENTSTRING=${releaseindex[$LINECOUNT]}
#echo $CURRENTSTRING
# Format the current line by removing everything after the first space and then removing ./ in the beginning of the name
cutstring=${CURRENTSTRING%[[:space:]]*}
cutstring=${cutstring%[[:space:]]*}
CURRENTFILE=${cutstring/.\//}
# Formatfile was added here because when requesting web URL space should be replaced with %20
FORMATFILE=${CURRENTFILE// /%20}
cutstring=${CURRENTSTRING%[[:space:]]*}
cutstring=${cutstring##*[[:space:]]}
CURRENTCKSUM=${cutstring%%[[:space:]]*}
OLDCKSUMSTRING=$(cksum "StarMade/$CURRENTFILE")
# Check to see if OLDCKSUMSTRING is set, if not this indicates the file does not exist
if [ -z "$OLDCKSUMSTRING" ]
then
echo "No existing file found - downloading file"
# Makes sure directory structure is created if it does not exist for the file write
if [ ! -f "StarMade/$CURRENTFILE" ]; then
mkdir -p "StarMade/$CURRENTFILE"
rm -r "StarMade/$CURRENTFILE"
fi
curl "http://files.star-made.org$newversion/$FORMATFILE" > "StarMade/$CURRENTFILE"
else
cutstring=${OLDCKSUMSTRING#*[[:space:]]}
OLDCKSUM=${cutstring%%[[:space:]]*}
echo "CURRENTFILE $CURRENTFILE CURRENTCKSUM $CURRENTCKSUM OLDCKSUM $OLDCKSUM"
# Check to see if the cksums differ
if [ "$CURRENTCKSUM" -ne "$OLDCKSUM" ]
then
# Download the new file and then copy it into the proper location
echo "Updated file detected - downloading file"
# Makes sure directory structure is created if it does not exist for the file write
if [ ! -f "StarMade/$CURRENTFILE" ]; then
mkdir -p "StarMade/$CURRENTFILE"
rm -r "StarMade/$CURRENTFILE"
fi
curl "http://files.star-made.org$newversion/$FORMATFILE" > "StarMade/$CURRENTFILE"
else
echo "Current file detected - no action"
fi
fi
let LINECOUNT++
done
else
echo "Version Current"
fi
else
echo "You must install Curl to run this"
fi

 

[Doomsider]

Okay well I think I can stick a fork in this one unless some bug pops up. Updated script to use checksums to determine if a file needs to be downloaded.

 

[Grimson]

I get an error when running this. It is in line 33 and is this: Syntax error: "(" unexpected (expecting "fi"). Any idea what is causing this. Also I made sure that I have curl installed already but it doesn't look like that is the problem. Also out of curiosity what Linux system are you testing this on.
Disregard all of that /\ , I wasn't executing the script correctly.
[DOUBLEPOST=1438100843,1438097640][/DOUBLEPOST]Now that I can execute the script I have a question. What does it do with 100% new files that have never existed before. There is no old checksum and it looks like you get an error and then it skips downloading the file. Also if the folder path doesn't exist it errors out. I see this where new blueprints are put. Is this the intended behavior?

 

[Doomsider]

The first question:

# Check to see if OLDCKSUMSTRING is set, if not this indicates the file does not exist
if [ -z "$OLDCKSUMSTRING" ]
then

The second question:

# Makes sure directory structure is created if it does not exist for the file write
if [ ! -f "StarMade/$CURRENTFILE" ]; then
mkdir -p "StarMade/$CURRENTFILE"
rm -r "StarMade/$CURRENTFILE"

Which files are not being downloaded exactly because it does work, but it is WIP so there may be some issue I missed.

 

[Doomsider]

I was unable to replicate any bugs that you mentioned. I put this in Github so you can address code related issues there as well. I made it so it will also download StarMade if it is not there and added a force option to force it to check all the files despite the version match.

 

[Grimson]

I have no clue how but I managed to get an older version of the script that didn't have the directory and file checks in it.

 

[Grimson]

Well I got the newer version from the link above and it works like a charm. The only problem is that the version.txt file has the same cksum value all the time and doesn't update.(I checked the cksum value of three different versions and they where all 23).

 

[Doomsider]

Umm it does not use checksum to determine version, it uses the actual version number in the version.txt. You can force an upgrade/check of files with the force parameter like ./update.sh force. If you have doubts you can change the version number to anything lower and run update. You will see that the checksum is different and the version.txt will be updated. If you have experienced something different from this let me know. Cheers :D

Edit: Noticed I missied one of your questions back there. I am using 14.04 Ubuntu currently to develop on but any Debian based distro should be very similar in operation.