Absurd Minds :: Bash script for monitoring/restarting servers

Absurd Minds http://forums.absurdminds.net/

Bash script for monitoring/restarting servers http://forums.absurdminds.net/viewtopic.php?f=58&t=1006	Page 1 of 1

Author:

Amaroq [ 13 Sep 2011 10:48 ]

Post subject:

Bash script for monitoring/restarting servers

I've figured out how to write a bash script to restart a server if it should crash. I mostly figured it out due to this thread at the Steam Forums, but I did have to figure a few things out on my own.

Here is what I did.

First off, I'm basically creating a "watchdog" that keeps its eye on my server. If the server goes down, it executes a script to restart the server.

In order to do this, I first had to install qstat2. That link is for a tarball file.

I downloaded the tar file onto my computer and used FileZilla to transfer it into my gamer's home directory (gamer is the user through which I run my game servers). I then unpacked the file:

Code:

tar xfz qstat-qstat2.tar.gz

This made a new directory in gamer's home directory called qstat2. I went to that directory:

Code:

cd ./qstat2

and then installed the software.

First, I run the script called autogen.sh.

Code:

autogen.sh

If you get the error "./autogen.sh: line 66: aclocal: command not found", that means you haven't installed a particular package. Su to root and install it. After you install it, switch back to the normal user.

Code:

su - root
password: rootpassword
yum install automake
exit

Now compile the program. These steps are also located in COMPILE.txt, which you can read using nano.

Code:

./configure && make
make install

Now that qstat2 is installed, I need to write a few scripts to get the watchdog running.

The first script is to start watchdog. It will get the whole process running. (Also, this is the process I will run to start my server, instead of using the ./hlds_run command in the server's root directory.)

start_watchdog.sh

Open up a new file in the qstat2 directory called start_watchdog.sh

Code:

nano start_watchdog.sh

Within this script I'm going to do three things. I'm going to kill any screen sessions called "wd" which is where I am running my watchdog script (I run everything through screens since I only have remote access to my machine). Then I'm going to generate a new screen called "wd". Then I'm going to run the program watchdog.sh. Note, the ./ in front of watchdog.sh means that watchdog.sh needs to be in the same directory as the start_watchdog.sh

Code:

#!/bin/sh

echo "Killing screen session..."
pkill -f "SCREEN -dmS wd"
sleep 2

echo "Starting new screen session..."
screen -dmS wd ./watchdog.sh

So, this script is calling another script called watchdog.sh. So I'll make that now. Still in the qstat2 directory, use nano watchdog.sh to make this new script.

watchdog.sh

I'm making watchdog.sh to watch a GunGame server. Within this script I launch qstat and tell it to look for either DOWN or no response. If it finds that answer, then it will wait 30 seconds (to make sure HLDS's auto-restart wasn't in progress). Once it has waited 30 seconds, if it still gets no response, it will execute the script start_gungame.sh.

Code:

#!/bin/sh

while true
do

if (~/qstat2/qstat -nh -a2s 64.79.103.75:27018 | egrep "DOWN|no response" >/dev$
then echo
date
echo "GunGame down (1)"
# Sleep 10s to honor restart timeout
sleep 10

if (~/qstat2/qstat -nh -a2s 64.79.103.75:27018 | egrep "DOWN|no response" >/dev$
then echo
date
echo "GunGame down (2)"
# Sleep 10s to honor restart timeout
sleep 10

if (~/qstat2/qstat -nh -a2s 64.79.103.75:27018 | egrep "DOWN|no response" >/dev$
then echo
date
echo "GunGame down (3)"
# Sleep 10s to honor restart timeout
sleep 10

~/qstat2/start_gungame.sh >/dev/null
fi
fi
fi


sleep 10
done

There are a couple of things to notice about this script. The IP that qstat is monitoring needs to be the IP address of your server when you type status in console. If it says unknown or nothing, you need to use the localhost (usually 127.0.0.1) as the IP address. Also, at the end, it launches "start_gungame.sh" by pointing to the directory ~/qstat2. Make sure you have the directory correct. I spent a while trying to fix my scripts when the only problem was an incorrect directory.

start_gungame.sh

Now comes the start_gungame.sh script, which is what launches the server if it crashes. This code kills my gungame screen session, changes to the gungame directory on my server, and then launches a new screen session called gungame. Then it launches the final script, start.sh.

Code:

#!/bin/sh

echo "Killing existing screen session..."
echo
pkill -f "SCREEN -dmS gungame"
sleep 2

echo "Starting new screen session..."
echo
cd ~/czero/gungame
screen -dmS gungame ../../qstat2/start.sh

Notice the very last line of this code: ../../qstat2/start.sh. This is saying "go up a directory, then up another directory, then into qstat2, then run start.sh." You could easily put the start.sh script in the server's main directory to avoid this. Once again, just make sure the directories are matching up correctly.

start.sh

The final script just launches the server.

Code:

#!/bin/sh

PORT=27018
UPDATE=false

if $UPDATE
then ./update.sh
fi

./hlds_run -game czero -port $PORT +ip 64.79.103.75 +maxplayers 16 +map de_dust2_cz -autoupdate

Now that I have these scripts, instead of going into my game server's root directory and launching my game that way, I go into qstat2 and just run start_watchdog. This will automatically start up the server because it will see that the server is down.

Some errors

I ran into some errors while trying to get these scripts working. The first, like I've mentioned, was just having the directories correct. Make sure you are being careful about the directories and where they are pointing.

The second error, when I first attempted to run start_watchdog.sh, I ended up getting a Permission denied error. All I had to do was change the mode of all the scripts to allow read and execute. (I also allowed write because I suck at linux and can't ever remember how to just do read and execute.) Here is an example of how to change a file to allow read, write, and execute.

Code:

chmod 777 start_watchdog.sh

A third error I was getting was start_watchdog.sh wasn't launching a screen instance. For some reason, all I had to do was update screen and it fixed the problem:

Code:

yum update screen

The last error I got was a pretty weird error.

Code:

./start_watchdog.sh: /bin/sh^M: bad interpreter: No such file or directory

Apparently this happened because I wrote this script in windows and then just dropped it into my server through ftp. I have no idea what the deal was, but when I re-wrote the script exactly the same in nano on the linux machine instead of in windows it worked perfectly.

Author:	luTheNinja [ 13 Sep 2011 21:12 ]
Post subject:	Re: Bash script for monitoring/restarting servers
the last error is because linux shell scripts written in windows need to be in a specific unicode. thats why. for writing linux shells on windows using notepad or notepadd++ save the encoding as unicode or UTF-8 i believe. thats why you has that error at the start. just a tip for next time anubis

Author:	Amaroq [ 14 Sep 2011 06:53 ]
Post subject:	Re: Bash script for monitoring/restarting servers
Ah, ok. I was using Notepad++, but I didn't do anything special with the saving. Do you know enough about shell scripting to write a script to restart the server machine at a certain time of the day and then automatically start all my processes when it reboots?

Author:	luTheNinja [ 14 Sep 2011 19:35 ]
Post subject:	Re: Bash script for monitoring/restarting servers
you can add a time function to the already made watchdog. im not that much of a linux guy, but i know how to find stuff about it. http://www.cyberciti.biz/faq/shell-scri ... ifference/ dunno if that will help tho

Page 1 of 1	All times are UTC-04:00
Powered by phpBB® Forum Software © phpBB Limited https://www.phpbb.com/