Writing Upstart Scripts

Upstart scripts are a great way to deal with starting and stopping system daemons in Ubuntu and CentOS 6, as well as many other flavors of Linux. While later replaced by systemd, Upstart allows quite a bit of customization when creating init scripts, without as much of the hassle and bash programming knowledge required with init.d System V scripts. In this tutorial, I'll show you how to create one for the Kibana 4.x executable distributed by Elastic.co the makers of ElasticSearch and LogStash. If you haven't messed with Kibana I highly recommend it along side LogStash and fluentd, but that's another tutorial. Any executable could be substituted because Kibana itself is simply a binary distribution.

There are many extra potental pieces in upstart scripts, but we're going to start with what we need for a simple start/stop script which will run upon system boot and shutdown properly on the flip side. All Upstart init scripts live in the /etc/init/ directory and end with '.conf'. This one is simply "kibana.conf". Consider the following config:

# cat /etc/init/kibana.conf
description 'Kibana Service startup'
author 'Joe Legeckis'
env NAME='kibana'
env LOG_FILE=/var/log/kibana.log
env USER=nginx
env BASEDIR=/var/www/kibana/
env BIN=/var/www/kibana/bin/kibana
env PARAMS=''
start on started network
stop on stopping network
# Respawn in case of a crash, with default parameters
respawn
script
 # Make sure logfile exists and can be written by the user we drop privileges to
 touch $LOG_FILE
 chown $USER:$USER $LOG_FILE
 date >> $LOG_FILE
 cd $BASEDIR
 exec su -s /bin/sh -c 'exec "$0" "$@"' $USER -- $BIN $PARAMS >> $LOG_FILE 2>&1
end script

post-start script
 echo "app $NAME post-start event" >> $LOG_FILE
end script

It starts off with a simple human readable information about the job we're defining. The Description and Author. After that, you'll see there are a number of Environment Variables defined as "env =". These are sometimes used by the script/binary itself, and other times, as seen here, used by the upstart script later on to execute said script/binary. In this case, we're defining the Name, Log File, User to run as, BaseDir where Kibana lives, the binary executed to run Kibana and any additional Parameters.

Next, we describe when it should start and stop. In this case, and in most cases, the following two lines will be accurate, but more detail can be provided if you need to chain things together, or make sure that something stops or starts according to another scripts timing. Here we're simply starting up after network starts successfully, and stopping prior to stopping network.

We then define what should happen with a crash by simply specifying that the script should respond. Importantly, there is a default limit which varies from system to system, to determine how many times and if any delay is enforced prior to responding a failed or prematurely exited script. 'respawn' does not check the output value of the script. If you didn't ask the script to stop, then it will run the respawn process and start it back up! It may be prudent to add a line after this:

respawn limit COUNT INTERVAL | unlimited

i.e.
reswpan limit 5 10

The example line will restart the process up to 5 times, with 10s in-between the failure and the next execution. This will help prevent race conditions. Specifying 0 (as in zero) for the count will restart it an unlimited number of times.

And now we get into the meat of the script. Between "script" and "end script" is where you put exactly what you want to run every time the daemon is to start up. In this case, we're going to make sure our log file exists, has the right permissions and gets a fresh datestamp (handy when it dies and you're not sure when it started back up). Then we change directory and run our executable. Using the previously defined variables to fill out the command it executes as the correct user, directing all it's output to the log file, including standard out, just in case. After that we close it up with the "end script" and move on!

"post-start" is simply what happens after starting and before exiting upstart. You can use it for all sorts of things like tests or initial setup checks if need be, here we're just adding a line to the log file.

Now, as long as you've put this script in your /etc/init/ directory (NOT /etc/init.d/) as 'name.conf' or in our case here, 'kibana.conf', you can execute it by running "initctl start name". Stop should work, as well as restart!