Writing Upstart Scripts

Upstart scripts are a great way to deal with starting and stopping system daemons in Ubuntu and CentOS 6, as well as many other flavors of Linux. While later replaced by systemd, Upstart allows quite a bit of customization when creating init scripts, without as much of the hassle and bash programming knowledge required with init.d System V scripts. In this tutorial, I'll show you how to create one for the Kibana 4.x executable distributed by Elastic.co the makers of ElasticSearch and LogStash. If you haven't messed with Kibana I highly recommend it along side LogStash and fluentd, but that's another tutorial. Any executable could be substituted because Kibana itself is simply a binary distribution.

There are many extra potental pieces in upstart scripts, but we're going to start with what we need for a simple start/stop script which will run upon system boot and shutdown properly on the flip side. All Upstart init scripts live in the /etc/init/ directory and end with '.conf'. This one is simply "kibana.conf". Consider the following config:

# cat /etc/init/kibana.conf
description 'Kibana Service startup'
author 'Joe Legeckis'
env NAME='kibana'
env LOG_FILE=/var/log/kibana.log
env USER=nginx
env BASEDIR=/var/www/kibana/
env BIN=/var/www/kibana/bin/kibana
env PARAMS=''
start on started network
stop on stopping network
# Respawn in case of a crash, with default parameters
respawn
script
 # Make sure logfile exists and can be written by the user we drop privileges to
 touch $LOG_FILE
 chown $USER:$USER $LOG_FILE
 date >> $LOG_FILE
 cd $BASEDIR
 exec su -s /bin/sh -c 'exec "$0" "$@"' $USER -- $BIN $PARAMS >> $LOG_FILE 2>&1
end script

post-start script
 echo "app $NAME post-start event" >> $LOG_FILE
end script

It starts off with a simple human readable information about the job we're defining. The Description and Author. After that, you'll see there are a number of Environment Variables defined as "env =". These are sometimes used by the script/binary itself, and other times, as seen here, used by the upstart script later on to execute said script/binary. In this case, we're defining the Name, Log File, User to run as, BaseDir where Kibana lives, the binary executed to run Kibana and any additional Parameters.

Next, we describe when it should start and stop. In this case, and in most cases, the following two lines will be accurate, but more detail can be provided if you need to chain things together, or make sure that something stops or starts according to another scripts timing. Here we're simply starting up after network starts successfully, and stopping prior to stopping network.

We then define what should happen with a crash by simply specifying that the script should respond. Importantly, there is a default limit which varies from system to system, to determine how many times and if any delay is enforced prior to responding a failed or prematurely exited script. 'respawn' does not check the output value of the script. If you didn't ask the script to stop, then it will run the respawn process and start it back up! It may be prudent to add a line after this:

respawn limit COUNT INTERVAL | unlimited

i.e.
reswpan limit 5 10

The example line will restart the process up to 5 times, with 10s in-between the failure and the next execution. This will help prevent race conditions. Specifying 0 (as in zero) for the count will restart it an unlimited number of times.

And now we get into the meat of the script. Between "script" and "end script" is where you put exactly what you want to run every time the daemon is to start up. In this case, we're going to make sure our log file exists, has the right permissions and gets a fresh datestamp (handy when it dies and you're not sure when it started back up). Then we change directory and run our executable. Using the previously defined variables to fill out the command it executes as the correct user, directing all it's output to the log file, including standard out, just in case. After that we close it up with the "end script" and move on!

"post-start" is simply what happens after starting and before exiting upstart. You can use it for all sorts of things like tests or initial setup checks if need be, here we're just adding a line to the log file.

Now, as long as you've put this script in your /etc/init/ directory (NOT /etc/init.d/) as 'name.conf' or in our case here, 'kibana.conf', you can execute it by running "initctl start name". Stop should work, as well as restart!

Resizing Online Disks in Linux with LVM and No Reboots

When we set this up, we only had a 16GB primary disk... Plenty of space. Until you start to write lots of logs and data... and then it fills up quick. So let's talk about how to resize an LVM based partition on a live server without reboots... Reboots are for Windows! This system is a CentOS 6.x machine running in VMWare 5.x that's currently got a 16GiB VMDK based drive. Let's see what we've got to work with:

$ df -h
Filesystem                                         Size  Used Avail Use% Mounted on
/dev/mapper/myfulldisk--vg-root                     12G   11G     0 100% /
none                                               4.0K     0  4.0K   0% /sys/fs/cgroup
udev                                               7.4G  4.0K  7.4G   1% /dev
tmpfs                                              1.5G  572K  1.5G   1% /run
none                                               5.0M     0  5.0M   0% /run/lock
none                                               7.4G     0  7.4G   0% /run/shm
none                                               100M     0  100M   0% /run/user
/dev/sda1                                          236M   37M  187M  17% /boot
/dev/sdc1                                          246G   44G  190G  19% /data

Hmm... time to get on this then. Now, luckily we're running in VMWare. A quick edit to our VM to enlarge the VMDK (not covered in this how-to) will fix this... First, what device are we talking about?

$ dmesg|grep sd
[    1.562363] sd 2:0:0:0: Attached scsi generic sg1 type 0
[    1.562384] sd 2:0:0:0: [sda] 33554432 512-byte logical blocks: (17.1 GB/16.0 GiB)
[    1.562425] sd 2:0:0:0: [sda] Write Protect is off
[    1.562426] sd 2:0:0:0: [sda] Mode Sense: 61 00 00 00
[    1.562460] sd 2:0:0:0: [sda] Cache data unavailable
[    1.562461] sd 2:0:0:0: [sda] Assuming drive cache: write through
[    1.563331] sd 2:0:0:0: [sda] Cache data unavailable
[    1.563451] sd 2:0:1:0: Attached scsi generic sg2 type 0
[    1.563452] sd 2:0:1:0: [sdb] 8388608 512-byte logical blocks: (4.29 GB/4.00 GiB)
[    1.563479] sd 2:0:1:0: [sdb] Write Protect is off
[    1.563481] sd 2:0:1:0: [sdb] Mode Sense: 61 00 00 00
[    1.563507] sd 2:0:1:0: [sdb] Cache data unavailable
[    1.563508] sd 2:0:1:0: [sdb] Assuming drive cache: write through
[    1.563755] sd 2:0:2:0: Attached scsi generic sg3 type 0
[    1.563881] sd 2:0:2:0: [sdc] 524288000 512-byte logical blocks: (268 GB/250 GiB)
[    1.563942] sd 2:0:2:0: [sdc] Write Protect is off
[    1.563944] sd 2:0:2:0: [sdc] Mode Sense: 61 00 00 00
[    1.564008] sd 2:0:2:0: [sdc] Cache data unavailable
[    1.564010] sd 2:0:2:0: [sdc] Assuming drive cache: write through
[    1.564282] sd 2:0:2:0: [sdc] Cache data unavailable
[    1.564283] sd 2:0:2:0: [sdc] Assuming drive cache: write through
[    1.564360] sd 2:0:1:0: [sdb] Cache data unavailable
[    1.564362] sd 2:0:1:0: [sdb] Assuming drive cache: write through
[    1.564989] sd 2:0:0:0: [sda] Assuming drive cache: write through
[    1.571010]  sdb: sdb1
[    1.571426] sd 2:0:1:0: [sdb] Cache data unavailable
[    1.571514] sd 2:0:1:0: [sdb] Assuming drive cache: write through
[    1.571626] sd 2:0:1:0: [sdb] Attached SCSI disk
[    1.574181]  sda: sda1 sda2 < sda5 >
[    1.574797] sd 2:0:0:0: [sda] Cache data unavailable
[    1.574888] sd 2:0:0:0: [sda] Assuming drive cache: write through
[    1.575003] sd 2:0:0:0: [sda] Attached SCSI disk
[    1.579250]  sdc: sdc1
[    1.579805] sd 2:0:2:0: [sdc] Cache data unavailable
[    1.579944] sd 2:0:2:0: [sdc] Assuming drive cache: write through
[    1.580141] sd 2:0:2:0: [sdc] Attached SCSI disk
[    6.922330] Adding 4193276k swap on /dev/sdb1.  Priority:-1 extents:1 across:4193276k FS
[    7.137134] EXT4-fs (sda1): mounting ext2 file system using the ext4 subsystem
[    7.142419] EXT4-fs (sda1): mounted filesystem without journal. Opts: (null)
[    7.218150] EXT4-fs (sdc1): mounted filesystem with ordered data mode. Opts: (null)
[    7.384566] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).

The first one is the 16GB drive in question. Take the number on that line and use it in the next step:

$ echo 1 > /sys/class/scsi_device/2\:0\:0\:0/device/rescan
$ dmesg |tail
[1918441.322362] sd 2:0:0:0: [sda] 209715200 512-byte logical blocks: (107 GB/100 GiB)
[1918441.322596] sd 2:0:0:0: [sda] Cache data unavailable
[1918441.330685] sd 2:0:0:0: [sda] Assuming drive cache: write through
[1918441.489622] sda: detected capacity change from 17179869184 to 1073741824

So, that's good, it sees our increased size. Now, lets enlarge that Volume Group. First we get info about the volume group.

$ vgdisplay
  --- Volume group ---
  VG Name               myfulldisk-vg
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               15.76 GiB
  PE Size               4.00 MiB
  Total PE              4034
  Alloc PE / Size       4028 / 15.73 GiB
  Free  PE / Size       6 / 24.00 MiB
  VG UUID               dv3URd-EVvz-oTwY-WiDW-RPt1-4rbD-FnPxxM

That '6' is only 24MiB. It doesn't see our new space yet. In order to get it to, we need to make a new partition of the right type, then add it to the volume group. We'll then end up with more Free PE's. Here we go:

$ fdisk /dev/sda
Command (m for help): p

Disk /dev/sda: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders, total 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000ade37

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      499711      248832   83  Linux
/dev/sda2          501758    33552383    16525313    5  Extended
/dev/sda5          501760    33552383    16525312   8e  Linux LVM

Command (m for help): n
Partition type:
   p   primary (1 primary, 1 extended, 2 free)
   l   logical (numbered from 5)
Select (default p): p
Partition number (1-4, default 3): 3
First sector (499712-209715199, default 499712): 33552384
Last sector, +sectors or +size{K,M,G} (33552384-209715199, default 209715199): 
Using default value 209715199

Command (m for help): t
Partition number (1-5): 3
Hex code (type L to list codes): 8e
Changed system type of partition 3 to 8e (Linux LVM)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

At this point you could reboot... but we're not going to. Even though this is our root drive which makes this a little trickier, it's nothing we can't fix:

$ partprobe /dev/sda

Hopefully, partprobe has found your new partition for you and enlightened the kernel with it's wisdom (or at least fresh load of zeros). Now we need to make it an available volume to use for expanding the disk. This consists of making it a 'Physical volume', and then adding that physical volume to the Volume Group containing the disk we want to expand.

$ pvcreate /dev/sda3
  Physical volume "/dev/sda3" successfully created
$ pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda5
  VG Name               myfulldisk-vg
  PV Size               15.76 GiB / not usable 2.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              4034
  Free PE               6
  Allocated PE          4028
  PV UUID               a3mhvZ-ogyk-ao4y-2JSM-KVfL-i9no-q0LAUk
   
  "/dev/sda3" is a new physical volume of "84.00 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/sda3
  VG Name               
  PV Size               84.00 GiB
  Allocatable           NO
  PE Size               0   
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               IpyjOU-1GDy-bTLL-U9kE-iSGP-BYg1-a25LIm
   
$ vgextend /dev/myfulldisk-vg /dev/sda3
  Volume group "myfulldisk-vg" successfully extended
$ vgdisplay
  --- Volume group ---
  VG Name               myfulldisk-vg
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               99.76 GiB
  PE Size               4.00 MiB
  Total PE              25538
  Alloc PE / Size       4028 / 15.73 GiB
  Free  PE / Size       21510 / 84.02 GiB
  VG UUID               dv3URd-EVvz-oTwY-WiDW-RPt1-4rbD-FnPxxM

Awesome, we now have 21510 free PE's that we can use... That's, apparently, 84.02GB in this case. Next up, we'll need to know what portion of the VG we need to extend. Looking back up at a 'df' output, and knowing our system, it says "root" in there. doing a quick ls of /dev/myfulldisk-vg/ shows that there's only really a choice between root" and "swap". So, knowing it's root we move on with:

$ lvextend -L95G /dev/myfulldisk-vg/root
  Extending logical volume root to 95.00 GiB
  Logical volume root successfully resized
$ df -h
Filesystem                                         Size  Used Avail Use% Mounted on
/dev/mapper/myfulldisk--vg-root                     12G   11G  114M  99% /
none                                               4.0K     0  4.0K   0% /sys/fs/cgroup
udev                                               7.4G  4.0K  7.4G   1% /dev
tmpfs                                              1.5G  576K  1.5G   1% /run
none                                               5.0M     0  5.0M   0% /run/lock
none                                               7.4G     0  7.4G   0% /run/shm
none                                               100M     0  100M   0% /run/user
/dev/sda1                                          236M   37M  187M  17% /boot
/dev/sdc1                                          246G   44G  190G  19% /data

Okay, the VG might be bigger, but no one else knows that. Because the filesystem ON the VG is still the same size, luckily there's a command for that too!

$ resize2fs /dev/myfulldisk-vg/root
resize2fs 1.42.9 (4-Feb-2014)
Filesystem at /dev/myfulldisk-vg/root is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 6
The filesystem on /dev/myfulldisk-vg/root is now 24903680 blocks long.

$ df -h
Filesystem                                         Size  Used Avail Use% Mounted on
/dev/mapper/myfulldisk--vg-root                     94G   11G   79G  12% /
none                                               4.0K     0  4.0K   0% /sys/fs/cgroup
udev                                               7.4G  4.0K  7.4G   1% /dev
tmpfs                                              1.5G  576K  1.5G   1% /run
none                                               5.0M     0  5.0M   0% /run/lock
none                                               7.4G     0  7.4G   0% /run/shm
none                                               100M     0  100M   0% /run/user
/dev/sda1                                          236M   37M  187M  17% /boot
/dev/sdc1                                          246G   44G  190G  19% /data

Ha, there we go! 79GB available, enjoy!

Using Docker Behind a Network Proxy

I was working on spinning up some Docker containers (if you haven't heard of Docker, I highly recommend it) and found some difficulty getting it to pull images from behind a proxy. You can manually export the http_proxy and https_proxy settings prior to execution or fire up the Docker service from a script that exports those prior to running the Docker service, but those are both hack-like to me, so I found a better way.

On CentOS (in my case 6.5), simple edit the file docker-network in /etc/sysconfig and add the lines:

export HTTP_PROXY=http://proxy.mynetwork.net:80/
export HTTPS_PROXY=http://proxy.mynetwork.net:80/

Then restart the docker service with service docker restart and you should be able to pull down images!

To go along with that, you should be able to include username/password for proxies as you normally would, "http://<user>:<password>@<host>:<port>"

Log Rotation to NFS Mount with Logrotate.d

Logrotate.d is one of the most flexible log rotation programs for linux servers. I recently was required to rotate a lot of large files daily from the server to a central NFS Mount and wanted to share just how easy this is to accomplish.

As background, this app is a Node.js application that uses Bunyon for a logging output plugin. Luckily, it accepts a USR2 signal to tell it "hey, I just rotated your log file, make a new one and keep on truckin'". See, the first issue is that moving a file that a process is writing to, simply renames where the inode is pointing, so your process will continue writing to the new filename as if nothing happened. Telling it to make a new file is key to rotation.

The following code block is the entirety of my logrotate.d script. Placed in the /etc/logrotate.d/ folder, it will be read in and processed every night. I've used the date command to create sub dirs per month to keep things organized in the long term, and the hostname to put them each in their own directories per hostname.

/var/log/myApp/*log {
    daily
    dateext
    rotate 7
    missingok
    compress
    sharedscripts
    postrotate
        [ ! -f /var/run/myApp.pid ] || kill -USR2 `cat /var/run/myApp.pid`
    endscript
    lastaction
 mkdir -p /logsMount/myApp/`hostname`/`date +%Y\/%m`/
 mv /var/log/myApp/*gz /logsMount/myApp/`hostname`/`date +%Y\/%m`/ 
    endscript
}

As this reads, top down: For all files ending in "log" in the /var/log/myApp directory, on a daily basis, throw a date extension on them (-20150102 for instance. Not putting 'dateext' will cause it to simply number them .1, .2, etc.), keep the last 7 days (useful if the nfs mount goes down). If the files are missing don't throw an error. Compress the files (with gzip by default). As your Post Rotate command, send a signal to the process ID noted in the /var/run/myApp.pid file to tell that process to regenerate it's logs. Then, when that's done, make sure there's a directory on the target nfs mount and move all .gz files to it.

It's important to note that the 'postrotate' commans occur AFTER file rotation, but PRIOR to compression. the 'lastaction' section occurs as the last thing it does in this script, i.e. after all other things are done including compression.

This will move things like: /var/log/myApp/myLogFile.log -> /logsMount/myApp/myHost/2015/01/myLogFile.log-20150102.gz

Hope this helps someone out!

LogStash ElasticSearch Index Cleanup

LogStash is a great way to track logs from lots of different sources and store them in a central location where metrics and monitoring can occur. I've started pushing LOTS of data into our setup which uses the ElasticSearch back end. To quote their site, "ElasticSearch is a flexible and powerful open source, distributed, real-time search and analytics engine." and I think it has a really bright future... but currently it's soaking up a lot of disk space. I'm sure I'm not the only one with this issue, after all, when something can handle LOADS of data, you want to give it all you've got! So, we've got 3 hosts running ElasticSearch processes, each with 250GB of data storage. Sometimes one will start to fill up. Looking into the API, I found it's really, REALLY easy to delete old data to keep the size within requested parameters. First off, looking at LogStash's ElasticSearch plugin, it notes that by default LogStash indexes are "logstash-%{+YYYY.MM.dd}". Keeping that in mind, the following info would work for anything as long as you know the indexes you want to delete, but let's start off simple.

curl -s -XDELETE  'http://127.0.0.1:9200/logstash-2014.02.28'

That'll delete the "logstash-2014.02.28" index. I've had to connect in and do this sometimes. Great to do when you need it on demand, but we can do better. Assuming that I'm cool keeping the last 7 days up there, let's edit up a quick bash script:

#!/bin/bash
DATETODELETE=`date +%Y.%m.%d -d '7 days ago'`
curl -s -XDELETE  http://127.0.0.1:9200/logstash-${DATETODELETE}

Now, we could put that in the crontab, have it run once or twice a day and be good to go... And if you knew you could ALWAYS keep 7 days worth of data on your system, that'd be acceptable. But let's have some more fun. Let's assume that we want to keep as much as we can on our system and still keep 10% space free and that the drive we store this on is mounted on /data

#!/bin/bash
#This is about 10% of a 250GB volume (not GiB) using 1000 = k 
DESIRED=24410000
AVAIL=`df /data|grep -v Filesystem|awk '{print $4}'`
if [ $AVAIL -lt $DESIRED ] 
then
    curl -s -XDELETE 127.0.0.1:9200/`curl -s 127.0.0.1:9200/_stats?pretty|grep logstash|sort|awk -F'"' '{ print $2 }'|head -n1`
fi

Let's explain this sample a bit... First off, we set DESIRED to be the amount of "Available" space we desire the system to retain. in our case above, I calculated 10% of a 250GB drive and put that in. So if it ever starts to go below 10% remaining (90%+ used) the if statement will fire.

Next, I pull the Available space. If you take what's in the backquotes and put it on a command line, you'll see what happens. I run df, limited to just the filesystem I care about, I get rid of the line with the labels and then awk pulls out the 4th column (Avail). This number gets stored in AVAIL and we move on.

The If statement then compares the two, if DESIRED is less than AVAIL, we are bumping our limit and have something's got to give, so we run the curl... This curl is a combination of two actually. Starting inside out, we do a "curl -s 127.0.0.1:9200/_stats?pretty" which prints out a list of indexes and a bunch of cool stats about them... then we grep for logstash to get rid of all the cool stats and just keep the names, then we use sort to make sure they're in order of lowest to highest (since they have dates 2014-03-04 and such, that works) and then we use some awk magic to pull out JUST the name of the index and get rid of the other chars 'pretty' uses. That then gets placed back in the right place for the outter curl to execute a delete on it and bye, bye index!

If you put this in your crontab and run it often (it won't do anything if the drive has more than the desired available space remaining) you'll be able to maintain free space on your ElasticSearch hosts without having to set a hard limit on days to keep.

Thinking further into it, you can use the same script with different commands inside the if statement to keep free space on many other systems as well.

Running Custom SNMPd Checks in CentOS 6

I've been fighting with a problem between CentOS 5.x and CentOS 6.x SNMPD configs. In CentOS 5.x we have two lines in /etc/snmpd/snmpd.conf like the following:

exec 1.3.6.1.4.1.5001.100 mailq-check /usr/local/nagios/libexec/check_mailq -w2 -c4 -t7 
exec 1.3.6.1.4.1.5002.1 fscheck /bin/touch /opt/rocheck && /bin/touch /tmp/rocheck

They're simple commands that extend snmp. In CentOS 5.x this form is: "exec <return oid> <name> <command>" and when setup the command or script's output will be returned via snmpd to those that request the MIB/OID. All well and good, but copying this into the standard snmpd.conf file from CentOS 6 gave me nothing. running snmpd with verbose logging gave me nothing useful other than being able to see that it was being requested. Digging deeper, I found the following command and it's output:

# snmpwalk -v 2c 10.93.90.209 -c rsprod .1.3.6.1.4.1.2021.8
UCD-SNMP-MIB::extIndex.1 = INTEGER: 1
UCD-SNMP-MIB::extIndex.2 = INTEGER: 2
UCD-SNMP-MIB::extNames.1 = STRING: 1.3.6.1.4.1.5001.100
UCD-SNMP-MIB::extNames.2 = STRING: 1.3.6.1.4.1.5002.1
UCD-SNMP-MIB::extCommand.1 = STRING: mailq-check
UCD-SNMP-MIB::extCommand.2 = STRING: fscheck
UCD-SNMP-MIB::extResult.1 = INTEGER: 1
UCD-SNMP-MIB::extResult.2 = INTEGER: 1
UCD-SNMP-MIB::extOutput.1 = STRING: mailq-check: No such file or directory
UCD-SNMP-MIB::extOutput.2 = STRING: fscheck: No such file or directory
UCD-SNMP-MIB::extErrFix.1 = INTEGER: 0
UCD-SNMP-MIB::extErrFix.2 = INTEGER: 0
UCD-SNMP-MIB::extErrFixCmd.1 = STRING: 
UCD-SNMP-MIB::extErrFixCmd.2 = STRING: 

Looking at this, it doesn't look like it's running my script "/usr/local/nagios/libexec/check_mailq -w2 -c4 -t7" but attempting to run the name "mailq-check". Of course, that isn't the script name+path, so it doesn't find it. Turns out that in CentOS 6, you don't use the oid portion:

exec mailq-check "/usr/local/nagios/libexec/check_mailq -w2 -c4 -t7 -Mpostfix"
exec fscheck "/bin/touch /opt/rocheck && /bin/touch /tmp/rocheck"

Which, when called with the same snmpwalk above, comes back as:

UCD-SNMP-MIB::extIndex.1 = INTEGER: 1
UCD-SNMP-MIB::extIndex.2 = INTEGER: 2
UCD-SNMP-MIB::extNames.1 = STRING: mailq-check
UCD-SNMP-MIB::extNames.2 = STRING: fscheck
UCD-SNMP-MIB::extCommand.1 = STRING: /usr/local/nagios/libexec/check_mailq -w2 -c4 -t7 -Mpostfix
UCD-SNMP-MIB::extCommand.2 = STRING: /bin/touch /opt/rocheck && /bin/touch /tmp/rocheck
UCD-SNMP-MIB::extResult.1 = INTEGER: 0
UCD-SNMP-MIB::extResult.2 = INTEGER: 0
UCD-SNMP-MIB::extOutput.1 = STRING: OK: mailq reports queue is empty|unsent=0;2;4;0
UCD-SNMP-MIB::extOutput.2 = STRING: 
UCD-SNMP-MIB::extErrFix.1 = INTEGER: 0
UCD-SNMP-MIB::extErrFix.2 = INTEGER: 0
UCD-SNMP-MIB::extErrFixCmd.1 = STRING: 
UCD-SNMP-MIB::extErrFixCmd.2 = STRING: 

Now, let me explain what all of these MIBs mean.

• extIndex.#
  The Index number of the command
• extNames.#
  The Name you gave the command. This is the first parameter you pass to exec
• extCommand.#
  The full command you had SNMPd call to return you this info. Second parameter you passed to exec
• extResult.#
  Exit Code from the command exec'd
• extOutput.#
  Raw output of the command exec'd
• extErrFix.#
  This is a bit that can be flipped by the client. Flipping this to a 1 kicks off the extErrFixCmd by SNMPd. Generally this is used to 'fix' and 'error' condition.
• extErrFixCmd.#
  The command to be executed by the SNMPd server

Restoring MS SQL Analysis Server Database or Cube with Specific ID

Restoring a MS SQL Database in 2008 and later is really easy, but I recently found that it won't always result in the Database having the ID you want it to. In my case, I needed to restore the database next to one that already existed. Upon doing so, it automatically changed the ID of the database to match that of the name I gave it. This is nice, but not what I wanted. I have a lot of jobs that refer to the database by it's ID, and you can't change an ID once it's in place. Furthermore, the restore dialog doesn't give you the option to specify your ID either, however, the Script button does... XMLA Scripts to the rescue!

Step Through It

1. Open up Management Studio
2. Connect to your Analysis Services Server
3. Right Click on the Databases and hit "Restore..."
4. When the Restore window comes up, fill in as much info as you can, but instead of hitting "OK", hit the "Script" button at the top. A new script will open behind your window. Close the window and go to the script, it should look something like this.

<Restore xmlns="http://schemas.microsoft.com/analysisservices/2003/engine">
  <File>\\Gibson\Awesome\AnalysisServices\Awesome_Data\Awesome_Data_20140201030032.abf</File>
  <DatabaseName>Awesome Data New</DatabaseName>
 <DatabaseID>Awesome ID</DatabaseID>
  <AllowOverwrite>true</AllowOverwrite>
</Restore>

Finally, Add that DatabaseID line and Hit F5 to run the restore!

When it's done you should have a newly restored Cube/Database, with the ID you were hoping for!