Error Performing Checksum

Getting something like this when running an update from your repo?

MyRepo/primary http://repo.example.com/repo/CentOS/5.8/x86_64/repodata/primary.xml.gz: [Errno -3] Error performing checksum

Well, it seems CentOS 5.x requires sha, not sha256. So when you run createrepo, make sure to change it to 'createrepo --checksum=sum' instead.

Trac: Fixing the dreaded "Duplicate entry" Error during SVN Rescan

So, The other day I was minding my own business, resyncing an SVN repository in trac and I ran across this error:

File "/usr/bin/trac-admin", line 7, in ? sys.exit(
  File "/usr/lib/python2.4/site-packages/Trac-0.11.6-py2.4.egg/trac/admin/console.py", line 1325, in run admin.run()
  File "/usr/lib/python2.4/site-packages/Trac-0.11.6-py2.4.egg/trac/admin/console.py", line 155, in run self.cmdloop()
  File "/usr/lib64/python2.4/cmd.py", line 142, in cmdloop stop = self.onecmd(line)
  File "/usr/lib/python2.4/site-packages/Trac-0.11.6-py2.4.egg/trac/admin/console.py", line 138, in onecmd rv = cmd.Cmd.onecmd(self, line) or 0
  File "/usr/lib64/python2.4/cmd.py", line 219, in onecmd return func(arg)
  File "/usr/lib/python2.4/site-packages/Trac-0.11.6-py2.4.egg/trac/admin/console.py", line 680, in do_resync repos = env.get_repository().sync(self._resync_feedback)
  File "/usr/lib/python2.4/site-packages/Trac-0.11.6-py2.4.egg/trac/versioncontrol/cache.py", line 214, in sync (str(next_youngest),
  File "/usr/lib/python2.4/site-packages/Trac-0.11.6-py2.4.egg/trac/db/util.py", line 64, in execute return self.cursor.execute(sql_escape_percent(sql), args)
  File "/usr/lib64/python2.4/site-packages/MySQLdb/cursors.py", line 163, in execute    self.errorhandler(self, exc, value)
  File "/usr/lib64/python2.4/site-packages/MySQLdb/connections.py", line 35, in defaulterrorhandler raise errorclass, errorvalue
_mysql_exceptions.IntegrityError: (1062, "Duplicate entry '37295-releases/may0712-D' for key 'PRIMARY'")

How did we get here?!

As background, this is a copy of a repository we've been working with for a while. We're copying it to a new repository as we have two sets of files in it that were originally connected projects but have grown farther and farther apart to the point where they don't really reference each other or need to be together. At one point we had a few people who somehow checked in duplicate deletions of the same file in the original repository. (How that got past SVN is another story, but it wasn't anything out of the ordinary). In any case, this isn't the first time I've run across this error and since we made a copy of the repo that had that issue, we now have it here.

Now, if this was the very last revision and the repo was quiescent, you could do a little clever hacking on the svn repo to remove the duplication... but that's not possible to do here. No, instead we're going to get MySQL Crafty.

Time for Mysql Sluthing

First off, I open up my trusty commandline mysql client and pick my database. Looking at the list of tables you'll see the "revision" and "node_change" tables which look like this:

mysql> describe revision;
+---------+---------+------+-----+---------+-------+
| Field   | Type    | Null | Key | Default | Extra |
+---------+---------+------+-----+---------+-------+
| rev     | text    | NO   | PRI | NULL    |       |
| time    | int(11) | YES  | MUL | NULL    |       |
| author  | text    | YES  |     | NULL    |       |
| message | text    | YES  |     | NULL    |       |
+---------+---------+------+-----+---------+-------+
4 rows in set (0.00 sec)

mysql> describe node_change;
+-------------+------+------+-----+---------+-------+
| Field       | Type | Null | Key | Default | Extra |
+-------------+------+------+-----+---------+-------+
| rev         | text | NO   | PRI | NULL    |       |
| path        | text | NO   | PRI | NULL    |       |
| node_type   | text | YES  |     | NULL    |       |
| change_type | text | NO   | PRI | NULL    |       |
| base_path   | text | YES  |     | NULL    |       |
| base_rev    | text | YES  |     | NULL    |       |
+-------------+------+------+-----+---------+-------+

A Plan Comes Together

'rev' is a Primary Key (or part of one) in both cases and, according to the message above, we're trying to insert the same primary key twice. So how are we going to fix this? If I delete the offending records, it'll rescan the repo up to that point, see that it's not there, add it, then add the second one. So, that doesn't help us at all. We need to leave it there but not let it complain the next time. If we leave it there but drop the unique requirement, it'll write it for the first rev, and then write over it again with the second rev, or (if it doesn't look for an existing record first) it'll just add the line twice. In the latter case, we can clean that up later and then re-institute the unique requirement. Sounds like a win, let's do it.

Dropping Constraints

mysql> alter table revision drop primary key;
Query OK, 37295 rows affected (0.11 sec)
Records: 37295  Duplicates: 0  Warnings: 0

mysql> alter table node_change drop primary key;
Query OK, 328916 rows affected (3.13 sec)
Records: 328916  Duplicates: 0  Warnings: 0


mysql> describe revision;
+---------+---------+------+-----+---------+-------+
| Field   | Type    | Null | Key | Default | Extra |
+---------+---------+------+-----+---------+-------+
| rev     | text    | NO   |     | NULL    |       |
| time    | int(11) | YES  | MUL | NULL    |       |
| author  | text    | YES  |     | NULL    |       |
| message | text    | YES  |     | NULL    |       |
+---------+---------+------+-----+---------+-------+
4 rows in set (0.00 sec)

mysql> describe node_change;
+-------------+------+------+-----+---------+-------+
| Field       | Type | Null | Key | Default | Extra |
+-------------+------+------+-----+---------+-------+
| rev         | text | NO   | MUL | NULL    |       |
| path        | text | NO   |     | NULL    |       |
| node_type   | text | YES  |     | NULL    |       |
| change_type | text | NO   |     | NULL    |       |
| base_path   | text | YES  |     | NULL    |       |
| base_rev    | text | YES  |     | NULL    |       |
+-------------+------+------+-----+---------+-------+
6 rows in set (0.00 sec)

Alright! Let's resync again... just running "tracadmin resync"... And watching it count... for a while. Eventually we'll see something like:

37309 revisions cached.
Done.

Searching for UFO's

So, now we've got all that data in the db... including the duplicate revision. We can't just add back the keys because it'll fail the constraint validation. (If you run "alter table node_change add primary key (rev(16), path(512), change_type(1));" you'll get "ERROR 1062 (23000): Duplicate entry '32152-branches/...' for key 'PRIMARY'".We're going to have to clear that extra one out first, so let's find it.

mysql> select count(*) as c,rev,path,change_type from node_change group by concat(rev, path, change_type) order by c desc limit 10;
+---+-------+---------------------------------+-------------+
| c | rev   | path                            | change_type |
+---+-------+---------------------------------+-------------+
| 2 | 37295 | releases/May0712                | D           |
| 1 | 33277 | releases/S2611/myApp/main.cpp   | E           |
| 1 | 33277 | releases/S2611/myApp            | E           |
...

Well, there she is, 'c'=2, so we've got two of the same thing, just like we thought. I put a limit of 10 instead of only pulling the top result so we could make sure that there were no other problem entries pulled in after that one during the sync. If there were, we'd have to do the following steps for each.

Dropping the Dupes

mysql> select * from node_change where rev=37295 and path='releases/May0712';
+-------+------------------+-----------+-------------+------------------+----------+
| rev   | path             | node_type | change_type | base_path        | base_rev |
+-------+------------------+-----------+-------------+------------------+----------+
| 37295 | releases/May0712 | D         | D           | releases/May0712 | 37294    |
| 37295 | releases/may0712 | D         | D           | releases/may0712 | 37294    |
+-------+------------------+-----------+-------------+------------------+----------+
2 rows in set (0.20 sec)

mysql> delete from node_change where rev=37295 and path='releases/may0712' limit 1;
Query OK, 1 row affected (0.19 sec)

mysql> select * from node_change where rev=37295 and path='releases/may0712';
+-------+------------------+-----------+-------------+------------------+----------+
| rev   | path             | node_type | change_type | base_path        | base_rev |
+-------+------------------+-----------+-------------+------------------+----------+
| 37295 | releases/may0712 | D         | D           | releases/may0712 | 37294    |
+-------+------------------+-----------+-------------+------------------+----------+
1 row in set (0.24 sec)

You can see that MySQL honors the 'limit' clause on deletes, saving us a little work here. Now we should be able to add our constraint back on.

mysql> alter table node_change add primary key (rev(16), path(512), change_type(1));
Query OK, 329329 rows affected (6.94 sec)
Records: 329329  Duplicates: 0  Warnings: 0

mysql> alter table revision add primary key (rev(16));
Query OK, 37309 rows affected (0.29 sec)
Records: 37309  Duplicates: 0  Warnings: 0

And we're done!

PHP 5.4 Install on CentOS 5.x

So, PHP 5.4 is out now, and I figured it time I upgraded from the PHP that comes with CentOS 5.x (PHP 5.1.x!) to something modern. After all, 5.3+ gives you lots of benefits and years of bug killing and hole plugging. I pulled down the php 5.4 tarball from php.net and got to work compiling. Starting with what I had from a php -i, and fixing things in the configure line I ended up using:

./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --target=x86_64-redhat-linux-gnu --program-prefix= --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info --cache-file=../config.cache --with-libdir=lib64 --with-config-file-path=/etc --with-config-file-scan-dir=/etc/php.d --disable-debug --with-pic --disable-rpath --without-pear --with-bz2 --with-curl --with-freetype-dir=/usr --with-png-dir=/usr --enable-gd-native-ttf --without-gdbm --with-gettext --with-gmp --with-iconv --with-jpeg-dir=/usr --with-openssl --with-pspell --with-pcre-regex=/usr --with-zlib --with-layout=GNU --enable-exif --enable-ftp --enable-sockets --enable-sysvsem --enable-sysvshm --enable-sysvmsg --enable-trans-sid --enable-wddx --enable-shmop --enable-calendar --with-libxml-dir=/usr --enable-pcntl --with-imap=shared --with-imap-ssl --enable-mbstring=shared --enable-mbregex --with-ncurses=shared --with-gd=shared --enable-bcmath=shared --enable-dba=shared --with-xmlrpc=shared --with-mysql=/usr --with-mysqli=/usr/bin/mysql_config --enable-dom=shared --with-snmp=shared,/usr --enable-soap=shared --with-xsl=shared,/usr --enable-xmlreader=shared --enable-xmlwriter=shared --with-pdo-mysql=/usr/bin/mysql_config --with-kerberos --with-sapi --with-apxs2

This got me everything I needed, except that I realized that I then lost memcache. I did a bit of searching and I couldn't yet find a memcache extension for 5.4 (this will probably be fixed in the future). I finally found this site http://www.sohailriaz.com/how-to-install-memcached-with-memcache-php-extension-on-centos-5x/ which went into detail on how to install memcached from scratch which I didn't need to do... but it had one section that I'd like to echo here. How to use pecl and phpize to get yourself a shiny new memcache.so file for 5.4. Basically run:

wget http://pecl.php.net/get/memcache-2.2.5.tgz

then untar it and go into the directory. Simply running:

phpize
./configure
make && make install

Will create a configure script, configure the extension, make it (make sure you have gcc, but I assume if you compiled php that you do), and copy it all to where it needs to go! You may have to change where you're memcache.ini file points to for the extension or something similar, but this should get you back in business!

Fixing Ethernet Device Creep

In CentOS, udev remembers various things about your network adapters from boot to boot, most importantly, the MAC address. The reasoning is simple: Boot once, get it setup, shut it down, add a second NIC, boot back up, now which one is supposed to have which address? Generally the idea is this, udev keeps the adapter MAC associated with the device name (so ::08 get's eth0, and ::2b gets eth1. remove ::08 and you'll still have eth1). Then the network scripts in /etc/sysconfig/network-scripts/ named ifcfg- are read by /etc/init.d/network on start up and it associates eth0 -> IP1 eth1 -> IP2.

Problem is, say you're not really set on the MAC you've got (it's a VM and you cloned it, clones get a new MAC. It's a physical box and that card died, replace it and it's got a new MAC). How do you get that coveted eth0 label back so you don't have to change a lot of things**? Well, to quote Dr. Seuss "We can do it, we know how".

Pull yourself a quick ifconfig listing. Note the MAC address of the NIC you want to be, say, eth0 in this case.

[root@mysystem ~] # ifconfig |grep HWaddr
eth1 Link encap:Ethernet HWaddr 00:50:56:88:00:74
eth2 Link encap:Ethernet HWaddr 00:50:56:88:00:75
eth3 Link encap:Ethernet HWaddr 00:50:56:88:00:7b

In this case, there used to be an eth0 and when I cloned this box, I got eth3. Let's change that.

[root@mysystem ~] # cat /etc/udev/rules.d/70-persistent-net.rules
# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:88:00:74", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:88:00:75", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:88:00:7b", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"

What we need to do is edit this file and change that last 'NAME="eth3"' to 'NAME="eth0"'.

Now, I'd say reboot, but you need to do one more check first... Make sure you have something in /etc/sysconfig/network-scripts/ifcfg-eth0 and make sure of two things, if you have a Device= line that it says eth0 (In case you thought you'd be clever by copying ifcfg-eth3 to ifcfg-eth0 and rolling with it). Also, make sure if it has a HWADDR= line that it equals the new MAC and not the old one. Truthfully, since udev does the association from MAC->device name, this line isn't necessary and I almost always comment it out (# at the front of the line works in this file), but CentOS's "system-config-network" script adds it in there, so watch out. Okay, now reboot!

** I know, this doesn't seem like that big of a deal, but if you have a complex setup of many NICs and/or are using a monitoring program to watch traffic on a specific NIC, you now have more to change. Especially if there's a group of servers that match and now this one doesn't, etc.

GeoIP in CentOS 6.x

Simple one today... So our nginx rpm requires GeoIP. It's useful, and its also been a standard part of CentOS for a bit now, but apparently it's not in CentOS 6. To get it, simply run

rpm -Uvh http://download.fedora.redhat.com/pub/epel/6/x86_64/epel-release-6-5.noarch.rpm

This will add in the right repo to include it in your yum searches. Then you can just run

yum install geoip

and you'll be good to go!

SSLCertificateFile errors in Apache httpd-2.2.21

Apache 2.2.21 is out, so, being the 'on the ball' sys-admin I am, I pulled it down and rolled myself an rpm using rpm build (which I should probably make a post on later). After a few minutes of compiling, I had my spiffy rpm and I was ready to deploy it.
I first pushed it to all of our beta sites as they most closely matched our live servers and got them running in no time with a little 'rpm -Uvh httpd-2.2.21.rpm' command. After a week of that running well, I pushed it to our dev server this morning and ran into a problem... Where I previously had:

SSLEngine on
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
SSLCertificateFile /etc/httpd/conf/ssl.crt/secure_mysite_com.crt
SSLCertificateKeyFile /etc/httpd/conf/ssl.key/secure_mysite_com.key

in our vhost config, it no longer allowed me to start httpd. Looked in the error log and I found:

[Tue Nov 08 09:38:46 2011] [error] Server should be SSL-aware but has no certificate configured [Hint: SSLCertificateFile] ((null):0)

Of course my first response was "But I DO!". I dug around and found many places where it was defined, always correct. After paring down our long list of vhosts a bit, I was able to get one to work. Apparently it had the following lines:

SSLEngine on
SSLCipherSuite HIGH:!SSLv2:!ADH:!aNULL:!eNULL:!NULL
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCertificateFile /etc/httpd/conf/ssl.crt/secure_mysite_com.crt
SSLCACertificateFile /etc/httpd/conf/ssl.crt/intermediate_bundle.crt
SSLCertificateKeyFile /etc/httpd/conf/ssl.key/secure_mysite_com.key

I don't know why I had to explicitly define my CACert on top of the others, but this made it happy and I was able to go along my way!

I found out after rolling this to a few other servers that it's also hypersensitive to Vhosts that are on port 443 but don't have ssl configs. Make sure you have the above lines or an include file with them in every vhost you have configured to run on port 443!

Date Changes on XenServer VMs

If you've ever tried to run 'date -s' on a virtual machine running on a XenServer host, you'll have no doubt noticed that it returns the right date, only to be back to it's previous setting the second after. This has to do with the kernel level date sync between the XenServer Host and it's Virtual Machines. While this is very handy as it means that all servers have their date set by the hardware node without having to run ntp in all of the virtual machines in your setup (though running ntp between XenServer Hosts in a pool is recommended), it means you can't update a single vm's date to something other than that of the XenServer Host. Luckily, this is simple to fix!

To allow a single VM to have a unique clock, log into that VM and run:

echo 1 > /proc/sys/xen/independent_wallclock

And to change it back:

echo 0 > /proc/sys/xen/independent_wallclock

I've tested this on Citrix XenServer 5.5 and 5.6. If you're going to leave it like this for any decent length of time, I'd Highly recommend setting up an NTP client on the VM to keep thing's synced somewhere. If it's a completely weird time setting, yet based on real time, you can always have a script on a host with NTP run an ssh command hourly to set the time based on your offset.