Ganeti is open source software from Google to manage a cluster of linux hosts running virtual machines using the Xen or KVM hypervisors.  I was attracted to it because it handled everything from creating new instances to managing network disk mirroring (via DRBD) to help with instance availability.  It even supports migrating a running instance from one host to another with one short command line.  Now that I have it working, its possible to have a new Ubuntu vm running in about 5 minutes, probably less if I don’t need disk mirroring.

That’s the good part, the bad part is that it is developed to run Debian instances on Debian hosts with the Xen hypervisor.  Ok, that’s not the bad part, the bad part is that even though it works with other linux guests and hosts and the with KVM hypervisor, there are a number of hitches and gaps, even when using the Debian-derviced Ubuntu Linux distribution.  I still need to update an earlier post I made on trying to get it to work on Ubuntu to reflect what I finally did to get things working, but I wanted to share a big piece of the work I did.

Ganeti relies on something called debootstrap + some wrapper scripts to do the work of creating a new instance.  On Ubuntu, the provided scripts can even be used to create a new Ubuntu instance, but I didn’t find the experience satisfactory.  The provided scripts don’t install a boot-loader (like Grub) on the new instance, so you are required to tell the KVM hypervisor load an external kernel.  I found this complicated things.  With an external kernel, you have to be careful that your vm instance has the correct kernel module versions installed and you have to coordinate updates and upgrades more carefully.  I wanted the running virtual machines to act, as much as possible, like physical machines, which at this point means updating the kernel via aptitude or apt-get.  So, I set out to modify the scripts provided for use with debootstrap to instead take advantage of a script that Ubuntu provides for creating virtual machines called vmbuilder.

It was a little harder to get things working than expected.  For some reason vmbuilder on Jaunty Jackelope (Ubuntu 9.04) had trouble partitioning a mapped drbd device when run inside my wrapper script.  I ended up working around it by creating a temporary loopback disk and then copying the image to the drbd device.  It adds some time, but it works (for me, at least).  There are other kludgy bits, like the fact that it just fails if the provided disk device is smaller than 5GB because I accept vmware’s default of 4GB for the filesystem and 1GB for swap (which is probably too much).  Related to that is the fact that anything larger than 5GB is left unused.

I’m sure there are other problems too and I hope that if you find and fix them, you’ll pass your fixes back to me so I can update the scripts.

So, without further adieu, here are the files (ganeti-intance-vmbuilder-r1).  I started with the files installed by ganeti-instance-debootstrap-0.7.tar.gz and altered the create and and common.sh scripts, along with the config file, which is used for chosing which ubuntu release to use, whether to create a 32 or 64-bit instance and which packages to install.

To use, first install the ‘ubuntu-vm-builder’ package using ‘apt-get’ or ‘aptitude’, then download the tarball of my scripts, extract it, move the extracted vmbuilder directory to /srv/ganeti/os/vmbuilder and then move, copy or symlink the ganeti-instance-vmbuilder config file to /etc/default/ganeti-instance-vmbuilder and edit it to your liking.  (This assumes you’ve used the paths suggested in the ganeti installation documentation).

Once you’ve done so, you should be able to install the ubuntu-vm-builder package specify ‘–os-type=vmbuilder’ when you add a new instance.