Openstack compute node cleanup

I’ve never used openstack before, which I imagine is similar to many other people out there. Its actually pretty cool, although I encountered a problem the other day that I think is worthy of some more documentation. Openstack runs virtual machines for users, in much the same manner as Amazon’s EC2 system. These instances are started with a base image, and then copy on write is used to write differences for the instance as it changes stuff. This makes sense in a world where a given machine might be running more than one copy of the instance.

However, I encountered a compute node which was running low on disk. This is because there is currently nothing which cleans up these base images, so even if none of the instances on a machine require that image, and even if the machine is experiencing disk stress, the images still hang around. There are a few blog posts out there about this, but nothing really definitive that I could find. I’ve filed a bug asking for the Ubuntu package to include some sort of cleanup script, and interestingly that led me to learn that there are plans for a pretty comprehensive image management system. Unfortunately, it doesn’t seem that anyone is working on this at the moment. I would offer to lend a hand, but its not clear to me as an openstack n00b where I should start. If you read this and have some pointers, feel free to contact me.

Anyways, we still need to cleanup that node experiencing disk stress. It turns out that nova uses qemu for its copy on write disk images. We can therefore ask qemu which are in use. It goes something like this:

    $ cd /var/lib/nova/instances
    $ find -name "disk*" | xargs -n1 qemu-img info | grep backing | \
      sed -e's/.*file: //' -e 's/ .*//' | sort | uniq > /tmp/inuse

/tmp/inuse will now contain a list of the images in _base that are in use at the moment. Now you can change to the base directory, which defaults to /var/lib/nova/instances/_base and do some cleanup. What I do is I look for large image files which are several days old. I then check if they appear in that temporary file I created, and if they don’t I delete them.

I’m sure that this could be better automated by a simple python script, but I haven’t gotten around to it yet. If I do, I will be sure to mention it here.

Via M10000, video, and a Belkin wireless USB thing

I’ve had a Via Eden M10000 literally lying around since soon after I moved to the US two years ago, and I finally decided that it was time for a full blown home workstation the other day, having got fed up with the small screen on my laptops. Sometimes I really miss my 30 inch monitor at work, and the 24 inch I am using at home is a much better substitute than a laptop screen.

Anyway, getting the machine working was surprisingly difficult given its a three or four year old design. The problems:

  • The PCI 802.11g wireless card didn’t work. In fact, the machine wouldn’t boot with it installed. I suspect this was a PCI version problem, as I have had pain with this card in the past.
  • The PCI 802.11b wireless card I tried next wasn’t much better. The connection would drop out randomly, and the machine would occasionally lock up. This was the card I used as my first access point about six years ago (using hostap), so perhaps its just old. It got swapped out as well.
  • The Belkin USB 802.11g thingie didn’t work reliably. It would stay connected to the network for five minutes before something went wrong. This made me annoyed, especially when it turns out this is because the latest release of Ubuntu (gutsy) installs an old version of the rt73usb driver, which is known not to work with this card. Following these instructions from the canonical wiki which tell you to install drivers from here fixed the problem. Its annoying that Ubuntu ships with known broken drivers though.
  • Next, video. The video card built into the mother board sucks. I’m running a t24 inch LCD at 1920×1200, and there was significant ghosting on the monitor. Additionally, I couldn’t run at 24 bit, I had to use 16 bit because the video card was running out of RAM bandwidth.
  • Not to worry, I installed a Matrox G450 I had lying around, and now the monitor works nicely as well, with no ghosting.
  • Finally, the whole thing is much louder than I expected from an Eden machine (there is a fan on the board, a fan in the case, and the hard disk). Then again, given I built the entire machine for $90, I can’t complain too much.

I got MythBurn working!

It took a few hours, but I have MythBurn (a web interface for creating DVDs with MythTV) finally working. The problem was that the mjpegtools shipped with Ubuntu Dapper crashes with an invalid free(). Compiling from source fixed that.

Update: I might have lied. I now have troubles getting Xine to playback the DVD when I mount the ISO image through a loop back. I don’t have any DVD media handy, so I will have to wait a couple of days until I can do more testing. The symptom I see is that the intro MythTV screen plays, and then Xine freezes before displaying the menu.

Compiling MythTV 0.19 for Ubuntu Dapper

I’ve been finding that MythTV is really unreliable after dist-upgrading from Breezy to Dapper. In fact, it’s been crashing six or seven times a day, which has been really annoying. I figured I though therefore bight the bullet and upgrade to the latest stable version of MythTV. I was previously running a SVN snapshot because it fixed a couple of bugs I was having when I first installed Myth.

Installation was pretty easy. First, grab the one or two libraries you’re missing and that the configure script doesn’t necessarily detect the dependency for properly:

    sudo apt-get install mesa-common-dev libgl1-mesa-dev libglu1-mesa-dev \
    libqt3-mt-dev libqt4-dev mesa-common-dev qt3-apps-dev libglu1-mesa fftw3 \
    fftw3-dev libsdl1.2-dev festival festival-dev libdvdread3 libdvdread3-dev

And then the usual configure, make, make install in the mythtv and mythplugins directories.

Update on the Dell e310

I’ve just realised that I never got around to commenting on how to get the Dell working with Ubuntu. Here’s what I did (I’m hoping I remember it all correctly):

  • Setup network booting
  • Changed the boot config for the network boot to automatically boot into the right install after a small timeout
  • Disabled the onboard USB controller
  • Installed a random PCI USB controller
  • Rebooted the target machine
  • The installer now netboots, and goes into the kernel with no interaction from the keyboard
  • Install like normal

So, I just leave the onboard USB controller disabled, and use the PCI one.

Update on the Dell

A previous entry discusses the Ubuntu installer stalling on my new Dell Dimension e310. In that post I decided that it was the USB controller causing pain, but the plot has thickened…

Knoppix: has no problems booting on the machine.

Debian’s Stable: the installer boots fine, but DHCP doesn’t work and it cant find the SATA disk.

The current Debian Testing: the installer freezes just like the Ubuntu one.

So, perhaps it’s time for a chroot install of Debian from Knoppix? Although qtparted doesn’t work on Knoppix properly, so I might have to do the filesystem resizing by hand.

Installing Linux on a Dell Dimension e310

I got a very good price on a Dell e310, so I have been trying to install Linux on it for about an hour now without much joy. It turns out (or at least this is my current theory) that the USB chipset in the machine isn’t supported by the Ubuntu 5.10 installer, and that causes the kernel to crash on boot. There certainly seem to be other people experiencing this as well.

Unfortunately, I really need USB on this machine, as there are no legacy ports and I really like keyboards. Specifically, the Ubuntu install CD needs me to enter a boot line before it boots the installer kernel, and I can’t do that with my spare USB controller installed and the on board one disabled, as the BIOS wont allow me.

I’ll keep you posted when I come up with a solution.

TECRA M4 P-M 770 and Ubuntu live CDs

My work machine at the moment is a Toshiba Tecra M4 tablet, and I figured I’d give a Ubuntu live CD a go over the week and play with setting up this disc array. I only have a Ubuntu live CD for version 5.04 lying around, so that’s what I used.

The machine booted fine, once I’d disconnected the USB external keyboard, which stopped the hardware detection from working for some reason. Dunno why. In fact plugging the USB keyboard in at any time killed the X server so that the mouse and keyboard never worked again. Odd.