Tag: computing

Further adventures with base images in OpenStack

I was bored over the New Years weekend, so I figured I’d have a go at implementing image cache management as discussed previously. I actually have an implementation of about 75% of that blueprint now, but its not ready for prime time yet. The point of this post is more to document some stuff I learnt about VM startup along the way so I don’t forget it later.

So, you want to start a VM on a compute node. Once the scheduler has selected a node to run the VM on, the next step is the compute instance on that machine starting the VM up. First the specified disk image is fetched from your image service (in my case glance), and placed in a temporary location on disk. If the image is already a raw image, it is then renamed to the correct name in the instances/_base directory. If it isn’t a raw image then it is converted to raw format, and that converted file is put in the right place. Optionally, the image can be extended to a specified size as part of this process.

Then, depending on if you have copy on write (COW) images turned on or not, either a COW version of the file is created inside the instances/$instance/ directory, or the file from _base is copied to instances/$instance.

This has a side effect that had me confused for a bunch of time yesterday — the checksums, and even file sizes, stored in glance are not reliable indicators of base image corruption. Most of my confusion was because image files in glance are immutable, so how come they differed from what’s on disk? The other problem was that the images I was using on my development machine were raw images, and checksums did work. It was only when I moved to a slightly more complicated environment that I had enough data to work out what was happening.

We therefore have a problem for that blueprint. We can’t use the checksums from glance as a reliable indicator of if something has gone wrong with the base image. I need to come up with something nicer. What this probably means for the first cut of the code is that checksums will only be verified for raw images which weren’t extended, but I haven’t written that code yet.

So, there we go.

Openstack compute node cleanup

I’ve never used openstack before, which I imagine is similar to many other people out there. Its actually pretty cool, although I encountered a problem the other day that I think is worthy of some more documentation. Openstack runs virtual machines for users, in much the same manner as Amazon’s EC2 system. These instances are started with a base image, and then copy on write is used to write differences for the instance as it changes stuff. This makes sense in a world where a given machine might be running more than one copy of the instance.

However, I encountered a compute node which was running low on disk. This is because there is currently nothing which cleans up these base images, so even if none of the instances on a machine require that image, and even if the machine is experiencing disk stress, the images still hang around. There are a few blog posts out there about this, but nothing really definitive that I could find. I’ve filed a bug asking for the Ubuntu package to include some sort of cleanup script, and interestingly that led me to learn that there are plans for a pretty comprehensive image management system. Unfortunately, it doesn’t seem that anyone is working on this at the moment. I would offer to lend a hand, but its not clear to me as an openstack n00b where I should start. If you read this and have some pointers, feel free to contact me.

Anyways, we still need to cleanup that node experiencing disk stress. It turns out that nova uses qemu for its copy on write disk images. We can therefore ask qemu which are in use. It goes something like this:

    $ cd /var/lib/nova/instances
$ find -name "disk*" | xargs -n1 qemu-img info | grep backing | \
  sed -e's/.*file: //' -e 's/ .*//' | sort | uniq > /tmp/inuse

/tmp/inuse will now contain a list of the images in _base that are in use at the moment. Now you can change to the base directory, which defaults to /var/lib/nova/instances/_base and do some cleanup. What I do is I look for large image files which are several days old. I then check if they appear in that temporary file I created, and if they don’t I delete them.

I’m sure that this could be better automated by a simple python script, but I haven’t gotten around to it yet. If I do, I will be sure to mention it here.

Looking for Women studying computing in Australia

I was in an unrelated meeting at work today, and it came up that the first annual Anti Borg scholarship is closing it’s application window in a few days. I thought it was worth mentioning here, in case there are people who are interested in applying. The basic deal is:

Dr. Anita Borg (1949 – 2003) devoted her adult life to revolutionizing the way we think about technology and dismantling barriers that keep women and minorities from entering computing and technology fields. Her combination of technical expertise and fearless vision continues to inspire and motivate countless women to become active participants and leaders in creating technology.

As part of Google’s ongoing commitment to furthering Anita’s vision, we are pleased to announce the 2006 Google Australia Anita Borg Scholarship. Through the scholarship, we would like to encourage women to excel in computing and technology and become active role models and leaders.

Scholarships will be awarded based on the strength of candidates’ academic background and demonstrated leadership. A group of female undergraduate and postgraduate student finalists will be chosen from the applicant pool. The scholarship recipients, selected from the finalists, will each receive a $5,000 AUD scholarship for the 2007 academic year.

All finalists will be invited to visit Google Sydney in November 2006 for a networking retreat. It will include workshops with a series of speakers, breakout sessions and social activities. The visit is meant to be an opportunity for all finalists to meet and share their experiences.

If that sounds interesting to you and you’re a lady studying computing or a related field, then you should checkout the announcement page before it’s too late.