OpenStack – Page 7 – Made by Mikal

Merged in Havana: configurable iptables drop actions in nova

Post author:mikal
Post published:April 1, 2013
Post category:OpenStack

LaunchPad bug 1013893 asked nicely if the drop action for iptables rules created by nova-network could be configured. The idea here is that you might want to do something other than a plain old drop -- for example logging before dropping. This has now been implemented in Havana. To configure the drop action, set the iptables_drop_action to the name of an already existing iptables target. Creating this target is not managed by nova, and you'll need to do it on every compute node. When iptables creates or deletes rules on compute nodes it will now use this new target. There's a bit of an upgrade problem here in that this will stop nova from deleting rules which use the old hard coded drop target. However, if an instance is torn down then all of its tables are torn down as well and rules will be deleted correctly, so this is only a problem if a security group is changed while the instance is running. It occurs to me that we can do better here, so I've sent off this review to handle the case where a rule is being removed and used the default drop action. For safety, I would…

Upgrade problems with the new Fixed IP quota

Post author:mikal
Post published:March 31, 2013
Post category:OpenStack

In the last few weeks a new quota has been added to Nova covering Fixed IPs. This was done in response to LaunchPad bug 1125468, which was disclosed as CVE 2013-1838. To be honest I think there are some things the vulnerability management team learned the hard way with this disclosure. For example, we didn't realize that we needed to update python-novaclient to allow users to set the quota, or that adding a quota would require changes in Horizon. Both of these errors have been corrected. More importanly, the default value of the new quota was set to 10. I made this decision based on the default value of the instances quota coupled with a desire to protect deployments from denial of service. However, this decision combined with a failure to explicitly call out the new quota in the release notes for the Folsom stable release have resulted in some deployers experiencing upgrade problems. This was drawn to our attention by LaunchPad bug 1161190. We have therefore moved to set the default quota for fixed IPs to unlimited. If you want to protect yourself from a potential DoS, then you should seriously consider changing this default value in your deployment.…

Havana Nova PTL elections

Post author:mikal
Post published:March 14, 2013
Post category:OpenStack

This is just a quick reminder that there are only a couple more days to vote in the Nova PTL elections for the Havana cycle. If you're eligible to vote, you should have a voting URL in your email. The candidates: Russell Bryant -- announcement Michael Still -- announcement The incumbent PTL, Vishvananda Ishaya, has chosen not to run.

OpenStack at linux.conf.au 2013

Post author:mikal
Post published:January 5, 2013
Post category:OpenStack

As some of you might know, I'm the Director for linux.conf.au 2013. I've tried really hard to not use my powers for evil and make the entire conference about OpenStack -- in fact I haven't pulled rank and demanded that specific content be included at all. However, the level of interest in OpenStack has grown so much since LCA 2012 that there is now a significant amount of OpenStack content in the conference without me having to do any of that. I thought I'd take a second to highlight some of the OpenStack content that I think is particularly interesting -- these are the talks I'll be going to if I have the time (which remains to be seen): Monday Cloud Infrastructure, Distributed Storage and High Availability Miniconf: while not specifically about OpenStack, this miniconf is going to be a good warm up for all things IaaS at the conference. Here's a list of the talks for that miniconf:<ul Delivering IaaS with Apache CloudStack - Joe Brockmeier oVirt - Dan Macpherson Aeolus - Dan Macpherson Ops: From bare metal to cloud space - Phil Ingram VMs on VLANs on Bridges on Bonds on many NICs - Kim Hawtin OpenStack Swift…

Image handlers (in essex)

Post author:mikal
Post published:December 23, 2012
Post category:OpenStack

George asks in the comments on my previous post about loop and nbd devices an interesting question about the behavior of this code on essex. I figured the question was worth bringing out into its own post so that its more visible. I've edited George's question lightly so that this blog post flows reasonably. Can you please explain the order (and conditions) in which the three methods are used? In my Essex installation, the "img_handlers" is not defined in nova.conf, so it takes the default value "loop,nbd,guestfs". However, nova is using nbd as the chose method. The handlers will be used in the order specified -- with the caveat that loop doesn't support Copy On Write (COW) images and will therefore be skipped if the libvirt driver is trying to create a COW image. Whether COW images are used is configured with the use_cow_images flag, which defaults to True. So, loop is being skipped because you're probably using COW images. My ssh keys are obtained by cloud-init, and still whenever I start a new instance I see in the nova-compute.logs this sequence of events: qemu-nbd -c /dev/nbd15 /var/lib/nova/instances/instance-0000076d/disk kpartx -a /dev/nbd15 mount /dev/mapper/nbd15p1 /tmp/tmpxGBdT0 umount /dev/mapper/nbd15p1 kpartx -d /dev/nbd15 qemu-nbd…

Some quick operational notes for users of loop and nbd devices

Post author:mikal
Post published:December 16, 2012
Post category:OpenStack

A quick note for OpenStack operators -- if you are using loop or nbd devices to mount disk images in nova-compute, then you would be well served to have plenty of device files hanging around to reduce contention. For loop devices, that seems as simple as making more of them with MAKEDEV. With nbd, you'll also need to increase the value of the max_nbd_devices flag to nova-compute. The latter is improved in grizzly, where we will autodetect nbd devices.

Moving on

Post author:mikal
Post published:December 9, 2012
Post category:OpenStack

Thursday this week is my last day at Canonical. After a little over a year at Canonical, I'm moving on to the private cloud team at Rackspace -- my first day with Rackspace will be the 17th of December. I'm very excited to be joining Rackspace -- I'm excited by the project, the team, and the opportunity to make OpenStack even better. We've also talked about some interesting stuff we'd like to do in the Australian OpenStack community, but I'm going to hold off on talking about that until I've had a chance to settle in. I am appreciative of my time at Canonical -- when I joined I was unaware of the existence of OpenStack, and without Canonical I might never have found this awesome project that I love. I also had the chance to work with some really smart people who taught me a lot. This move is about spending more time on OpenStack than Canonical was able to allow.

A first pass at glance replication

Post author:mikal
Post published:July 11, 2012
Post category:OpenStack

A few weeks back I was tasked with turning up a new OpenStack region. This region couldn't share anything with existing regions because the plan was to test pre-release versions of OpenStack there, and if we shared something like glance then we would either have to endanger glance for all regions during testing, or not test glance. However, our users already have a favorite set of images uploaded to glance, and I really wanted to make it as easy as possible for them to use the new region -- I wanted all of their images to magically just appear there. What I needed was some form of glance replication. I'd sat in on the glance replication session at the Folsom OpenStack Design Summit. The NeCTAR use case at the bottom is exactly what I wanted, so its reassuring that other people wanted something like that too. However, no one was working on this feature. So I wrote it. In fact, because of the code review process I wrote it twice, but let's not dwell on that too much. So, as of change id I7dabbd6671ec75a0052db58312054f611707bdcf there is a very simple replicator script in glance/bin. Its not perfect, and I expect it…

Folsom Dev Summit sessions

Post author:mikal
Post published:April 11, 2012
Post category:OpenStack

I thought I should write up the dev summit sessions I am hosting now that the program is starting to look solid. This is mostly for my own benefit, so I have a solid understanding of where to start these sessions off. Both are short brainstorm sessions, so I am not intending to produce slide decks or anything like that. I just want to make sure there is something to kick discussion off. Image caching, where to from here (nova hypervisors) As of essex libvirt has an image cache to speed startup of new instances. This cache stores images direct from glance, as well as resized images. There is a periodic task which cleans up images in the cache which are no longer needed. The periodic task can also optionally detect images which have become corrupted on disk. So first off, do we want to implement this for other hypervisors as well? As mentioned in a recent blog post I'd like to see the image cache manager become common code and have all the hypervisors deal with this in exactly the same manner -- that makes it easier to document, and means that on-call operations people don't need to determine…

Reflecting on Essex

Post author:mikal
Post published:April 6, 2012
Post category:OpenStack

This post is kind of long, and a little self indulgent. However, I really wanted to spend some time thinking about what I did for the Essex release cycle, and what I want to do for the Folsom release. I spent Essex mostly hacking on things in isolation, except for when Padraig Brady and I were hacking in a similar space. I'd like to collaborate more for Folsom, and I'm hoping talking about what I'm interested in doing in public might help with that. I came relatively late to the Essex development cycle, having never even heard of OpenStack before joining Canonical. We can talk about how I'd worked in the cloud space for six years and yet wasn't aware of the open source implementations at some other time. My initial introduction to OpenStack was being paged for compute nodes which were continually running out of disk. I googled around a bit and discovered that cached images for instances were never cleaned up (to start an instance, an image is fetched from glance, possibly has its format converted, is resized, and then an instance started with that resulting image, all those images were never being cleaned up). I filed bug…