Nova database continuous integration

I’ve had some opportunity recently to spend a little quality time off line, and I spent some of that time working on a side project I’ve wanted to do for a while — continuous integration testing of nova database migrations. Now, the code isn’t perfect at the moment, but I think its an interesting direction to take and I will keep pursuing it.

One of the problems nova developers have is that we don’t have a good way of determining whether a database migration will be painful for deployers. We can eyeball code reviews, but whether code looks reasonable or not, its still hard to predict how it will perform on real data. Continuous integration is the obvious solution — if we could test patch sets on real databases as part of the code review process, then reviewers would have more data about whether to approve a patch set or not. So I did that.

At the moment the CI implementation I’ve built isn’t posting to code reviews, but that’s because I want to be confident that the information it gathers is accurate before wasting other reviewers’ time. You can see results at openstack.stillhq.com/ci. For now, I am keeping an eye on the test results and posting manually to reviews when an error is found — that has happened twice so far.

The CI tests work by restoring a MySQL database to a known good state, upgrading that database from Folsom to Grizzly (if needed). It then runs the upgrades already committed to trunk, and then the proposed patch set. Timings for each step are reported — for example with my biggest test database the upgrade from Folsom to Grizzly takes between about 7 and 9 minutes to run, which isn’t too bad. You can see an example log at here.

I’d be interested in know if anyone else has sample databases they’d like to see checks run against. If so, reach out to me and we can make it happen.

A historical note from November 2020: the links in this post to the OpenStack CI system have been broken for quite some time and have therefore been removed.

MythBuntu 8.10 just made me sad

I figured it was time to give MythBuntu a try, so I set up a MythBuntu 8.10 instance in VirtualBox today. That was a mistake. I’m not 100% sure I understand how it happened, but MythBuntu somehow managed to delete my entire mythconverg MySQL database instance. Not pleased. I’ve restored it from last night’s backup, but now I’ll need to recover recordings which happened today, assuming I can be bothered.

I’m writing this just as a warning to others — if you’re playing with MythBuntu, backup your MySQL instance if its not a test one.

Managing MySQL the Slack Way: How Google Deploys New MySQL Servers

I’ll be presenting about Slack (the open sourced tool kit we use for deployment software configuration) at the MySQL user’s conference in Santa Clara in late April. The talk will focus on the interesting aspects of Slack as it relates to MySQL and should be fun. A DBA mate of mine is gonna present with me, so it should be a barrel of laughs.