I’ve had some opportunity recently to spend a little quality time off line, and I spent some of that time working on a side project I’ve wanted to do for a while — continuous integration testing of nova database migrations. Now, the code isn’t perfect at the moment, but I think its an interesting direction to take and I will keep pursuing it.
One of the problems nova developers have is that we don’t have a good way of determining whether a database migration will be painful for deployers. We can eyeball code reviews, but whether code looks reasonable or not, its still hard to predict how it will perform on real data. Continuous integration is the obvious solution — if we could test patch sets on real databases as part of the code review process, then reviewers would have more data about whether to approve a patch set or not. So I did that.
At the moment the CI implementation I’ve built isn’t posting to code reviews, but that’s because I want to be confident that the information it gathers is accurate before wasting other reviewers’ time. You can see results at openstack.stillhq.com/ci. For now, I am keeping an eye on the test results and posting manually to reviews when an error is found — that has happened twice so far.
The CI tests work by restoring a MySQL database to a known good state, upgrading that database from Folsom to Grizzly (if needed). It then runs the upgrades already committed to trunk, and then the proposed patch set. Timings for each step are reported — for example with my biggest test database the upgrade from Folsom to Grizzly takes between about 7 and 9 minutes to run, which isn’t too bad. You can see an example log at here.
I’d be interested in know if anyone else has sample databases they’d like to see checks run against. If so, reach out to me and we can make it happen.
A historical note from November 2020: the links in this post to the OpenStack CI system have been broken for quite some time and have therefore been removed.
