Tag: tips

Exploring a single database migration

Yesterday I was having some troubles with a database migration download step, and a Joshua Hesketh suggested I step through the migrations one at a time and see what they were doing to my sqlite test database. That’s a great idea, but it wasn’t immediately obvious to me how to do it. Now that I’ve figured out the steps required, I thought I’d document them here.

First off we need a test environment. I’m hacking on nova at the moment, and tend to build throw away test environments in the cloud because its cheap and easy. So, I created a new Ubuntu 12.04 server instance in Rackspace’s Sydney data center, and then configured it like this:

    $ sudo apt-get update
$ sudo apt-get install -y git python-pip git-review libxml2-dev libxml2-utils
libxslt-dev libmysqlclient-dev pep8 postgresql-server-dev-9.1 python2.7-dev
python-coverage python-netaddr python-mysqldb python-git virtualenvwrapper
python-numpy virtualenvwrapper sqlite3
$ source /etc/bash_completion.d/virtualenvwrapper
$ mkvirtualenv migrate_204
$ toggleglobalsitepackages


Simple! I should note here that we probably don’t need the virtualenv because this machine is disposable, but its still a good habit to be in. Now I need to fetch the code I am testing. In this case its from my personal fork of nova, and the git location to fetch will obviously change for other people:

    $ git clone http://github.com/mikalstill/nova

Now I can install the code under test. This will pull in a bunch of pip dependencies as well, so it takes a little while:

    $ cd nova
$ python setup.py develop

Next we have to configure nova because we want to install specific database schema versions.

    $ mkdir /etc/nova
$ sudo mkdir /etc/nova
$ sudo vim /etc/nova/nova.conf
$ sudo chmod -R ugo+rx /etc/nova

The contents of my nova.conf looks like this:

    $ cat /etc/nova/nova.conf
[DEFAULT]
sql_connection = sqlite:////tmp/foo.sqlite

Now I can step up to the version before the one I am testing:

    $ nova-manage db sync --version 203

You do the same thing but with a different version number to step somewhere else. Its also pretty easy to get the schema for a table under sqlite. I just do this:

    $ sqlite3 /tmp/foo.sqlite
SQLite version 3.7.9 2011-11-01 00:52:41
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .schema instances
CREATE TABLE "instances" (
        created_at DATETIME,
        updated_at DATETIME,
[...]

So there you go.

Disclaimer — I wouldn’t recommend upgrading to a specific version like this for real deployments, because the models in the code base wont match the tables. If you wanted to do that you’d need to work out what git commit added the version after the one you’ve installed, and then checkout the commit before that commit.

Faster pip installs

Last week with the help of the lovely openstack-infra people, I discovered that you can have a local cache of pip downloads. This speeds up rebuilding test environments when you need to jump between branches with different dependencies. Its as simple as chucking something like:

    export PIP_DOWNLOAD_CACHE=~/cache/pip

…into your .bashrc or equivalent.

Couldn’t determine the video directory?

A couple of people have contacted me in the last couple of days with an error where MythNetTV can’t determine the directory to put videos in. The error would look something like this:

    
Importing data/tekzilla--0050--tehbunniez--hd.h264.mp4
Traceback (most recent call last):
  File "./mythnettv", line 453, in 
    main(sys.argv)
  File "./mythnettv", line 212, in main
    mythnettvcore.DownloadAndImport(db, guid, out=out)
  File "/home/mythbox/Scripts/mythnettv/mythnettvcore.py", line 92, in
  DownloadAndImport
    prog.Import(out=out)
  File "/home/mythbox/Scripts/mythnettv/program.py", line 472, in Import
    raise FilenameException(self.db, 'Could not determine the video '
program.FilenameException: Could not determine the video directory for this
machine. Please report this to mythnettv@stillhq.com

The stack trace is mostly irrelevant. The problem here is that MythNetTV couldn’t decide what directory to put the video in once downloaded. Please execute the following SQL statements against your MythTV database:

    select * from storagegroup where groupname="MythNetTV";
select * from storagegroup where groupname="Default";
select * from settings where value="RecordFilePrefix";

This will dump all of the possible places MythNetTV will look for a video directory.

Once you’ve dumped this information, perform some simple checks:

  • Is there anything listed at all? If not, you need to configure storage groups with the MythTV user interface
  • Is there anything listed with the hostname that MythNetTV is running on? If not, you need to add configuration entries for your current hostname.

If you’re still having problems, please send email to mythnettv@stillhq.com, with the output of those select commands, and the output of the hostname command.