python – Made by Mikal

Chasing OpenStack memory usage: the memory your Python services spend on code they never run

Post author:mikal
Post published: October 25, 2026
Post category:OpenStack Python

OpenStack’s CI runs most changes through devstack, which usually builds an entire cloud on a single virtual machine. The underlying machines are often forced to swap out memory, and swapping makes jobs slow and flaky. This is especially true because a lot of the machines donated for hosting OpenStack CI are genuinely really quite old now. For years one of the suspects has been privsep, the privilege separation daemon that a member of my team wrote while we both worked at Rackspace Australia. The basic idea of privsep is to replace a predecessor called rootwrap which relied on complicated sudo configuration files and forked processes all over the place. privsep on the other hand works by decorating Python functions with the security context they should operate in and then doing… magic… to hand off actual execution of that method to a separate daemon which holds the right permissions. That is, privsep is the right technical solution, but to my mild surprise a decade later we’re still half-way through a migration and not everyone is completely happy. privsep’s memory usage came up again at a recent Technical Committee meeting, so I went and measured it.

(more…)

Tensorflow, ROCm, and version mismatches

Post author:mikal
Post published: September 21, 2025
Post category:Machine learning Python

I’ve fallen into this pattern where I do an hour or so of self-directed learning in the mornings before going to work. Until recently it was an excellent CMU course on the design of SQL database systems, which I’ve mentioned previously here. I’ve finished that, so I thought I would do something shorter and fun as a break before finding another course to do. I chose The freeCodeCamp.org hot dog or not hot dog tensorflow course. 90 minutes seemed achievable, and I too wish to know if an object in front of me is a hot dog or not.

(more…)

Converting from setup.py and PBR to pyproject.toml

Post author:mikal
Post published: September 9, 2025
Post category:Python

I make no claim to be an expert at this, but I did just need to convert a project from a slightly complicated setup.py / PBR configuration to pyproject.toml and thought I should write up where I landed. I say “slightly complicated” because there are a few very OpenStacky things I like to do in these things. Specifically:

version numbers are driven by git tags not hard coded in the configuration file.
console scripts are a thing.
I often include data files in the built package.

So here’s an example of all of those things that is working ok for me:

(more…)

virtio-vsock: python examples of running the server in the guest

Post author:mikal
Post published: March 10, 2025
Post category:libvirt Linux Python qemu

I’ve been using virtio-serial for communications between Linux hypervisors and guest virtual machines for ages. Lots of other people do it to — the qemu guest agent for example is implemented like this. In fact, I think that’s where I got my original thoughts on the matter from. However, virtio-serial is actually fairly terrible to write against as a programming model, because you’re left to do all the multiplexing of various requests down the channel and surely there’s something better?

Well… There is! virtio-vsock is basically the same concept, except it uses the socket interface. You can have more than one connection open and the sockets layer handles multiplexing by magic. This massively simplifies the programming model for supporting concurrent users down the channel. So that’s actually pretty cool. I should credit Kata Containers with noticing this quality of life improvement nearly a decade before I did, but I get there in the end.

(more…)

An easier to parse version of “yum history”

Post author:mikal
Post published: November 15, 2024
Post category:Python Red Hat

I got interested today in trying to come up with a solid way of determining when updates were last applied to a RHEL-derived Linux instance. Previously we’d been inferring it from the kernel version, but it turns out there is a convenient “yum history” or “dnf history” command which will show you all the previous transactions that the package database has seen. However, the output is hard to parse in a script.

(more…)

A gotcha with the Walrus operator

Post author:mikal
Post published: April 20, 2024
Post category:Python

In New python syntax I was previously unaware of, I discussed some new operators I'd recently discovered. One of them is called the Walrus operator, which lets you write code like this: list = ['a', 'b', 'c'] def get_one(): if not list: return None return list.pop() while one := get_one(): print(one) See where we do the assignment inside the while? That code returns: c b a Which is as expected. However, the Walrus operator is strict about needing a None returned to end the iteration. I had code which was more like this: list = [('a', 1), ('b', 2), ('c', 3)] def get_one(): if not list: return None, None return list.pop() while one := get_one(): print(one) And the while loop never terminates. It just prints (None, None) over and over. So there you go.

New python syntax I was previously unaware of

Post author:mikal
Post published: March 25, 2024
Post category:Python

This post documents the new syntax features I learned about while reading cpython internals. You can create more than one context manager on a single line. So for example Shaken Fist contains code like this: with open(path + '.new', 'w') as o: with open(path, 'r') as i: ... That can now be written like this: with open(path + '.new', 'w') as o, open(path, 'r') as i: ... You can assign values in a while statement, but only one. Instead of this: d = f.read(8000) while f: ... d = f.read(8000) You can write this: while d := f.read(8000): ... But unfortunately this doesn't work: while a, b := thing(): ... You can use underscores as commands in long numbers to make them easier to read. For example, you can write 1000000 or 1_000_000 and they both mean the same thing. You can refer to positional arguments by name, but you can also disable that. I didn't realise that this was valid python: def foo(bar=None): print(bar) foo(bar='banana') You can turn it off with a forward slash in the argument list though, which should separate positional arguments from named arguments: def foo(bar, /, extra=None): print(bar) print(extra) foo('banana', extra='frog') The above example…

cpython internals

Post author:mikal
Post published: March 25, 2024
Post category:Book Python

Title: CPython Internals
Author: Anthony Shaw
Release Date: May 5, 2021
Pages: 396

★★★★☆

Get your guided tour through the Python 3.9 interpreter: Unlock the inner workings of the Python language, compile the Python interpreter from source code, and participate in the development of CPython. Are there certain parts of Python that just seem like magic? This book explains the concepts, ideas, and technicalities of the Python interpreter in an approachable and hands-on fashion. Once you see how Python works at the interpreter level, you can optimize your applications and fully leverage the power of Python.

I have been paid money to write Python code since about 2006, so I figured it was probably time that I should understand some of the inner workings of Python. I therefore picked up two books on the topic, this one being the first of the two.

This book to be honest isn’t completely what I expected. Its very well written and quite interesting, but its more about the things you’d need to know to become a Python core developer, rather than the things you should know as a user of Python like how the Python dictionary implementation is built.

(If you want that specifically, this video is an excellent introduction).

(more…)

Validating a keystone token

Post author:mikal
Post published: January 3, 2024
Post category:OpenStack

Once again I venture into the lands of poorly documented keystoneauth1 calls. This time, I want to be able to validate if a stored keystone authentication token is valid. Here's the best I could come up with, I'd be interested in others have something better. For this to work, we need a service account to create a keystone client with, and then we can ask that client questions about random other tokens... from keystoneauth1 import exceptions from keystoneauth1.identity import v3 from keystoneauth1 import session from keystoneclient.v3 import client def validate_keystone_token(service_auth, token): """Validate a keystone token. Returns True if the token is valid, False otherwise. """ # We need a keystone client as the service service_session = session.Session(auth=service_auth) service_keystone = client.Client(session=service_session) try: user = service_keystone.tokens.validate(token) except exceptions.http.NotFound: return False # Require that there be an access group with our configured name group = None for g in service_keystone.groups.list(): if g.name == 'mygroup': group = g if not group: return False # Require that the user be in that group try: service_keystone.users.check_in_group(user.user_id, group.id) except exceptions.http.NotFound: return False return True # Authenticate the service user service_auth = v3.Password( auth_url='http://kolla.home.stillhq.com:5000', username='admin', password='...', project_name='admin', user_domain_id='default', project_domain_id='default') # Create a token we can test user_auth =…

Using the openstacksdk with authentication arguments

Post author:mikal
Post published: August 21, 2023
Post category:OpenStack

I wanted to authenticate against OpenStack recently, and had a lot of trouble finding documentation about how to authenticate just by passing arguments (as opposed to by using clouds.yaml or environment variables). Now that I have a working incantation, I figure I should write it down so I can find it again. Its also disappointing the OpenStack documentation doesn't appear to cover this particularly well... from keystoneauth1.identity import v3 from keystoneauth1 import session from openstack import connection auth = v3.Password( auth_url='http://kolla.home.stillhq.com:5000', username='admin', password='...', project_name='admin', user_domain_id='default', project_domain_id='default') sess = session.Session(auth=auth) conn = connection.Connection(session=sess) print([x.name for x in conn.list_servers()]) This code will authenticate using the arguments provided, and then list all the servers (instances) visible to that user. You're welcome.