OpenStack’s CI runs most changes through devstack, which usually builds an entire cloud on a single virtual machine. The underlying machines are often forced to swap out memory, and swapping makes jobs slow and flaky. This is especially true because a lot of the machines donated for hosting OpenStack CI are genuinely really quite old now. For years one of the suspects has been privsep, the privilege separation daemon that a member of my team wrote while we both worked at Rackspace Australia. The basic idea of privsep is to replace a predecessor called rootwrap which relied on complicated sudo configuration files and forked processes all over the place. privsep on the other hand works by decorating Python functions with the security context they should operate in and then doing… magic… to hand off actual execution of that method to a separate daemon which holds the right permissions. That is, privsep is the right technical solution, but to my mild surprise a decade later we’re still half-way through a migration and not everyone is completely happy. privsep’s memory usage came up again at a recent Technical Committee meeting, so I went and measured it.
I was talking to a friend the other day about our shared mutual appreciation of virtio-vsock, and it made me wonder something. How do virtual machines on Linux actually work? I know it involves qemu and the kernel’s KVM virtual machine implementation, but exactly how do they interact? How does the kernel get qemu to do emulation tasks as required?
qemu is several things hanging out together in a trench coat, but one of those things is software which can configure Linux’s built-in KVM virtual machine functionality to run a virtual machine, and then handle emulation of the devices that virtual machine is attached to which cannot be represented with actual physical hardware. This part of qemu is called a “KVM client” in the Linux kernel documentation. Its called that because if we ignore the emulation part for now it is just literally a client calling established APIs to the Linux kernel.
I’ve fallen into this pattern where I do an hour or so of self-directed learning in the mornings before going to work. Until recently it was an excellent CMU course on the design of SQL database systems, which I’ve mentioned previously here. I’ve finished that, so I thought I would do something shorter and fun as a break before finding another course to do. I chose The freeCodeCamp.org hot dog or not hot dog tensorflow course. 90 minutes seemed achievable, and I too wish to know if an object in front of me is a hot dog or not.
I make no claim to be an expert at this, but I did just need to convert a project from a slightly complicated setup.py / PBR configuration to pyproject.toml and thought I should write up where I landed. I say “slightly complicated” because there are a few very OpenStacky things I like to do in these things. Specifically:
version numbers are driven by git tags not hard coded in the configuration file.
console scripts are a thing.
I often include data files in the built package.
So here’s an example of all of those things that is working ok for me:
I’ve been using virtio-serial for communications between Linux hypervisors and guest virtual machines for ages. Lots of other people do it to — the qemu guest agent for example is implemented like this. In fact, I think that’s where I got my original thoughts on the matter from. However, virtio-serial is actually fairly terrible to write against as a programming model, because you’re left to do all the multiplexing of various requests down the channel and surely there’s something better?
Well… There is! virtio-vsock is basically the same concept, except it uses the socket interface. You can have more than one connection open and the sockets layer handles multiplexing by magic. This massively simplifies the programming model for supporting concurrent users down the channel. So that’s actually pretty cool. I should credit Kata Containers with noticing this quality of life improvement nearly a decade before I did, but I get there in the end.
I got interested today in trying to come up with a solid way of determining when updates were last applied to a RHEL-derived Linux instance. Previously we’d been inferring it from the kernel version, but it turns out there is a convenient “yum history” or “dnf history” command which will show you all the previous transactions that the package database has seen. However, the output is hard to parse in a script.
In New python syntax I was previously unaware of, I discussed some new operators I'd recently discovered. One of them is called the Walrus operator, which lets you write code like this: list = ['a', 'b', 'c'] def get_one(): if not list: return None return list.pop() while one := get_one(): print(one) See where we do the assignment inside the while? That code returns: c b a Which is as expected. However, the Walrus operator is strict about needing a None returned to end the iteration. I had code which was more like this: list = [('a', 1), ('b', 2), ('c', 3)] def get_one(): if not list: return None, None return list.pop() while one := get_one(): print(one) And the while loop never terminates. It just prints (None, None) over and over. So there you go.
This post documents the new syntax features I learned about while reading cpython internals. You can create more than one context manager on a single line. So for example Shaken Fist contains code like this: with open(path + '.new', 'w') as o: with open(path, 'r') as i: ... That can now be written like this: with open(path + '.new', 'w') as o, open(path, 'r') as i: ... You can assign values in a while statement, but only one. Instead of this: d = f.read(8000) while f: ... d = f.read(8000) You can write this: while d := f.read(8000): ... But unfortunately this doesn't work: while a, b := thing(): ... You can use underscores as commands in long numbers to make them easier to read. For example, you can write 1000000 or 1_000_000 and they both mean the same thing. You can refer to positional arguments by name, but you can also disable that. I didn't realise that this was valid python: def foo(bar=None): print(bar) foo(bar='banana') You can turn it off with a forward slash in the argument list though, which should separate positional arguments from named arguments: def foo(bar, /, extra=None): print(bar) print(extra) foo('banana', extra='frog') The above example…
Get your guided tour through the Python 3.9 interpreter: Unlock the inner workings of the Python language, compile the Python interpreter from source code, and participate in the development of CPython. Are there certain parts of Python that just seem like magic? This book explains the concepts, ideas, and technicalities of the Python interpreter in an approachable and hands-on fashion. Once you see how Python works at the interpreter level, you can optimize your applications and fully leverage the power of Python.
I have been paid money to write Python code since about 2006, so I figured it was probably time that I should understand some of the inner workings of Python. I therefore picked up two books on the topic, this one being the first of the two.
This book to be honest isn’t completely what I expected. Its very well written and quite interesting, but its more about the things you’d need to know to become a Python core developer, rather than the things you should know as a user of Python like how the Python dictionary implementation is built.
(If you want that specifically, this video is an excellent introduction).
So last night Shaken Fist CI jobs started failing with errors like this (editted lightly for clarity): Building wheels for collected packages: shakenfist-ci Building wheel for shakenfist-ci (setup.py): started Building wheel for shakenfist-ci (setup.py): finished with status 'error' error: subprocess-exited-with-error × python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [86 lines of output] ... ...setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. setuptools.SetuptoolsDeprecationWarning, installing to build/bdist.linux-x86_64/wheel running install ... warning: install_lib: byte-compiling is disabled, skipping. running install_egg_info Copying shakenfist_ci.egg-info to build/bdist.linux-x86_64/wheel/shakenfist_ci-0.0.1.dev2544-py3.7.egg-info running install_scripts error: invalid command 'bdist_wininst' [end of output] This was pretty concerning. I know that a setup.py / setup.cfg style install is a little old school, but it was unexpected that it broke entirely. At first I thought I'd have to convert to poetry to unblock this, but Chet helpfully pointed out that this is as simple as adding a pyproject.toml file to the directory which contains your setup.py and setup.cfg. The basic issue is that a modern pip doesn't assume that you're going to use setuptools, so you need to tell it that you're doing that in pyproject.toml. Then you're unblocked. So, just create a file named…
