Blog

OpenStack’s CI runs most changes through devstack, which usually builds an entire cloud on a single virtual machine. The underlying machines are often forced to swap out memory, and swapping makes jobs slow and flaky. This is especially true because a lot of the machines donated for hosting OpenStack CI are genuinely really quite old now. For years one of the suspects has been privsep, the privilege separation daemon that a member of my team wrote while we both worked at Rackspace Australia. The basic idea of privsep is to replace a predecessor called rootwrap which relied on complicated sudo configuration files and forked processes all over the place. privsep on the other hand works by decorating Python functions with the security context they should operate in and then doing… magic… to hand off actual execution of that method to a separate daemon which holds the right permissions. That is, privsep is the right technical solution, but to my mild surprise a decade later we’re still half-way through a migration and not everyone is completely happy. privsep’s memory usage came up again at a recent Technical Committee meeting, so I went and measured it.

The short answer was that the suspicion was partly fair. privsep daemons used 0.3 to 0.45 GB on an OVN all-in-one node, and about 1 GB on an ML2 / OVS controller, which is 5 to 14% of the node’s RSS. That is not nothing, but qemu guests and API workers are bigger. The more interesting answer was why the daemons were that big. They were not doing much. They were mostly carrying around code they had imported and would never run.

Once I started looking for that, it was everywhere: not just in privsep, but in the services themselves. This post covers why that happens, the techniques I used to find and remove it, and a Claude Code skill that packages those techniques up so they can be used on other projects.

Why imports cost so much

A long-running Python process’s memory is dominated by the code it imported, not by the data it holds. Every module is a module object, a dictionary, function objects, code objects, classes, and whatever constants were built at import time. A typical OpenStack service imports 1,500 to 2,600 modules before it does any work.

None of that is shared between processes that were started with exec(), which is how privsep daemons are started (via sudo and rootwrap), and how most services are started by systemd. Forked worker processes share the pages with their parent only until they are written to. CPython’s reference counting writes to object headers all the time, so in practice the sharing erodes quickly.

So if a process imports a package it never calls that package costs real memory, in every copy of that process, on every host. Some examples I found:

• Every neutron agent loaded SQLAlchemy, oslo.db, pecan and mako. The agents don’t have a database. They got it because neutron_lib’s __init__.py imports its DB API.
• neutron imported the whole of openstacksdk at module level for one optional middleware.
• neutron-lib imported os_ken’s packet parsing library to get a table of ethertype constants.
• keystone loaded pysaml2 for SAML federation regardless of whether or not federation was configured, and placement loaded alembic for database migrations it only runs when a startup flag is set.
• Services that have moved from eventlet to threads still loaded eventlet, because the libraries underneath them imported it to check whether it was in use.

The techniques

Measure each process in a fresh interpreter

The first step is a probe: a small script that imports a process’s entry point in a fresh python -I interpreter, then reads RssAnon from /proc/self/status. Anonymous RSS is the memory the process allocated itself, which is where imported code lives. -I matters, because anything a sitecustomize or .pth file imported first would be measured but not attributed. However, importing the entry point alone is not enough. Services load their drivers through stevedore or entry points after start-up, so the probe has to import those plugins too. It also has to do whatever the entry point does before its imports. nova’s API entry points select the threading backend first, which keeps eventlet out. My first probe skipped that step, and over-reported both nova API services by about 8 MB.

Before going further, compare the probe’s number with the process’s real peak RSS from a CI job. For most neutron agents and several nova services the two were within about 10%. That means the memory is almost all imports, and there is something to find. For nova-compute the peak was far higher. Its memory goes on runtime state, and this approach will not find it. This is why the patch series uploaded over the weekend starts with fixing the instrumentation in devstack and then the patches in every project depend on those fixes — without those we cannot measure whether we’ve improved things or not (patch 1, patch 2, patch 3, patch 4).

Find out who imported what

Knowing that a process loaded SQLAlchemy is only useful if you know why. The probe installs a finder at the front of sys.meta_path. A finder sees every import, whether it comes from an import statement, importlib.import_module() or a plugin loader. Mine never finds anything. It just records which modules were on the stack when each new module was first requested, and steps aside. The result is an import chain for every package, read innermost first.

Two details turned out to matter: libraries often check whether a package is installed with importlib.util.find_spec(). That consults the finders but does not import anything. The probe only reports chains for modules that actually ended up in sys.modules, and a later real import replaces the record. So a library that merely looked is not blamed; tracemalloc is no help here — it charges code objects to importlib, which ends up with about half of everything an import allocates.

Cost it before you fix it

To estimate what removing a package would save, import a base set of modules that the process loads anyway, then the package, and measure the difference. The base set matters a lot. Imported into an empty interpreter, eventlet costs 22.6 MB. On top of requests and cryptography, which every OpenStack service loads anyway, it costs 11 MB, and 11 MB is the real saving.

There is always a second import site

This is an important point which I initially missed. The import chain shows the first place a package was imported. Remove that import and the package usually still loads through the next site. The saving only arrives when the last site is gone. My first eventlet fix was in futurist, which was the obvious culprit. It saved precisely nothing. So I changed approach and mapped every chain before writing any more patches. In a scratch copy of the source trees, I made the crudest edit that removed the first site, probed again, and read the next chain, repeating until the package was gone or the next site was outside OpenStack. For eventlet in neutron there were seven sites in seven projects: futurist, oslo.utils, oslo.messaging, oslo.service, oslo.privsep, ovsdbapp, and python-ovs (which is Open vSwitch, not OpenStack at all). Each real patch then says in its commit message that it saves nothing on its own, and names the others in the chain.

The fixes themselves are usually small

Most fixes are a lazy import — all that needed to be done was moving the import into the function that uses it. A few other patterns came up repeatedly:

• To ask “is eventlet in use?”, look in sys.modules. If eventlet hasn’t been imported, it hasn’t monkey-patched anything, and no import is needed to find that out.
• To ask “is it installed?”, use importlib.util.find_spec(), and catch ImportError. nova installs an import hook that raises it.
• Move a small helper out of a heavy module, and leave an alias behind for anyone outside the tree who imported it from the old place.
• Spell out a handful of constants rather than importing a large module for them, with a unit test that they still match.
• Watch for side effects at import time. Moving an import also moves whatever that module did when it was imported, such as registering objects or options, and something may depend on that.

Guard the fix with a test

A lazy import is easy to break — the next unrelated change that imports something at module level brings the whole package back. So every patch carries a unit test that imports the relevant modules in a new interpreter (the test runner has usually imported everything already) and fails if the package shows up in sys.modules. Honestly the tests are a bit ugly to behold, but they work which is important. I checked that each test fails without its fix, by restoring the original source and running it. Where another project still loads the package, that test can’t pass yet. For those cases I added a second test that checks the project’s own modules don’t hold a reference to the package. That one can pass as soon as the patch lands.

The results

These are anonymous RSS figures per process, in MB, measured with the whole chain applied. That includes two fixes outside this set of patches: oslo.privsep’s eventlet change, which is in review, and ovsdbapp 3.0.0, which is released but not yet allowed by OpenStack’s upper-constraints.

Differences of 2 MB or less are noise. The nova and cinder services, which none of these patches touch, move by that much between runs. A few honest caveats:

• These are per process. A node runs many copies of some of these processes, so multiply by the process count to get a per-node figure.
• RSS counts copy-on-write pages in both parent and child, so summing RSS across a pool of forked workers overstates the shared part.
• The memory saved is mostly cold. Under memory pressure these are the pages that get swapped out first. Removing them reduces swap volume, but that on its own doesn’t prove it reduces the thrashing that makes jobs slow.

The patches are now in review upstream if you’re interested in watching me recheck them over and over: the first set which was privsep focused; and then a second set which attacks the more general case.

Doing this on other projects

None of this is specific to OpenStack. Any long-running Python program that imports a lot of code has the same problem: Celery workers, Django and Flask applications, data pipelines, agents on small devices, and anything else with several copies of itself running on one host. So I packaged the techniques as a Claude Code skill called import-bloat, with a helper script that does the measuring. Skills are just markdown files with a better marketing department, and you can find the current version of this skill here, although you’ll also need this helper script too.

The helper is one Python file that only uses the standard library, so it runs under the interpreter of whatever virtualenv or container holds the code you are measuring. It has five subcommands:

$ ./import-bloat --help
usage: import-bloat [-h] {probe,roles,marginal,report,compare} ...

Measure what a Python process's imports cost it, and which import pulled each package in.

positional arguments:
  {probe,roles,marginal,report,compare}
    probe               import one role and print its memory, modules and chains
    roles               probe every role in a roles file, each in a fresh interpreter
    marginal            the anonymous RSS TARGET adds on top of BASE
    report              summarise probe output
    compare             per-role savings of one or more after runs against a before run

options:
  -h, --help            show this help message and exit

A roles file is one line per process type: a name, then the modules that process imports, including its plugins and any setup it does first:

python-baseline sys
web myapp.wsgi myapp.views myapp.middleware.auth
worker call:myapp.setup.configure(backend='threads') myapp.worker myapp.tasks

In practice you ask something like “why is our Celery worker 400 MB, and how much of that is code it doesn’t use?”, and the agent writes the roles file, runs the probe and the chains, and comes back with a table of import sites and proposed fixes for you to decide on.

Process notes

This work would not have been possible for me without Claude Code. While the use of LLM coding tools might currently be contentious, I simply wouldn’t have had the time to take an offhand comment from an OpenStack TC meeting I wasn’t even in and turn it into an investigation and a series of fixes without the help of such tooling.