I went on a side quest a couple of weekends ago which seems like it might have ended up being a big win. That win likely applies to other people running both OpenStack, but also other cloud orchestration systems using libvirt like oVirt, Proxmox, Shaken Fist, and so forth.
By way of back story, I have a stack of patches which I maintain on top of OpenStack’s Kolla and Kolla-Ansible. Most of those patches continue to progress the new “spice-direct” console type in OpenStack Nova, but I also occasionally find little bugs or CI regressions along the way that I want to clean up. One example of such a bug was Kolla’s failure to setup virtlogd to provide instance console log rotation, which is currently being backported to various releases. Another one is the topic of this post.
You see, I was getting weird CI failures for my patches that I could not explain — retrying the CI run would pass, but I accidentally got lucky while chasing what was flaky and realized that libvirt was logging this in its log (this example failure is from the voting kolla-ansible-debian-bookworm-upgrade job on stable/2025.2):
warning : Client hit max requests limit 5. This may result in
keep-alive timeouts. Consider tuning the max_client_requests server
parameter
...
error : connection closed due to keepalive timeoutThe libvirt daemon ships with max_workers=20 and max_client_requests=5. Those defaults suit a multi-tenant host serving many mutually distrustful clients, where the per-client request cap protects the daemon from any single client monopolizing it. Under Kolla-Ansible, nova-compute is effectively the sole privileged client of libvirtd, so the cap of 5 concurrent requests per client becomes the binding constraint rather than a safety net.
When several instances build at once — as they do during the tempest run of the upgrade jobs — nova-compute’s concurrent libvirt RPCs (instance defineXML() calls plus the resource-tracker periodic calls) exceed five. Further requests queue, the connection’s keepalive responses are starved, and after keepalive_interval (5s) x keepalive_count (5) = 25s libvirtd closes the connection. nova then sees “Cannot recv data: Input/output error” mid-defineXML(), the build is rescheduled, and once all hosts are exhausted the instance is left in ERROR. This presents as flaky upgrade-job failures.
You can see the original patch for this on gerrit, which links to the relevant backports as well. I must say I was impressed with how quickly the Kolla team engaged with the idea that there might be something to fix here and landed the patch series, especially the backports. I hope in return CI just that little bit less annoying now.
So there are two reasons I wanted to whip up this post — others should likely tune these parameters in libvirt — but also check out this super satisfying graph:
That’s something to be proud of I think.