virtio-vsock: python examples of running the server in the guest

I’ve been using virtio-serial for communications between Linux hypervisors and guest virtual machines for ages. Lots of other people do it to — the qemu guest agent for example is implemented like this. In fact, I think that’s where I got my original thoughts on the matter from. However, virtio-serial is actually fairly terrible to write against as a programming model, because you’re left to do all the multiplexing of various requests down the channel and surely there’s something better?

Well… There is! virtio-vsock is basically the same concept, except it uses the socket interface. You can have more than one connection open and the sockets layer handles multiplexing by magic. This massively simplifies the programming model for supporting concurrent users down the channel. So that’s actually pretty cool. I should credit Kata Containers with noticing this quality of life improvement nearly a decade before I did, but I get there in the end.

The virtio-vsock model is only a little bit weird. The “address” for the guest virtual machine is a “CID” (Context ID). The hypervisor process is always at CID 0, CID 1 is reserved and unused, and CID 2 is any process on the host which is not the hypervisor. You can also use port numbers like you would with IP, which means you can have multiple services down a single vsock channel if you’d like. I haven’t looked deeply into the history of vsock, but pages like this one make me think it originated with VMWare.

So let’s write some code. All of the examples I could find (this gist, or the page linked in the previous paragraph) have the server on the host, and then the client being inside the guest virtual machine, but I want the exact opposite. It wasn’t as hard as I thought it would be to figure out, especially once I found this very helpful gist about how to lookup the CID from inside the guest virtual machine.

(As an aside, my life would be much worse if github took the gist feature offline. There is so much useful random stuff out there in gists.)

So here’s our server, which runs inside the virtual machine:

import fcntl
import socket
import struct


PORT = 1025


# Lookup our CID. This is a 32 bit unsigned int returned from an ioctl
# against /dev/vsock. As best as I can tell the empty string argument
# at the end is because that is used as a buffer to return the result
# in. Yes really.
with open('/dev/vsock', 'rb') as f:
    r = fcntl.ioctl(f, socket.IOCTL_VM_SOCKETS_GET_LOCAL_CID, '    ')
    cid = struct.unpack('I', r)[0]
print(f'Our CID is {cid}.')


s = socket.socket(socket.AF_VSOCK, socket.SOCK_STREAM)
s.bind((cid, PORT))
s.listen()

conn, (remote_cid, remote_port) = s.accept()
print(f'Connection from {remote_cid} on with remote port {remote_port}')

while True:
    buf = conn.recv(1024)
    print(f' in: {buf}')
    if not buf:
        print('Nothing received, exiting')
        break

    print(f'out: {buf}')
    conn.sendall(buf)

conn.close()
print('Done')

And here’s our client, which runs on the host OS. Note that the CID is hard coded here because I specified it when I created the virtual machine, it just happens to be my favorite 32 bit unsigned integer:

import socket

CID = 347338050
PORT = 1025
STRINGS = [
    b'Hello World!',
    b'Banana',
    b'Duck'
]

s = socket.socket(socket.AF_VSOCK, socket.SOCK_STREAM)
s.connect((CID, PORT))

for out in STRINGS:
    s.sendall(out)
    print(f'out: {out}')

    buf = s.recv(1024)
    if not buf:
        print('Nothing received, exiting')
        break
    print(f' in: {buf}')

s.close()
print('Done')

And that’s that. I hope this helps someone.