The simplest boot target for the Kerbside SPICE VDI proxy CI

For the last couple of years I have been working on a SPICE protocol native proxy called Kerbside. The basic idea is to be able to provide SPICE Virtual Desktop Interface (VDI) consoles to users from cloud platforms such as Shaken Fist, OpenStack, or oVirt. Think Citrix, but for Open Source cloud platforms. SPICE is attractive here because it has some features that other more common VDI protocols like VNC don’t have — good cut and paste support, USB device pass-through, multiple monitor support, and so on. RDP has these, but RDP was not a supported VDI protocol when using qemu on Linux with KVM until incredibly recently — literally the last couple of months.

(In terms of clouds that Kerbside supports, I think it would be relatively trivial to also support Proxmox, KubeVirt, or a list of static manually created virtual machines, but there’s only so many things one Mikal can do at once…)

Some of these cloud platforms have supported SPICE consoles for a while, but generally with warts. OpenStack for example only exposes them as HTML5 transcoded sessions with reduced functionality. oVirt exposes them via a “proxy” which is just squid (or equivalent), but its fairly dumb — it exposes the underlying hypervisor details to the client for example. I thought I could do better than that.

(more…)

Continue ReadingThe simplest boot target for the Kerbside SPICE VDI proxy CI

Issues building Kolla images with recent versions of Python requests

  • Post author:
  • Post category:OpenStack

If you find yourself having issues building Kolla docker container images with errors like this: INFO:kolla.common.utils:Using engine: docker INFO:kolla.common.utils:Found the container image folder at /home/mikal/src/kerbside-patches/venv-stable/2023.1-patches/share/kolla/docker ERROR:kolla.common.utils:Unable to connect to container engine daemon, exiting INFO:kolla.common.utils:Exception caught: Error while fetching server API version: Not supported URL scheme http+docker It is likely that you're running a recent version of requests which has an unresolved issue. Try downgrading requests manually, like this: $ ...venvpath.../bin/pip install requests==2.31.0 Collecting requests==2.31.0 Downloading requests-2.31.0-py3-none-any.whl (62 kB) ...snip... Successfully installed requests-2.31.0 Thanks for this stack overflow post for the pointer.  

Continue ReadingIssues building Kolla images with recent versions of Python requests

Validating a keystone token

  • Post author:
  • Post category:OpenStack

Once again I venture into the lands of poorly documented keystoneauth1 calls. This time, I want to be able to validate if a stored keystone authentication token is valid. Here's the best I could come up with, I'd be interested in others have something better. For this to work, we need a service account to create a keystone client with, and then we can ask that client questions about random other tokens... from keystoneauth1 import exceptions from keystoneauth1.identity import v3 from keystoneauth1 import session from keystoneclient.v3 import client def validate_keystone_token(service_auth, token): """Validate a keystone token. Returns True if the token is valid, False otherwise. """ # We need a keystone client as the service service_session = session.Session(auth=service_auth) service_keystone = client.Client(session=service_session) try: user = service_keystone.tokens.validate(token) except exceptions.http.NotFound: return False # Require that there be an access group with our configured name group = None for g in service_keystone.groups.list(): if g.name == 'mygroup': group = g if not group: return False # Require that the user be in that group try: service_keystone.users.check_in_group(user.user_id, group.id) except exceptions.http.NotFound: return False return True # Authenticate the service user service_auth = v3.Password( auth_url='http://kolla.home.stillhq.com:5000', username='admin', password='...', project_name='admin', user_domain_id='default', project_domain_id='default') # Create a token we can test user_auth =…

Continue ReadingValidating a keystone token

Using the openstacksdk with authentication arguments

  • Post author:
  • Post category:OpenStack

I wanted to authenticate against OpenStack recently, and had a lot of trouble finding documentation about how to authenticate just by passing arguments (as opposed to by using clouds.yaml or environment variables). Now that I have a working incantation, I figure I should write it down so I can find it again. Its also disappointing the OpenStack documentation doesn't appear to cover this particularly well... from keystoneauth1.identity import v3 from keystoneauth1 import session from openstack import connection auth = v3.Password( auth_url='http://kolla.home.stillhq.com:5000', username='admin', password='...', project_name='admin', user_domain_id='default', project_domain_id='default') sess = session.Session(auth=auth) conn = connection.Connection(session=sess) print([x.name for x in conn.list_servers()]) This code will authenticate using the arguments provided, and then list all the servers (instances) visible to that user. You're welcome.

Continue ReadingUsing the openstacksdk with authentication arguments

Turnover of Companies in OpenStack: Prevalence and Rationale

  • Post author:
  • Post category:OpenStack

This paper examines the withdrawal behaviour of corporate contributors to OpenStack, which seems particularly relevant given most contributions in OpenStack are corporately supported, and corporate engagement is declining over time. Its also directly relevant to my own experiences contributing to the project, so seemed like a thing I should read. One interesting aspect of the study is how they define withdrawal from contributions. For each company, they calculate an individual frequency of contribution, and then use that to determine if the company is still making contributions. That is, of a company only ever contributed once a year, we must wait at least a year to know that they have indeed stopped contributing. The paper finds that in more recent OpenStack releases, more companies are leaving contributions than joining. The authors assert that in general engaged developers are now less experienced than previously, which presents risks in terms of developer effectiveness as well as code quality. However, the paper does note that companies with smaller contributions are more likely to disengage than "sustaining companies", however that's largely because there are a huge number of companies contributing only one developer who makes a small number of commits. Unsurprisingly, the paper notes that…

Continue ReadingTurnover of Companies in OpenStack: Prevalence and Rationale

Deciding when to filter out large scale refactorings from code analysis

  • Post author:
  • Post category:OpenStack

I want to be able to see the level of change between OpenStack releases. However, there are a relatively small number of changes with simply huge amounts of delta in them -- they're generally large refactors or the delete which happens when part of a repository is spun out into its own project. I therefore wanted to explore what was a reasonable size for a change in OpenStack so that I could decide what maximum size to filter away as likely to be a refactor. After playing with a couple of approaches, including just randomly picking a number, it seems the logical way to decide is to simply plot a histogram of the various sizes, and then pick a reasonable place on the curve as the cutoff. Due to the large range of values (from zero lines of change to over a million!), I ended up deciding a logarithmic axis was the way to go. For the projects listed in the OpenStack compute starter kit reference set, that produces the following histogram:I feel that filtering out commits over 10,000 lines of delta feels justified based on that graph. For reference, the raw histogram buckets are:

Continue ReadingDeciding when to filter out large scale refactorings from code analysis

A quick summary of OpenStack release tags

I wanted a quick summary of OpenStack git release tags for a talk I am working on, and it turned out to be way more complicated than I expected. I ended up having to compile a table, and then turn that into a code snippet. In case its useful to anyone else, here it is: Or in python form for those so inclined: RELEASE_TAGS = { 'austin': {'all': '2010.1'}, 'bexar': {'all': '2011.1'}, 'cactus': {'all': '2011.2'}, 'diablo': {'all': '2011.3'}, 'essex': {'all': '2012.1.3'}, 'folsom': {'all': '2012.2.4'}, 'grizzly': {'all': '2013.1.5'}, 'havana': {'all': '2013.2.4'}, 'icehouse': {'all': '2014.1.5'}, 'juno': {'all': '2014.2.4'}, 'kilo': {'all': '2015.1.4'}, 'liberty': { 'glance': '11.0.2', 'keystone': '8.1.2', 'neutron': '7.2.0', 'nova': '12.0.6' }, 'mitaka': { 'glance': '12.0.0', 'keystone': '9.3.0', 'neutron': '8.4.0', 'nova': '13.1.4' }, 'newton': { 'glance': '13.0.0', 'keystone': '10.0.3', 'neutron': '9.4.1', 'nova': '14.1.0' }, 'ocata': { 'glance': '14.0.1', 'keystone': '11.0.4', 'neutron': '10.0.7', 'nova': '15.1.5' }, 'pike': { 'glance': '15.0.2', 'keystone': '12.0.3', 'neutron': '11.0.8', 'nova': '16.1.8' }, 'queens': { 'glance': '16.0.1', 'keystone': '13.0.4', 'neutron': '12.1.1', 'nova': '17.0.13' }, 'rocky': { 'glance': '17.0.1', 'keystone': '14.2.0', 'neutron': '13.0.7', 'nova': '18.3.0' }, 'stein': { 'glance': '18.0.1', 'keystone': '15.0.1', 'neutron': '14.4.2', 'nova': '19.3.2' }, 'train': { 'glance': '19.0.4', 'keystone': '16.0.1', 'neutron': '15.3.0', 'nova': '20.4.1' }, 'ussuri': {…

Continue ReadingA quick summary of OpenStack release tags

Rejected talk proposal: Shaken Fist, thought experiments in simpler IaaS clouds

This proposal was submitted for FOSDEM 2021. Given that acceptances were meant to be sent out on 25 December and its basically a week later I think we can assume that its been rejected. I've recently been writing up my rejected proposals, partially because I've put in the effort to write them and they might be useful elsewhere, but also because I think its important to demonstrate that its not unusual for experienced speakers to be rejected from these events. OpenStack today is a complicated beast -- not only does it try to perform well for large clusters, but it also embraces a diverse set of possible implementations from hypervisors, storage, networking, and more. This was a deliberate tactical choice made by the OpenStack community years ago, forming a so called "Big Tent" for vendors to collaborate in to build Open Source cloud options. It made a lot of sense at the time to be honest. However, OpenStack today finds itself constrained by the large number of permutations it must support, ten years of software and backwards compatability legacy, and a decreasing investment from those same vendors that OpenStack courted so actively. Shaken Fist makes a series of simplifying assumptions…

Continue ReadingRejected talk proposal: Shaken Fist, thought experiments in simpler IaaS clouds

Introducing Shaken Fist

The first public commit to what would become OpenStack Nova was made ten years ago today -- at Thu May 27 23:05:26 2010 PDT to be exact. So first off, happy tenth birthday to Nova! A lot has happened in that time -- OpenStack has gone from being two separate Open Source projects to a whole ecosystem, developers have come and gone (and passed away), and OpenStack has weathered the cloud wars of the last decade. OpenStack survived its early growth phase by deliberately offering a "big tent" to the community and associated vendors, with an expansive definition of what should be included. This has resulted in most developers being associated with a corporate sponser, and hence the decrease in the number of developers today as corporate interest wanes -- OpenStack has never been great at attracting or retaining hobbist contributors. My personal involvement with OpenStack started in November 2011, so while I missed the very early days I was around for a lot and made many of the mistakes that I now see in OpenStack. What do I see as mistakes in OpenStack in hindsight? Well, embracing vendors who later lose interest has been painful, and has increased the…

Continue ReadingIntroducing Shaken Fist

Exporting volumes from Cinder and re-creating COW layers

Today I wandered into a bit of a rat hole discovering how to export data from OpenStack Cinder volumes when you don't have admin permissions, and I thought it was worth documenting here so I remember it for next time. Let's assume that you have a Cinder volume named "child1", which is a 64gb volume originally cloned from "parent1". parent1 is a 7.9gb VMDK, but the only way I can find to extract child1 is to convert it to a glance image and then download the entire volume as a raw. Something like this: $ cinder upload-to-image $child1 "extract:$child1" Where $child1 is the UUID of the Cinder volume. You then need to find the UUID of the image in Glance, which the Cinder upload-to-image command will have told you, but you can also find by searching Glance for your image named "extract:$child1": $ glance image-list | grep "extract:$cinder_uuid" You now need to watch that Glance image until the status of the image is "active". It will go through a series of steps with names like "queued", and "uploading" first. Now you can download the image from Glance: $ glance image-download --file images/$child1.raw --progress $glance_uuid And then delete the intermediate glance…

Continue ReadingExporting volumes from Cinder and re-creating COW layers

End of content

No more pages to load