python

Tensorflow, ROCm, and version mismatches

I’ve fallen into this pattern where I do an hour or so of self-directed learning in the mornings before going to work. Until recently it was an excellent CMU course on the design of SQL database systems, which I’ve mentioned previously here. I’ve finished that, so I thought I would do something shorter and fun as a break before finding another course to do. I chose The freeCodeCamp.org hot dog or not hot dog tensorflow course. 90 minutes seemed achievable, and I too wish to know if an object in front of me is a hot dog or not.

(more…)

Continue ReadingTensorflow, ROCm, and version mismatches

Converting from setup.py and PBR to pyproject.toml

  • Post author:
  • Post category:Python

I make no claim to be an expert at this, but I did just need to convert a project from a slightly complicated setup.py / PBR configuration to pyproject.toml and thought I should write up where I landed. I say “slightly complicated” because there are a few very OpenStacky things I like to do in these things. Specifically:

  • version numbers are driven by git tags not hard coded in the configuration file.
  • console scripts are a thing.
  • I often include data files in the built package.

So here’s an example of all of those things that is working ok for me:

(more…)

Continue ReadingConverting from setup.py and PBR to pyproject.toml

virtio-vsock: python examples of running the server in the guest

I’ve been using virtio-serial for communications between Linux hypervisors and guest virtual machines for ages. Lots of other people do it to — the qemu guest agent for example is implemented like this. In fact, I think that’s where I got my original thoughts on the matter from. However, virtio-serial is actually fairly terrible to write against as a programming model, because you’re left to do all the multiplexing of various requests down the channel and surely there’s something better?

Well… There is! virtio-vsock is basically the same concept, except it uses the socket interface. You can have more than one connection open and the sockets layer handles multiplexing by magic. This massively simplifies the programming model for supporting concurrent users down the channel. So that’s actually pretty cool. I should credit Kata Containers with noticing this quality of life improvement nearly a decade before I did, but I get there in the end.

(more…)

Continue Readingvirtio-vsock: python examples of running the server in the guest

An easier to parse version of “yum history”

I got interested today in trying to come up with a solid way of determining when updates were last applied to a RHEL-derived Linux instance. Previously we’d been inferring it from the kernel version, but it turns out there is a convenient “yum history”  or “dnf history” command which will show you all the previous transactions that the package database has seen. However, the output is hard to parse in a script.

(more…)

Continue ReadingAn easier to parse version of “yum history”

A gotcha with the Walrus operator

  • Post author:
  • Post category:Python

In New python syntax I was previously unaware of, I discussed some new operators I'd recently discovered. One of them is called the Walrus operator, which lets you write code like this: list = ['a', 'b', 'c'] def get_one(): if not list: return None return list.pop() while one := get_one(): print(one) See where we do the assignment inside the while? That code returns: c b a Which is as expected. However, the Walrus operator is strict about needing a None returned to end the iteration. I had code which was more like this: list = [('a', 1), ('b', 2), ('c', 3)] def get_one(): if not list: return None, None return list.pop() while one := get_one(): print(one) And the while loop never terminates. It just prints (None, None) over and over. So there you go.

Continue ReadingA gotcha with the Walrus operator

New python syntax I was previously unaware of

  • Post author:
  • Post category:Python

This post documents the new syntax features I learned about while reading cpython internals. You can create more than one context manager on a single line. So for example Shaken Fist contains code like this: with open(path + '.new', 'w') as o: with open(path, 'r') as i: ... That can now be written like this: with open(path + '.new', 'w') as o, open(path, 'r') as i: ...   You can assign values in a while statement, but only one. Instead of this: d = f.read(8000) while f: ... d = f.read(8000) You can write this: while d := f.read(8000): ... But unfortunately this doesn't work: while a, b := thing(): ...   You can use underscores as commands in long numbers to make them easier to read. For example, you can write 1000000 or 1_000_000 and they both mean the same thing.   You can refer to positional arguments by name, but you can also disable that. I didn't realise that this was valid python: def foo(bar=None): print(bar) foo(bar='banana') You can turn it off with a forward slash in the argument list though, which should separate positional arguments from named arguments: def foo(bar, /, extra=None): print(bar) print(extra) foo('banana', extra='frog') The above example…

Continue ReadingNew python syntax I was previously unaware of

cpython internals

  • Post author:
  • Post category:BookPython

I have been paid money to write Python code since about 2006, so I figured it was probably time that I should understand some of the inner workings of Python. I therefore picked up two books on the topic, this one being the first of the two.

This book to be honest isn’t completely what I expected. Its very well written and quite interesting, but its more about the things you’d need to know to become a Python core developer, rather than the things you should know as a user of Python like how the Python dictionary implementation is built.

(If you want that specifically, this video is an excellent introduction).

(more…)

CPython Internals Book Cover CPython Internals
Anthony Shaw
May 5, 2021
396

Get your guided tour through the Python 3.9 interpreter: Unlock the inner workings of the Python language, compile the Python interpreter from source code, and participate in the development of CPython. Are there certain parts of Python that just seem like magic? This book explains the concepts, ideas, and technicalities of the Python interpreter in an approachable and hands-on fashion. Once you see how Python works at the interpreter level, you can optimize your applications and fully leverage the power of Python.

Continue Readingcpython internals

Validating a keystone token

  • Post author:
  • Post category:OpenStack

Once again I venture into the lands of poorly documented keystoneauth1 calls. This time, I want to be able to validate if a stored keystone authentication token is valid. Here's the best I could come up with, I'd be interested in others have something better. For this to work, we need a service account to create a keystone client with, and then we can ask that client questions about random other tokens... from keystoneauth1 import exceptions from keystoneauth1.identity import v3 from keystoneauth1 import session from keystoneclient.v3 import client def validate_keystone_token(service_auth, token): """Validate a keystone token. Returns True if the token is valid, False otherwise. """ # We need a keystone client as the service service_session = session.Session(auth=service_auth) service_keystone = client.Client(session=service_session) try: user = service_keystone.tokens.validate(token) except exceptions.http.NotFound: return False # Require that there be an access group with our configured name group = None for g in service_keystone.groups.list(): if g.name == 'mygroup': group = g if not group: return False # Require that the user be in that group try: service_keystone.users.check_in_group(user.user_id, group.id) except exceptions.http.NotFound: return False return True # Authenticate the service user service_auth = v3.Password( auth_url='http://kolla.home.stillhq.com:5000', username='admin', password='...', project_name='admin', user_domain_id='default', project_domain_id='default') # Create a token we can test user_auth =…

Continue ReadingValidating a keystone token

Using the openstacksdk with authentication arguments

  • Post author:
  • Post category:OpenStack

I wanted to authenticate against OpenStack recently, and had a lot of trouble finding documentation about how to authenticate just by passing arguments (as opposed to by using clouds.yaml or environment variables). Now that I have a working incantation, I figure I should write it down so I can find it again. Its also disappointing the OpenStack documentation doesn't appear to cover this particularly well... from keystoneauth1.identity import v3 from keystoneauth1 import session from openstack import connection auth = v3.Password( auth_url='http://kolla.home.stillhq.com:5000', username='admin', password='...', project_name='admin', user_domain_id='default', project_domain_id='default') sess = session.Session(auth=auth) conn = connection.Connection(session=sess) print([x.name for x in conn.list_servers()]) This code will authenticate using the arguments provided, and then list all the servers (instances) visible to that user. You're welcome.

Continue ReadingUsing the openstacksdk with authentication arguments

Exploring more efficient remote large file storage

My primary personal project is a thing called Shaken Fist these days — it is an infrastructure as a service cloud akin to OpenStack Compute, but smaller and simpler. Shaken Fist doesn’t have an equivalent to the OpenStack Image service, instead letting your describe your instance images by a standard URL. One of the things Shaken Fist does to be easier to use is it maintains an official repository of common images, which allows users to refer to those images with a shorthand syntax instead of a complete URL. The images also contain small customizations (mainly including the Shaken Fist in-guest agent), which means I can’t just use the official upstream cloud images like OpenStack does.

The images were stored at DreamHost until this week, when a robot decided that they looked like offline backups, despite being served to the Internet via HTTP and being used regularly (although admittedly not frequently). DreamHost unilaterally decided to delete the web site, so now I am looking for new image hosting services, and thinking about better ways to build an image store.

(Oh, and recommending to anyone who asks that they consider using someone less capricious than DreamHost for their hosting needs).

(more…)

Continue ReadingExploring more efficient remote large file storage

End of content

No more pages to load

Older Posts