Category: Python

Posts relating to the python programming language.

All python packages require a pyproject.toml with modern pip

So last night Shaken Fist CI jobs started failing with errors like this (editted lightly for clarity):

Building wheels for collected packages: shakenfist-ci
  Building wheel for shakenfist-ci (setup.py): started
  Building wheel for shakenfist-ci (setup.py): finished with status 'error'
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [86 lines of output]
...
      ...setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        setuptools.SetuptoolsDeprecationWarning,
      installing to build/bdist.linux-x86_64/wheel
      running install
...
      warning: install_lib: byte-compiling is disabled, skipping.
      
      running install_egg_info
      Copying shakenfist_ci.egg-info to build/bdist.linux-x86_64/wheel/shakenfist_ci-0.0.1.dev2544-py3.7.egg-info
      running install_scripts
      error: invalid command 'bdist_wininst'
      [end of output]

This was pretty concerning. I know that a setup.py / setup.cfg style install is a little old school, but it was unexpected that it broke entirely. At first I thought I’d have to convert to poetry to unblock this, but Chet helpfully pointed out that this is as simple as adding a pyproject.toml file to the directory which contains your setup.py and setup.cfg. The basic issue is that a modern pip doesn’t assume that you’re going to use setuptools, so you need to tell it that you’re doing that in pyproject.toml. Then you’re unblocked.

So, just create a file named pyproject.toml in the setup.py / setup.cfg directory which contains this:

[build-system]
requires = ["setuptools >= 40.6.0", "wheel"]
build-backend = "setuptools.build_meta"

And you’re good to go. If you’re really curious, this page was quite helpful in working out what was happening.

Debian 10 buster bcrypt pip install breakage

So, as of today by Shaken Fist CI jobs for Debian 10 are failing to install bcrypt, with an error that looks like this:

Running setup.py install for bcrypt: started
    Running setup.py install for bcrypt: finished with status 'error'
    [ ... snip ... ]
    running build_rust
    
        =============================DEBUG ASSISTANCE=============================
        If you are seeing a compilation error please try the following steps to
        successfully install bcrypt:
        1) Upgrade to the latest pip and try again. This will fix errors for most
           users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
        2) Ensure you have a recent Rust toolchain installed. bcrypt requires
           rustc >= 1.56.0.
    
        Python: 3.7.3
        platform: Linux-4.19.0-21-amd64-x86_64-with-debian-10.12
        pip: 18.1
        setuptools: 65.2.0
        setuptools_rust: 1.5.1
        rustc: n/a
        =============================DEBUG ASSISTANCE=============================

I’m not really interested in debating why installing a python package requires a rust compiler, that has been dicussed elsewhere.

This specific breakage has been caused by bcrypt releasing 4.0.0, which has this in the changelog: “bcrypt is now implemented in Rust. Users building from source will need to have a Rust compiler available. Nothing will change for users downloading wheels.”

Unfortunately, you can’t just install rustc with apt, as it is both quite big (350mb), and too old (version 1.41.1 versus the required 1.56.0 or better). I also couldn’t find an Ubuntu PPA to misuse to get a more recent rustc.

Another answer here is to use rustup, which is yet another curl to a root shell installer, which isn’t a satisfying answer to me. The other option is of course just to pin bcrypt to pre 4.0.0, but I’d have to do that on every distribution, not just Debian 10 as best as I can tell.

Update: and then I re-read the ChangeLog. It turns out that pip wasn’t offering me wheels because the version of pip was too old. As long as you’re ok with not using an official Debian packaged version of pip, you can do this to get unstuck:

# pip3 install -U pip
# apt-get remove python3-pip
# /usr/local/bin/pip3 install -v bcrypt==4.0.0

A quick summary of OpenStack release tags

I wanted a quick summary of OpenStack git release tags for a talk I am working on, and it turned out to be way more complicated than I expected. I ended up having to compile a table, and then turn that into a code snippet. In case its useful to anyone else, here it is:

ReleaseRelease dateFinal release tag
AustinOctober 20102010.1
BexarFebruary 20112011.1
CactusApril 20112011.2
DiabloSeptember 20112011.3
EssexApril 20122012.1.3
FolsomSeptember 20122012.2.4
GrizzlyApril 20132013.1.5
HavanaOctober 20132013.2.4
IcehouseApril 20142014.1.5
JunoOctober 20142014.2.4
KiloApril 20152015.1.4
LibertyOctober 2015Glance: 11.0.2
Keystone: 8.1.2
Neutron: 7.2.0
Nova: 12.0.6
MitakaApril 2016Glance: 12.0.0
Keystone: 9.3.0
Neutron: 8.4.0
Nova: 13.1.4
NewtonOctober 2016Glance: 13.0.0
Keystone: 10.0.3
Neutron: 9.4.1
Nova: 14.1.0
OcataFebruary 2017Glance: 14.0.1
Keystone: 11.0.4
Neutron: 10.0.7
Nova: 15.1.5
PikeAugust 2017Glance: 15.0.2
Keystone: 12.0.3
Neutron: 11.0.8
Nova: 16.1.8
QueensFebruary 2018Glance: 16.0.1
Keystone: 13.0.4
Neutron: 12.1.1
Nova: 17.0.13
RockyAugust 2018Glance: 17.0.1
Keystone: 14.2.0
Neutron: 13.0.7
Nova: 18.3.0
SteinApril 2019Glance: 18.0.1
Keystone: 15.0.1
Neutron: 14.4.2
Nova: 19.3.2
TrainOctober 2019Glance: 19.0.4
Keystone: 16.0.1
Neutron: 15.3.0
Nova: 20.4.1
UssuriMay 2020Glance: 20.0.1
Keystone: 17.0.0
Neutron: 16.2.0
Nova: 21.1.1
VictoriaOctober 2020Glance: 21.0.0
Keystone: 18.0.0
Neutron: 17.0.0
Nova: 22.0.1

Or in python form for those so inclined:

RELEASE_TAGS = {
    'austin': {'all': '2010.1'},
    'bexar': {'all': '2011.1'},
    'cactus': {'all': '2011.2'},
    'diablo': {'all': '2011.3'},
    'essex': {'all': '2012.1.3'},
    'folsom': {'all': '2012.2.4'},
    'grizzly': {'all': '2013.1.5'},
    'havana': {'all': '2013.2.4'},
    'icehouse': {'all': '2014.1.5'},
    'juno': {'all': '2014.2.4'},
    'kilo': {'all': '2015.1.4'},
    'liberty': {
        'glance': '11.0.2',
        'keystone': '8.1.2',
        'neutron': '7.2.0',
        'nova': '12.0.6'
    },
    'mitaka': {
        'glance': '12.0.0',
        'keystone': '9.3.0',
        'neutron': '8.4.0',
        'nova': '13.1.4'
    },
    'newton': {
        'glance': '13.0.0',
        'keystone': '10.0.3',
        'neutron': '9.4.1',
        'nova': '14.1.0'
    },
    'ocata': {
        'glance': '14.0.1',
        'keystone': '11.0.4',
        'neutron': '10.0.7',
        'nova': '15.1.5'
    },
    'pike': {
        'glance': '15.0.2',
        'keystone': '12.0.3',
        'neutron': '11.0.8',
        'nova': '16.1.8'
    },
    'queens': {
        'glance': '16.0.1',
        'keystone': '13.0.4',
        'neutron': '12.1.1',
        'nova': '17.0.13'
    },
    'rocky': {
        'glance': '17.0.1',
        'keystone': '14.2.0',
        'neutron': '13.0.7',
        'nova': '18.3.0'
    },
    'stein': {
        'glance': '18.0.1',
        'keystone': '15.0.1',
        'neutron': '14.4.2',
        'nova': '19.3.2'
    },
    'train': {
        'glance': '19.0.4',
        'keystone': '16.0.1',
        'neutron': '15.3.0',
        'nova': '20.4.1'
    },
    'ussuri': {
        'glance': '20.0.1',
        'keystone': '17.0.0',
        'neutron': '16.2.0',
        'nova': '21.1.1'
    },
    'victoria': {
        'glance': '21.0.0',
        'keystone': '18.0.0',
        'neutron': '17.0.0',
        'nova': '22.0.1'
    }
}

Playing with the python prometheus query API

The last few days have been a bit icky around here, with my house apparently proudly residing in the major city with the dirtiest air in the world. So, I needed a distraction…

It has also been quite hot, so I wondered how my energy usage was going. I have prometheus monitoring of my power draw, so now seemed as good a time as any to learn how to do some historical querying over the API. I ended up with a python script which can output things like this: “Yesterday had a maximum temperature of 38 and we used 28.36 kwh. The average for similar days is 25.56 kwh.”

The code is on github if it is of interest to others. I am sure I could push more of this processing down into the prometheus engine, but I couldn’t see how to do it today. Hints welcome!

Quick hack: extracting the contents of a Docker image to disk

Hello! Please note I’ve written a little python tool called Occy Strap which makes this a bit easier, and can do some fancy things around importing and exporting multiple images. You might want to read about it?

For various reasons, I wanted to inspect the contents of a Docker image without starting a container. Docker makes it easy to get an image as a tar file, like this:

docker save -o foo.tar image

But if you extract that tar file you’ll find a configuration file and manifest as JSON files, and then a series of tar files, one per image layer. You use the manifest to determine in what order you extract the tar files to build the container filesystem.

That’s fiddly and annoying. So I wrote this quick python hack to extract an image tarball into a directory on disk that I could inspect:

#!/usr/bin/python3

# Call me like this:
#  docker-image-extract tarfile.tar extracted

import tarfile
import json
import os
import sys

image_path = sys.argv[1]
extracted_path = sys.argv[2]

image = tarfile.open(image_path)
manifest = json.loads(image.extractfile('manifest.json').read())

for layer in manifest[0]['Layers']:
    print('Found layer: %s' % layer)
    layer_tar = tarfile.open(fileobj=image.extractfile(layer))

    for tarinfo in layer_tar:
        print('  ... %s' % tarinfo.name)
        if tarinfo.isdev():
            print('  --> skip device files')
            continue

        dest = os.path.join(extracted_path, tarinfo.name)
        if not tarinfo.isdir() and os.path.exists(dest):
            print('  --> remove old version of file')
            os.unlink(dest)

        layer_tar.extract(tarinfo, path=extracted_path)

Hopefully that’s useful to someone else (or future me).

Support for Raspberry Pi and Orange Pi GPIOs in Home Assistant

So, I’ve been off in the GPIO library salt mines for a while, but am now ready to circle back and document how to get GPIO inputs and outputs working in Home Assistant. This now works on both Raspberry Pi and OrangePi, assuming that my patch gets merged.

First off, let’s talk about GPIO outputs. This is something which has been working for a while on both platforms (a while being a week or so, assuming you’ve patched Home Assistant with my pull request, but you’re all doing that right?).

To configure an output in Home Assistant, you would add the following to configuration.yaml:

rpi_gpio:
  board_family: orange_pi

switch:
 - platform: rpi_gpio
   ports:
     PA7: LED

Where board_family can be either “raspberry_pi” or “orange_pi”. Note that for Raspberry Pis, the pin numbers are always numbers whereas for OrangePi we are using “SUNXI” numbering, which is of the form “PA7”.

The circuit for this LED is really simple:

A simple LED circuit

Now we have a switch we can control in Home Assistant:

Raspberry Pi LED switch in Home Assistant

GPIO inputs are similar. The configuration looks like this:

rpi_gpio:
  board_family: orange_pi

binary_sensor:
 - platform: rpi_gpio
   invert_logic: true
   ports:
     PA7: PUSHYBUTTON

With a circuit like this:

A circuit with a button in it

invert_logic set to true is required because our circuit sends the value of PA7 to ground when the button is pressed.

A push button being pressed in Home AssistantNoting that sensors look different to switches in Home Assistant, you can see the binary sensor at the top right of the image, with its history being displayed in the dialog box in the foreground.

Updated examples for OrangePi GPIOs

As part of working through adding OrangePi support to Home Assistant, Alastair and I decided to change to a different GPIO library for OrangePi to avoid the requirement for Home Assistant to have access to /dev/mem.

I just realised that I hadn’t posted updated examples of how to do GPIO output with the new library. So here’s a quick post about that.

Assuming that we have an LED on GPIO PA7, which is pin 29, then the code to blink the LED would look like this with the new library:

import OPi.GPIO as GPIO
import time


# Note that we use SUNXI mappings here because its way less confusing than
# board mappsings. For example, these are all the same pin:
# sunxi: PA7 (the label on the board)
# board: 29
# gpio:  7

GPIO.setmode(GPIO.SUNXI)
GPIO.setwarnings(False)
GPIO.setup('PA7', GPIO.OUT)

while True:
    GPIO.output('PA7', GPIO.HIGH)
    time.sleep(1)
    GPIO.output('PA7', GPIO.LOW)
    time.sleep(1)

The most important thing there is the note about SUNXI pin mappings. I find the whole mapping scheme hugely confusing, unless you use SUNXI and then its all fine. So learn from my fail people!

What about input? Well, that’s not too bad either. Let’s assume that you have a button in a circuit like this:

A circuit with a button in it

The to read the button the polling way, you’d just do this:

import OPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.SUNXI)
GPIO.setwarnings(False)
GPIO.setup('PA7', GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

while True:
    print('Reading...')
    if GPIO.input('PA7') == GPIO.LOW:
        print('Pressed')
    else:
        print('Released')
    time.sleep(1)

Let’s pretend it didn’t take me ages to get that to work right because I had the circuit wrong, ok?

Now, we have self respect, so you wouldn’t actually poll like that. Instead you’d use edge detection, and end up with code like this:

import OPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.SUNXI)
GPIO.setwarnings(False)
GPIO.setup('PA7', GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

def event_callback(channel):
    print('Event detected: %s' % GPIO.input('PA7'))
    
GPIO.add_event_detect('PA7', GPIO.BOTH, callback=event_callback, bouncetime=50)

while True:
    time.sleep(1)

So there ya go.

GPIO inputs on Raspberry Pi

Now that I have GPIO outputs working nicely for Home Assistant using either a Raspberry Pi or an Orange Pi, I want to get GPIO inputs working as well. Naively, that’s pretty easy to do in python on the Raspberry Pi:

import RPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)
GPIO.setup(17, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

while True:
    print('Reading...')
    if GPIO.input(17) == GPIO.HIGH:
        print('Pressed')
    else:
        print('Released')
    time.sleep(1)

That code is of course horrid. Its horrid because its polling the state of the button, and its quite likely that I can sneak a button press in during one of those sleeps and it will never be noticed. Instead we can use edge detection callbacks to be informed of button presses as they happen:

import RPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)
GPIO.setup(17, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

def event_callback(channel):
    print('Event detected: %s' % GPIO.input(17))
    
GPIO.add_event_detect(17, GPIO.BOTH, callback=event_callback, bouncetime=50)

while True:
    time.sleep(1)

This second program provides helpful output like this:

pi@raspberrypi:~ $ <strong>python gpio_button_edge_detect.py</strong> 
Event detected: 1
Event detected: 0

Which is me pressing the button once (it go high when pressed, and then low again when released). This is of course with a button wired to GPIO17 with a current limiting resistor between the button and the 3.3v rail.

Pull Requests for the LCA2019 Home Automation tutorial

A quick list of things I did for the LCA2019 Home Automation tutorial. Of course Alistair did a lot more, but I still want to track these.

Further adventures in Home Assistant OrangePi GPIO

Its funny how a single sentence can change your course. In the last post about this work, I said:

We also need to run hass as root,  because OrangePi GPIO support requires access to /dev/mem for reasons I haven’t dug into just yet.

That’s turned out to be (reasonably) a pretty big sticking point upstream. Access to /dev/mem gives you a whole bunch of access to the machine that Home Assistant probably shouldn’t have.

Alastair went off spelunking because he’s more patient than me and found yet another OrangePi GPIO library. I think we’re up to three or four of these at the moment, but this is the first one we’ve found which supports the sysfs interface to GPIO pins. That’s exciting because it removes our run-as-root requirement. Its unexciting in that the sysfs interface has been deprecated by the kernel, but will remain supported for a while.

I think people would be within their rights to conclude that the state of GPIO libraries for OrangePi is a bit of a dumpster fire right now.

Anyways, the point of this post is mostly to write down how to use the sysfs interface to GPIO pins so that I can remember it later, I’ll take more about this new library and if it meets our needs in a later post.

The first step is to determine what pin number the GPIO pin is. On the OrangePi these are labelled with names like “PA7”, which is the 7th bit in the “A” GPIO register. To convert that into a pin number as used by sysfs you do this:

def pin_number(letter, digit):
    return (ord(letter) - ord('A')) * 32 + digit

So, pin_number(‘A’, 7) for PA7 is just … 7.

Note that I now recommend that people use SUNXI pin mapping, as its much less confusing. You can read more about alternative pin mappings in this post of worked OrangePi GPIO examples.

Now we can enable the pin, set it to output, and then blink the LED to prove it works:

# cd /sys/class/gpio
# echo 7 > export
# cd gpio7
# echo "out" > direction
# echo 1 > value
# sleep 1
# echo 0 > value

The next step? To make sure that the new GPIO library supports sysfs access to GPIOs on the OrangePi Prime.