Inspired by Alastair D’Silva‘s cunning plans for world domination, I’ve been googling around for automated lego sorting systems recently. This seems like a nice tractable machine learning problem with some robotics thrown in for fun.
The other day someone said to me that “they use the Spotify development model”, and I said “you who the what now?”. It was a super productive conversation that I am quite proud of.
So… in order to look like less of a n00b in the next conversation, what is the “Spotify development model”? Well, it turns out the Spotify came up with a series of tweaks to the standard Agile process in order to scale their engineering teams. If you google for “spotify development model” or “spotify agile” you’ll get lots and lots of third party blog posts about what Spotify did (I guess a bit like this one), but its surprisingly hard to find primary sources. The best I’ve found so far is this Quora answer from a former VP of Engineering at Spotify, although some of the resources he links to no longer exist.
For various reasons, I wanted to inspect the contents of a Docker image without starting a container. Docker makes it easy to get an image as a tar file, like this:
docker save -o foo.tar image
But if you extract that tar file you’ll find a configuration file and manifest as JSON files, and then a series of tar files, one per image layer. You use the manifest to determine in what order you extract the tar files to build the container filesystem.
That’s fiddly and annoying. So I wrote this quick python hack to extract an image tarball into a directory on disk that I could inspect:
# Call me like this:
# docker-image-extract tarfile.tar extracted
image_path = sys.argv
extracted_path = sys.argv
image = tarfile.open(image_path)
manifest = json.loads(image.extractfile('manifest.json').read())
for layer in manifest['Layers']:
print('Found layer: %s' % layer)
layer_tar = tarfile.open(fileobj=image.extractfile(layer))
for tarinfo in layer_tar:
print(' ... %s' % tarinfo.name)
print(' --> skip device files')
dest = os.path.join(extracted_path, tarinfo.name)
if not tarinfo.isdir() and os.path.exists(dest):
print(' --> remove old version of file')
Hopefully that’s useful to someone else (or future me).
The code is here, and the game can be played here.
I was curious about the newly available FTTN NBN service in my area, so I signed up to see what’s what. Of course, I need a usage API so that I can graph my usage in prometheus and grafana as everyone does these days. So I asked Aussie. The response I got was that I was welcome to reverse engineer the REST API that the customer portal uses.
I had toyed with using some spare GPIO lines and “hard coded” links on the HAT to identify board versions to the Raspberry Pi, but it turns out others have been here before and there’s a much better way. The Raspberry Pi folks have defined something called the “Hardware On Top” (HAT) specification which defines an i2c EEPROM which can be used to identify a HAT to the Raspberry Pi.
There are a couple of good resources I’ve found that help you do this thing — sparkfun have a tutorial which covers it, and there is an interesting forum post. However, I couldn’t find a simple tutorial for HAT designers that just covered exactly what they need to know and nothing else. There were also some gaps in those documents compared with my experiences, and I knew I’d need to look this stuff up again in the future. So I wrote this page.
First off, let’s talk about the hardware. I used an 24LC256P DIL i2c EEPROM — these are $2 on ebay, or $6 from Jaycar. The pins need to be wired like this:
Raspberry Pi Pin
GND (pins 6, 9, 14, 20, 25, 30, 34, 39)
All address pins tied to ground will place the EEPROM at address 50. This is the required address in the specification
You should also add a 3.9K pullup resistor from EEPROM pin 5 to 3.3V.
You must use this pin for the Raspberry Pi to detect the EEPROM on startup!
You should also add a 3.9K pullup resistor from EEPROM pin 6 to 3.3V.
You must use this pin for the Raspberry Pi to detect the EEPROM on startup!
Write protect. I don’t need this.
3.3V (pins 1 or 17)
The EEPROM is capable of being run at 5 volts, but must be run at 3.3 volts to work as a HAT identification EEPROM.
The specification requires that the data pin be on pin 27, the clock pin be on pin 28, and that the EEPROM be at address 50 on the i2c bus as described in the table above. There is also some mention of pullup resistors in both the data sheet and the HAT specification, but not in a lot of detail. The best I could find was a circuit diagram for a different EEPROM with the pullup resistors shown.
My test EEPROM wired up on a little breadboard looks like this:
And has a circuit diagram like this:
Next enable i2c on your raspberry pi. You also need to hand edit /boot/config.txt and then reboot. The relevant line of my config.txt look like this:
After reboot you should have an entry at /dev/i2c-0.
GOTCHA: you can’t probe the i2c bus that the HAT standard uses, and I couldn’t get flashing the EEPROM to work on that bus either.
Now time for our first gotcha — the version detection i2c bus is only enabled during boot and then turned off. An i2cdetect on bus zero wont show the device post boot for this reason. This caused an initial panic attack because I thought my EEPROM was dead, but that was just my twitchy nature showing through.
You can verify your EEPROM works by enabling bus one. To do this, add these lines to /boot/config.txt:
After a reboot you should have /dev/i2c-0 and /dev/i2c-1. You also need to move the EEPROM to bus 1 in order for it to be detected:
Raspberry Pi Pin
You’ll need to move the EEPROM back before you can use it for HAT detection.
Programming the EEPROM
You program the EEPROM with a set of tools provided by the raspberry pi folks. Check those out and compile them, they’re not packaged for raspbian that I can find:
pi@raspberrypi:~ $ git clone https://github.com/raspberrypi/hats
Cloning into 'hats'...
remote: Enumerating objects: 464, done.
remote: Total 464 (delta 0), reused 0 (delta 0), pack-reused 464
Receiving objects: 100% (464/464), 271.80 KiB | 119.00 KiB/s, done.
Resolving deltas: 100% (261/261), done.
pi@raspberrypi:~ $ cd hats/eepromutils/
pi@raspberrypi:~/hats/eepromutils $ ls
eepdump.c eepmake.c eeptypes.h README.txt
eepflash.sh eeprom_settings.txt Makefile
pi@raspberrypi:~/hats/eepromutils $ make
cc eepmake.c -o eepmake -Wno-format
cc eepdump.c -o eepdump -Wno-format
The file named eeprom_settings.txt is a sample of the settings for your HAT. Fiddle with that until it makes you happy, and then compile it:
And then we can flash our EEPROM, remembering that I’ve only managed to get flashing to work while the EEPROM is on bus 1 (pins 2 and 5):
$ sudo sh eepflash.sh -w -f=eeprom_settings.eep -t=24c256 -d=1
This will attempt to talk to an eeprom at i2c address 0xNOT_SET on bus 1. Make sure there is an eeprom at this address.
This script comes with ABSOLUTELY no warranty. Continue only if you know what you are doing.
Do you wish to continue? (yes/no): yes
0+1 records in
0+1 records out
107 bytes copied, 0.595252 s, 0.2 kB/s
Closing EEPROM Device.
Now move the EEPROM back to bus 0 (pins 27 and 28) and reboot. You should end up with entries in the device tree for the HAT. I get:
$ cd /proc/device-tree/hat/
$ for item in *
> echo "$item: "`cat $item`
Now I can have my code detect if the HAT is present, and if so what version. Comments welcome!
This one has been on my list for a little while — a nice 10km loop around the bottom of Urambi Hill. I did it as an out and back, although there is a loop option if you cross the bridge that was my turn around point. For the loop option cross the bridge, run a couple of hundred meters to the left and then cross the river again at the ford. Expect to get your feet wet if you choose that option!
Not particularly shady, but nice terrain. There is more vertical ascent than I expected, but it wasn’t crazy. I haven’t posted pictures of this run because it was super foggy when I did it so the pictures are just of white mist.
Gang Scan is an open source (and free) attendance tracking system based on custom RFID reader boards that communicate back to a server over wifi. The boards are capable of queueing scan events in the case of intermittent network connectivity, and the server provides simple reporting.
This is the third in a set of posts about the home automation tutorial from linux.conf.au 2019. You should probably read part 1 and part 2 before this post.
In the end Alistair decided that my home automation shield was defective, which is the cause of the errors from the past post. So I am instead running with the prototype shield that he handed me when I started helping with the tutorial preparation. That shield has some other bugs (misalignments of holes mainly), but is functional apart from that.
I have also decided that I’m not super excited by hassos, and just want to run the orangepi with the OWFS to MQTT gateway into my existing home assistant setup if possible, so I am going to focus on getting that bare component working for now.
Julian Yates kindly ran a bushwalking course for Scouts Australia over the last five days, which covered walking in Uncontrolled Terrain (the definition in the Australian VET scheme for the most difficult bushwalking — significant off track navigation in areas where emergency response will be hard to get). I helped with some of the instruction, but was also there working on my own bushwalking qualifications.
The walk was to Mount Bimberi, which is the highest point in the ACT. We started with a short night walk into Oldfield’s Hut on Friday night after a day of classroom work. The advantage of this was that we started Saturday at Oldfield’s Hut, which offered morning views which did not suck.
On Saturday morning we walked up to Mount Bimberi via Murray’s Gap. This involved following the ACT / NSW border up the hillside, which was reasonably well marked with tape and cairns.
Our route on the way to Bimberi:
And the way back:
On Sunday we walked back out to the cars and did the three hour drive back to Canberra. I’ll include the walk out here for completeness: