mikal – Page 14 – Made by Mikal

Prometheus 2.12, query logging, and startup failures on macos

Post author:mikal
Post published:September 15, 2019
Post category:Prometheus

Prometheus v2.12 added active query logging. The basic idea is that there is a mmaped JSON file that contains all of the queries currently running. If prometheus was to crash, that file would therefore be a list of the queries running at the time of the crash. Overall, not a bad idea. Some friends had recently added prometheus to their development environments. This is wired up to grafana dashboards for their microservices, and prometheus is configured to store 14 days worth of time series data via a persistent volume from the developer desktops. We did this because it is valuable for the developers to be able to see the history of metrics before and after their changes. Now we have a developer using macos as their primary development platform, and since prometheus 2.12 it hasn't worked. Specifically this developer is using parallels to provide the docker virtual machine on his mac. You can summarise the startup for prometheus in the dev environment like this: $ docker run ...stuff... ...snip... level=error ts=2019-09-15T02:20:23.520Z caller=query_logger.go:94 component=activeQueryTracker msg="Failed to mmap" file=/prometheus-data/data/queries.active Attemptedsize=20001 err="invalid argument" panic: Unable to create mmap-ed active query log goroutine 1 [running]: github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff9917af38, 0x15, 0x14, 0x2a6b7c0, 0xc00003c7e0, 0x2a6b7c0) /app/promql/query_logger.go:112 +0x4d2 main.main()…

An animated GIF resume

Post author:mikal
Post published:August 14, 2019
Post category:Uncategorized

The graphic designer at work and I were talking, and I challenged him to come up with a resume as an animated GIF. This is where he landed... I think its quite clever. Need a graphic designer or video team? Consider onefishsea.

The wonderful world of machine learning automated lego sorting

Post author:mikal
Post published:August 8, 2019
Post category:Electronics

Inspired by Alastair D'Silva's cunning plans for world domination, I've been googling around for automated lego sorting systems recently. This seems like a nice tractable machine learning problem with some robotics thrown in for fun. Some cool projects if you're that way inclined: Sorting 2 Metric Tons of Lego A lego sorter using tensorflow This sounds like a great way to misspend some evenings to me...

What is the Spotify model for Agile?

Post author:mikal
Post published:August 3, 2019
Post category:Agile

The other day someone said to me that “they use the Spotify development model”, and I said “you who the what now?”. It was a super productive conversation that I am quite proud of.

So… in order to look like less of a n00b in the next conversation, what is the “Spotify development model”? Well, it turns out the Spotify came up with a series of tweaks to the standard Agile process in order to scale their engineering teams. If you google for “spotify development model” or “spotify agile” you’ll get lots and lots of third party blog posts about what Spotify did (I guess a bit like this one), but its surprisingly hard to find primary sources. The best I’ve found so far is this Quora answer from a former VP of Engineering at Spotify, although some of the resources he links to no longer exist.

(more…)

Quick hack: extracting the contents of a Docker image to disk

Post author:mikal
Post published:July 29, 2019
Post category:Containers Docker Hacks Python

Hello! Please note I've written a little python tool called Occy Strap which makes this a bit easier, and can do some fancy things around importing and exporting multiple images. You might want to read about it? For various reasons, I wanted to inspect the contents of a Docker image without starting a container. Docker makes it easy to get an image as a tar file, like this: docker save -o foo.tar image But if you extract that tar file you'll find a configuration file and manifest as JSON files, and then a series of tar files, one per image layer. You use the manifest to determine in what order you extract the tar files to build the container filesystem. That's fiddly and annoying. So I wrote this quick python hack to extract an image tarball into a directory on disk that I could inspect: #!/usr/bin/python3 # Call me like this: # docker-image-extract tarfile.tar extracted import tarfile import json import os import sys image_path = sys.argv[1] extracted_path = sys.argv[2] image = tarfile.open(image_path) manifest = json.loads(image.extractfile('manifest.json').read()) for layer in manifest[0]['Layers']: print('Found layer: %s' % layer) layer_tar = tarfile.open(fileobj=image.extractfile(layer)) for tarinfo in layer_tar: print(' ... %s' % tarinfo.name) if tarinfo.isdev(): print(' -->…

Mastermind in JavaScript

Post author:mikal
Post published:July 23, 2019
Post category:JavaScript

I've been learning JavaScript for the last few days, and I figured I'd implement Jacqui's favourite board game as a learning exercise. Jacqui loves a simple colour guessing game called Mastermind. In the game someone picks four coloured pins and then the player has to progressively guess what those colours are. In my JavaScript version the computer picks four colours, and you need to work out what they are. Click on the white squares to cycle through colours and then hit the "guess" button when you're ready to see how many you got right. The gray boxes in the top row will progressively reveal their colours as you guess them. The code is here, and the game can be played here.

A nerd snipe, in which I reverse engineer the Aussie Broadband usage API

Post author:mikal
Post published:June 20, 2019
Post category:AussieBB

I was curious about the newly available FTTN NBN service in my area, so I signed up to see what's what. Of course, I need a usage API so that I can graph my usage in prometheus and grafana as everyone does these days. So I asked Aussie. The response I got was that I was welcome to reverse engineer the REST API that the customer portal uses. So I did. I give you my super simple implementation of an Aussie Broadband usage client in Python. Patches of course are welcome. I've now released the library on pypi under the rather innovative name of "aussiebb", so installing it is as simple as: $ pip install aussiebb

Raspberry Pi HAT identity EEPROMs, a simple guide

Post author:mikal
Post published:June 13, 2019
Post category:Electronics Gang Scan

I've been working on a RFID scanner than can best be described as an overly large Raspberry Pi HAT recently. One of the things I am grappling with as I get closer to production boards is that I need to be able to identify what version of the HAT is currently installed -- the software can then tweak its behaviour based on the hardware present. I had toyed with using some spare GPIO lines and "hard coded" links on the HAT to identify board versions to the Raspberry Pi, but it turns out others have been here before and there's a much better way. The Raspberry Pi folks have defined something called the "Hardware On Top" (HAT) specification which defines an i2c EEPROM which can be used to identify a HAT to the Raspberry Pi. There are a couple of good resources I've found that help you do this thing -- sparkfun have a tutorial which covers it, and there is an interesting forum post. However, I couldn't find a simple tutorial for HAT designers that just covered exactly what they need to know and nothing else. There were also some gaps in those documents compared with my experiences, and…

Trail run: the base of Urambi Hill

Post author:mikal
Post published:June 11, 2019
Post category:Trail run

This one has been on my list for a little while -- a nice 10km loop around the bottom of Urambi Hill. I did it as an out and back, although there is a loop option if you cross the bridge that was my turn around point. For the loop option cross the bridge, run a couple of hundred meters to the left and then cross the river again at the ford. Expect to get your feet wet if you choose that option! Not particularly shady, but nice terrain. There is more vertical ascent than I expected, but it wasn't crazy. I haven't posted pictures of this run because it was super foggy when I did it so the pictures are just of white mist. [kml: 20190609-1]

What is Gang Scan?

Post author:mikal
Post published:June 5, 2019
Post category:Gang Scan

Gang Scan is an open source (and free) attendance tracking system based on custom RFID reader boards that communicate back to a server over wifi. The boards are capable of queueing scan events in the case of intermittent network connectivity, and the server provides simple reporting.