Further thoughts on Azure instance start times

  • Post author:
  • Post category:Azure

My post from the other day about slow instance starts on Azure caused some commentary (mainly on reddit) that prompted me to think more about all this. In the end, there were a few more experiments I wanted to run to see if I could squeeze more performance out of Azure. First off, looking at the logs from my initial testing it looks like resource groups are slow. The original terraform creates a resource group as part of the test and then cleans it up at the end. What if instead we had a single permanent resource group and created instances within that? Here is a series of instance starts and deletes using the terraform from the last post: You'll notice that there's no delete value for the last instance. That's because terraform crashed and never deleted the instance. You can also see that instance starts are somewhat consistent, except for being slower in the second half of the test than the first, and occasionally spiking out to very very slow. Oh, and deletes are almost always really slow. What happens if we use a permanent resource group and network? This means that all the "instance start terraform" is doing…

Continue ReadingFurther thoughts on Azure instance start times

Why is Azure so slow to start instances?

  • Post author:
  • Post category:Azure

I've been playing with terraform recently, and decided to see how different the terraform for launching a simple Ubuntu instance in various clouds is. There are two big questions there for me -- how big is the variation between OpenStack derived clouds; and how painful is it to move between the proprietary clouds? Part of this is because terraform doesn't present a standardised layer of cloud functionality, it has a provider per cloud. (Although, I suspect there's nothing stopping someone from writing a libcloud provider or something like that. It is an interesting idea which requires some additional thought.) My terraform implementations for each cloud are on github if you're interested. I don't want to spend a lot of analysis on the actual terraform, because I think the really interesting thing I found isn't where I expected it to be (there's a hint in the title for this post). That said, the OpenStack clouds vary mostly by capabilities. vexxhost for example seems to only offer flavors that require boot-from-volume. The proprietary clouds are complete re-writes, but are generally relatively simple and well documented. However, that interesting accidental thing -- as best as I can tell, Microsoft Azure is really really slow to…

Continue ReadingWhy is Azure so slow to start instances?

Coming to grips with Kubernetes in 2020: podcasts

  • Post author:
  • Post category:Kubernetes

It has become clear to me that it is time to care about Kubernetes more. I'm sure many people have cared for ages, but the things I want to build at the moment are starting to be more container based now that I am thinking more at the application layer than the cloud infrastructure layer. So how to do that? I thought I'd write down some notes on what has worked (or not) for me, in the hope it will help others. In this post, podcasts. I thought podcasts would be an interesting way to get started with some nice overviews. This is especially true because I'm already a pretty heavy podcast user, so it was easy to slot into my existing routine. Unfortunately this hasn't really worked out. I started with the podctl podcast, but they only ever talk about Red Hat stuff. It is very rare for a guest to not be a Red Hat employee for example. The presenters of this podcast seem to also really dislike OpenStack for reasons they never explain, which is annoying. Then I figured maybe the Google Kubernetes podcast would be better, but it often lacks the depth I am interested in. I…

Continue ReadingComing to grips with Kubernetes in 2020: podcasts

If I Understood You, Would I Have This Look on My Face?

  • Post author:
  • Post category:Book

This book discusses science and technical communication from the perspective of someone who comes from professional theatre and acting. Alan explains how his accidental discovery of the application of theatre sports to communication created an opportunity to teach technical communicators how to be more effective. Essentially, the argument is that empathy is essential to communication -- you need to be able to understand where your audience is starting and and where they're likely to get stuck before you can take them on the journey. Unsurprisingly given the topic of the book, this is a well written and engaging read. The book is nicely structured and uses regular anecdotes (some of them humorous) to get its message across. A detailed and fun read.

Continue ReadingIf I Understood You, Would I Have This Look on My Face?

Prometheus 2.12, query logging, and startup failures on macos

  • Post author:
  • Post category:Prometheus

Prometheus v2.12 added active query logging. The basic idea is that there is a mmaped JSON file that contains all of the queries currently running. If prometheus was to crash, that file would therefore be a list of the queries running at the time of the crash. Overall, not a bad idea. Some friends had recently added prometheus to their development environments. This is wired up to grafana dashboards for their microservices, and prometheus is configured to store 14 days worth of time series data via a persistent volume from the developer desktops. We did this because it is valuable for the developers to be able to see the history of metrics before and after their changes. Now we have a developer using macos as their primary development platform, and since prometheus 2.12 it hasn't worked. Specifically this developer is using parallels to provide the docker virtual machine on his mac. You can summarise the startup for prometheus in the dev environment like this: $ docker run ...stuff... ...snip... level=error ts=2019-09-15T02:20:23.520Z caller=query_logger.go:94 component=activeQueryTracker msg="Failed to mmap" file=/prometheus-data/data/queries.active Attemptedsize=20001 err="invalid argument" panic: Unable to create mmap-ed active query log goroutine 1 [running]: github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff9917af38, 0x15, 0x14, 0x2a6b7c0, 0xc00003c7e0, 0x2a6b7c0) /app/promql/query_logger.go:112 +0x4d2 main.main()…

Continue ReadingPrometheus 2.12, query logging, and startup failures on macos

The wonderful world of machine learning automated lego sorting

Inspired by Alastair D'Silva's cunning plans for world domination, I've been googling around for automated lego sorting systems recently. This seems like a nice tractable machine learning problem with some robotics thrown in for fun. Some cool projects if you're that way inclined: Sorting 2 Metric Tons of Lego A lego sorter using tensorflow This sounds like a great way to misspend some evenings to me...

Continue ReadingThe wonderful world of machine learning automated lego sorting

What is the Spotify model for Agile?

  • Post author:
  • Post category:Agile

The other day someone said to me  that “they use the Spotify development model”, and I said “you who the what now?”. It was a super productive conversation that I am quite proud of.

So… in order to look like less of a n00b in the next conversation, what is the “Spotify development model”? Well, it turns out the Spotify came up with a series of tweaks to the standard Agile process in order to scale their engineering teams. If you google for “spotify development model” or “spotify agile” you’ll get lots and lots of third party blog posts about what Spotify did (I guess a bit like this one), but its surprisingly hard to find primary sources. The best I’ve found so far is this Quora answer from a former VP of Engineering at Spotify, although some of the resources he links to no longer exist.

(more…)

Continue ReadingWhat is the Spotify model for Agile?

Quick hack: extracting the contents of a Docker image to disk

Hello! Please note I've written a little python tool called Occy Strap which makes this a bit easier, and can do some fancy things around importing and exporting multiple images. You might want to read about it? For various reasons, I wanted to inspect the contents of a Docker image without starting a container. Docker makes it easy to get an image as a tar file, like this: docker save -o foo.tar image But if you extract that tar file you'll find a configuration file and manifest as JSON files, and then a series of tar files, one per image layer. You use the manifest to determine in what order you extract the tar files to build the container filesystem. That's fiddly and annoying. So I wrote this quick python hack to extract an image tarball into a directory on disk that I could inspect: #!/usr/bin/python3 # Call me like this: # docker-image-extract tarfile.tar extracted import tarfile import json import os import sys image_path = sys.argv[1] extracted_path = sys.argv[2] image = tarfile.open(image_path) manifest = json.loads(image.extractfile('manifest.json').read()) for layer in manifest[0]['Layers']: print('Found layer: %s' % layer) layer_tar = tarfile.open(fileobj=image.extractfile(layer)) for tarinfo in layer_tar: print(' ... %s' % tarinfo.name) if tarinfo.isdev(): print(' -->…

Continue ReadingQuick hack: extracting the contents of a Docker image to disk

Mastermind in JavaScript

I've been learning JavaScript for the last few days, and I figured I'd implement Jacqui's favourite board game as a learning exercise. Jacqui loves a simple colour guessing game called Mastermind. In the game someone picks four coloured pins and then the player has to progressively guess what those colours are. In my JavaScript version the computer picks four colours, and you need to work out what they are. Click on the white squares to cycle through colours and then hit the "guess" button when you're ready to see how many you got right. The gray boxes in the top row will progressively reveal their colours as you guess them. The code is here, and the game can be played here.

Continue ReadingMastermind in JavaScript

End of content

No more pages to load