Recent Posts

Read more about the article The KSM and I

The KSM and I

  • Post author:
  • Post category:Linux

I spent much of yesterday playing with KSM (Kernel Shared Memory, or Kernel Samepage Merging depending on which universe you come from). Unix kernels store memory in "pages" which are moved in and out of memory as a single block. On most Linux architectures pages are 4,096 bytes long. KSM is a Linux Kernel feature which scans memory looking for identical pages, and then de-duplicating them. So instead of having two pages, we just have one and have two processes point at that same page. This has obvious advantages if you're storing lots of repeating data. Why would you be doing such a thing? Well the traditional answer is virtual machines. Take my employer's systems for example. We manage virtual learning environments for students, where every student gets a set of virtual machines to do their learning thing on. So, if we have 50 students in a class, we have 50 sets of the same virtual machine. That's a lot of duplicated memory. The promise of KSM is that instead of storing the same thing 50 times, we can store it once and therefore fit more virtual machines onto a single physical machine. For my experiments I used libvirt /…

Continue ReadingThe KSM and I

Introducing Shaken Fist

The first public commit to what would become OpenStack Nova was made ten years ago today -- at Thu May 27 23:05:26 2010 PDT to be exact. So first off, happy tenth birthday to Nova! A lot has happened in that time -- OpenStack has gone from being two separate Open Source projects to a whole ecosystem, developers have come and gone (and passed away), and OpenStack has weathered the cloud wars of the last decade. OpenStack survived its early growth phase by deliberately offering a "big tent" to the community and associated vendors, with an expansive definition of what should be included. This has resulted in most developers being associated with a corporate sponser, and hence the decrease in the number of developers today as corporate interest wanes -- OpenStack has never been great at attracting or retaining hobbist contributors. My personal involvement with OpenStack started in November 2011, so while I missed the very early days I was around for a lot and made many of the mistakes that I now see in OpenStack. What do I see as mistakes in OpenStack in hindsight? Well, embracing vendors who later lose interest has been painful, and has increased the…

Continue ReadingIntroducing Shaken Fist

A totally cheating sour dough starter

  • Post author:
  • Post category:Cooking

This is the third in a series of posts documenting my adventures in making bread during the COVID-19 shutdown. I'd like to imagine I was running science experiments in making bread on my kids, but really all I was trying to do was eat some toast. I'm not sure what it was like in other parts of the world, but during the COVID-19 pandemic Australia suffered a bunch of shortages -- toilet paper, flour, and yeast were among those things stores simply didn't have any stock of. Luckily we'd only just done a costco shop so were ok for toilet paper and flour, but we were definitely getting low on yeast. The obvious answer is a sour dough starter, but I'd never done that thing before. In the end my answer was to cheat and use this recipe. However, I found the instructions unclear, so here's what I ended up doing: Starting off 2 cups of warm water 2 teaspoons of dry yeast 2 cups of bakers flour Mix these three items together in a plastic container with enough space for the mix to double in size. Place in a warm place (on the bench on top of the dish…

Continue ReadingA totally cheating sour dough starter

A super simple non-breadmaker loaf

  • Post author:
  • Post category:Cooking

This is the second in a series of posts documenting my adventures in making bread during the COVID-19 shutdown. Yes I know all the cool kids made bread for themselves during the shutdown, but I did it too! So here we were, in the middle of a pandemic which closed bakeries and cancelled almost all of my non-work activities. I found this animated GIF on Reddit for a super simple no-kneed bread and decided to give it a go. It turns out that a few things are true: animated GIFs are a super terrible way store recipes that animated GIF was a export of this YouTube video which originally accompanied this blog post and that I only learned these things while to trying and work out who to credit for this recipe The basic recipe is really easy -- chuck the following into a big bowl, stir, and then cover with a plate. Leave resting a warm place for a long time (three or four hours), then turn out onto a floured bench. Fold into a ball with flour, and then bake. You can see a more detailed version in the YouTube video above. 3 cups of bakers flour (not…

Continue ReadingA super simple non-breadmaker loaf

A breadmaker loaf my kids will actually eat

  • Post author:
  • Post category:Cooking

My dad asked me to document some of my baking experiments from the recent natural disasters, which I wanted to do anyway so that I could remember the recipes. Its taken me a while to get around to though, because animated GIFs on reddit are a terrible medium for recipe storage, and because I've been distracted with other shiney objects. That said, let's start with the basics -- a breadmaker bread that my kids will actually eat. This recipe took a bunch of iterations to get right over the last year or so, but I'll spare you the long boring details. However, I suspect part of the problem is that the receipe varies by bread maker. Oh, and the salt is really important -- don't skip the salt! Wet ingredients (add first) 1.5 cups of warm water (we have an instantaneous gas hot water system, so I pick 42 degrees) 0.25 cups of oil (I use bran oil) Dry ingredients (add second) I just kind of chuck these in, although I tend to put the non-flour ingredients in a corner together for reasons that I can't explain. 3.5 cups of bakers flour (must be bakers flour, not plain flour) 2…

Continue ReadingA breadmaker loaf my kids will actually eat

Exporting volumes from Cinder and re-creating COW layers

Today I wandered into a bit of a rat hole discovering how to export data from OpenStack Cinder volumes when you don't have admin permissions, and I thought it was worth documenting here so I remember it for next time. Let's assume that you have a Cinder volume named "child1", which is a 64gb volume originally cloned from "parent1". parent1 is a 7.9gb VMDK, but the only way I can find to extract child1 is to convert it to a glance image and then download the entire volume as a raw. Something like this: $ cinder upload-to-image $child1 "extract:$child1" Where $child1 is the UUID of the Cinder volume. You then need to find the UUID of the image in Glance, which the Cinder upload-to-image command will have told you, but you can also find by searching Glance for your image named "extract:$child1": $ glance image-list | grep "extract:$cinder_uuid" You now need to watch that Glance image until the status of the image is "active". It will go through a series of steps with names like "queued", and "uploading" first. Now you can download the image from Glance: $ glance image-download --file images/$child1.raw --progress $glance_uuid And then delete the intermediate glance…

Continue ReadingExporting volumes from Cinder and re-creating COW layers

The Calculating Stars

  • Post author:
  • Post category:Book

Winner of both a Hugo, Locus and a Nebula, this book is about a mathematical prodigy battling her way into a career as an astronaut in a post-apolocalyptic 1950s America. Along the way she has to take on the embedded sexism of America in the 50s, as well as her own mild racism. Worse, she suffers from an anxiety condition. The book is engaging and well written, with an alternative history plot line which believable and interesting. In fact, its quite topical for our current time. I really enjoyed this book and I will definitely be reading the sequel.

Continue ReadingThe Calculating Stars

Configuring load balancing and location headers on Google Cloud

I have a need at the moment to know where my users are in the world. This helps me to identify what compute resources to serve their request with in order to reduce the latency they experience. So how do you do that thing with Google Cloud? The first step is to setup a series of test backends to send traffic to. I built three regions: Sydney; London; and Los Angeles. It turns out in hindsight that wasn't actually nessesary though -- this would work with a single backend just as well. For my backends I chose a minimal Ubuntu install, running this simple backend HTTP service. I had some initial trouble finding a single page which walked through the setup of the Google Cloud load balancer to do what I wanted, which is the main reason for writing this post. The steps are: Create your test instances and configure the backend on them. I ended up with a setup like this: Next setup instance groups to contain these instances. I chose unmanaged instance groups (that is, I don't want autoscaling). You need to create one per region. But wait! There's one more layer of abstraction. We need a backend…

Continue ReadingConfiguring load balancing and location headers on Google Cloud

Interviewing hints (or, so you’ve been laid off…)

This post is an attempt to collect a set of general hints and tips for resumes and interviews. It is not concrete truth though, like all things this process is subjective and will differ from place to place. It originally started as a Google doc shared around a previous workplace during some layoffs, but it seems more useful than that so I am publishing it publicly. I'd welcome comments if you think it will help others. So something bad happened I have the distinction of having been through layoffs three times now. I think there are some important first steps: Take a deep breath. Hug your loved ones and then go and sweat on something -- take a walk, go to the gym, whatever works for you. Research shows that exercise is a powerful mood stabiliser. Make a plan. Who are you going to apply with? Who could refer you? What do you want to do employment wise? Updating your resume is probably a good first step in that plan. Treat finding a job as your job. You probably can’t do it for eight hours a day, but it should be your primary goal for each “workday”. Have a todo…

Continue ReadingInterviewing hints (or, so you’ve been laid off…)

Writing a terraform remote state server

  • Post author:
  • Post category:Terraform

Terraform is a useful tool for deploying cloud resources. This post isn't an introduction to terraform, so I'll assume you already know and love it. If you want more, then this getting started guide would be a sensible start. At its most basic level, terraform deploys cloud resources and stores information about those resources in a file on local disk called terraform.tfstate -- it needs that state information so it can make later changes to the deployment, be those modifying resources in use or tearing the whole deployment down. If you had an operations team working on an environment, then you could store the tfstate file in git or a shared filesystem so that the entire team could manage the deployment. However, there is nothing with that approach that stops two members of the team making overlapping changes. That's where terraform state servers come in. State servers can implement optional locking, which stops overlapping operations from happening. The protocol that these servers talk isn't well documented (that I could find). I wanted to explore that more, so I wrote a simple terraform HTTP state server in python. To use this state server, configure your terraform file as per demo.tf. The…

Continue ReadingWriting a terraform remote state server

End of content

No more pages to load

Newer Posts
Older Posts