Google – Made by Mikal

Exporting volumes from Cinder and re-creating COW layers

Today I wandered into a bit of a rat hole discovering how to export data from OpenStack Cinder volumes when you don’t have admin permissions, and I thought it was worth documenting here so I remember it for next time.

Let’s assume that you have a Cinder volume named “child1”, which is a 64gb volume originally cloned from “parent1”. parent1 is a 7.9gb VMDK, but the only way I can find to extract child1 is to convert it to a glance image and then download the entire volume as a raw. Something like this:

$ cinder upload-to-image $child1 "extract:$child1"

Where $child1 is the UUID of the Cinder volume. You then need to find the UUID of the image in Glance, which the Cinder upload-to-image command will have told you, but you can also find by searching Glance for your image named “extract:$child1”:

$ glance image-list | grep "extract:$cinder_uuid"

You now need to watch that Glance image until the status of the image is “active”. It will go through a series of steps with names like “queued”, and “uploading” first.

Now you can download the image from Glance:

$ glance image-download --file images/$child1.raw --progress $glance_uuid

And then delete the intermediate glance image:

$ glance image-delete $glance_uuid

I have a bad sample script which does this in my junk code repository if that is helpful.

What you have at the end of this is a 64gb raw disk file in my example. You can convert that file to qcow2 like this:

$ qemu-img convert $child1.raw $child1.qcow2

But you’re left with a 64gb qcow2 file for your troubles. I experimented with virt-sparsify to reduce the size of this image, but it doesn’t work in my case (no space is saved), I suspect because the disk image has multiple partitions because it originally came from a VMWare environment.

Luckily qemu-img can also re-create the COW layer that existing on the admin-only side of the public cloud barrier. You do this by rebasing the converted qcow2 file onto the original VMDK file like this:

$ qemu-img create -f qcow2 -b $parent1.qcow2 $child1.delta.qcow2
$ qemu-img rebase -b $parent1.vmdk $child1.delta.qcow2

In my case I ended up with a 289mb $child1.delta.qcow2 file, which isn’t too shabby. It took about five minutes to produce that delta on my Google Cloud instance from a 7.9gb backing file and a 64gb upper layer.

Configuring load balancing and location headers on Google Cloud

I have a need at the moment to know where my users are in the world. This helps me to identify what compute resources to serve their request with in order to reduce the latency they experience. So how do you do that thing with Google Cloud?

The first step is to setup a series of test backends to send traffic to. I built three regions: Sydney; London; and Los Angeles. It turns out in hindsight that wasn’t actually nessesary though — this would work with a single backend just as well. For my backends I chose a minimal Ubuntu install, running this simple backend HTTP service.

I had some initial trouble finding a single page which walked through the setup of the Google Cloud load balancer to do what I wanted, which is the main reason for writing this post. The steps are:

Create your test instances and configure the backend on them. I ended up with a setup like this:

Next setup instance groups to contain these instances. I chose unmanaged instance groups (that is, I don’t want autoscaling). You need to create one per region.

But wait! There’s one more layer of abstraction. We need a backend service. The configuration for these is cunningly hidden on the load balancing page, on a separate tab. Create a service which contains our three instance groups:

I’ve also added a health check to my service, which just requests “/healthz” from each instance and expects a response of “OK” for healthy backends.

The backend service is also where we configure our extra headers. Click on the “advanced configurations” link, and more options appear:

Here I setup the extra HTTP headers the load balancer should insert: X-Region; X-City; and X-Lat-Lon.

And finally we can configure the load balancer. I selected a “HTTP(S) load balancer”, as I only care about incoming HTTP and HTTPS traffic. Obviously you set the load balancer to route traffic from the Internet to your VMs, and you wire the backend of the load balancer to your service. Select your backend service for the backend.

Now we can test! If I go to my load balancer in a web browser, I now get a result like this:

The top part of the page is just the HTTP headers from the request. You can see that we’re now getting helpful location headers. Mission accomplished!

Setting up VXLAN between nested virt VMs on Google Compute Engine

I wanted to play with a VXLAN mesh between VMs on more than one hypervisor node, but the setup for VXLAN ended up being a separate post because it was a bit long. Read that post first if you want to follow the instructions here.

Now that we have a working VXLAN mesh between our two nodes we can move on to installing libvirt (which is called libvirt-daemon-system on Debian, not libvirt-bin as on Ubuntu):

sudo apt-get install -y qemu-kvm libvirt-daemon-system
sudo virsh net-start default
sudo virsh net-autostart --network default

I’m going to use a little python helper to launch my VMs, so I need some other dependancies as well:

sudo apt-get install -y python3-pip pkg-config libvirt-dev git

git clone https://github.com/mikalstill/shakenfist
cd shakenfist
git checkout 6bfac153d249752b27d224ad9d079095b640498e

sudo mkdir /srv/shakenfist
sudo cp template.debian.xml /srv/shakenfist/template.xml
sudo pip3 install -r requirements.txt

Let’s launch a quick test VM to make sure the helper works:

sudo python3 daemon.py
sudo virsh list

You can destroy that VM for now, it was just testing the install.

sudo virsh destroy ...name...

Next we need to tweak the template that shakenfist is using to start instances so that it uses the bridge for networking (that template is the one you copied to /srv/shakenfist/template.xml earlier). Replace the interface section in the template with this on both nodes:

<interface type='bridge'>
  <mac address={{eth0_mac}}/>
  <source bridge='br-vxlan0'/>
  <model type='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

I know the bridge mentioned here doesn’t exist yet, but we’ll deal with that in a second. Before we start VMs though, we need a way of getting IP addresses to them. shakenfist can configure interfaces using config drive, but I’d prefer to use DHCP because who doesn’t love some additional complexity?

On one of the nodes install docker:


sudo apt-get install apt-transport-https ca-certificates curl gnupg2 software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

Now we can setup DHCP. Create a place for the configuration file:

sudo mkdir /srv/shakenfist/dhcp

And then create the configuration file at /srv/shakenfist/dhcp/dhcpd.conf with contents like this:

default-lease-time 3600;
max-lease-time 7200;
option domain-name-servers 8.8.8.8;
authoritative;

subnet 192.168.200.0 netmask 255.255.255.0 {
  option routers 192.168.1.1;
  option broadcast-address 192.168.1.255;

  pool {
    range 192.168.200.10 192.168.200.254;
  }
}

Before we can start dhcpd, we need to move the VXLAN device into a bridge so we can add a device for the DHCP server to it. First off remove the vxlan0 device from the last post:

sudo ip link set down dev vxlan0
sudo ip link del vxlan0

And now recreate it with a bridge:

sudo ip link add vxlan0 type vxlan id 42 dev eth0 dstport 0
sudo bridge fdb append to 00:00:00:00:00:00 dst 34.70.161.180 dev vxlan0
sudo ip link add br-vxlan0 type bridge
sudo ip link set vxlan0 master br-vxlan0
sudo ip link set vxlan0 up
sudo ip link set br-vxlan0 up
sudo ip link add dhcp-vxlan0 type veth peer name dhcp-vxlan0p
sudo ip link set dhcp-vxlan0p master br-vxlan0
sudo ip link set dhcp-vxlan0 up
sudo ip link set dhcp-vxlan0p up
sudo ip addr add 192.168.200.1/24 dev dhcp-vxlan0

This block of commands:

recreated the vxlan0 interface
added it to the mesh with the other node again
created a bridge named br-vxlan0
moved the vxlan0 interface into it
created a veth pair called dhcp-vxlan0 and dhcp-vlan0p
moved the peer part of that veth pair into the bridge
and then configured an IP on the external half of the veth pair

To make the bridge survive reboots you would need to add it to either /etc/network/interfaces or /etc/netplan/01-netcfg.yml depending on your distribution, but that’s outside the scope of this post.

You should be able to ping again. From the other node give it a try:

$ ping 192.168.200.1
PING 192.168.200.1 (192.168.200.1) 56(84) bytes of data.
64 bytes from 192.168.200.1: icmp_seq=1 ttl=64 time=19.3 ms
64 bytes from 192.168.200.1: icmp_seq=2 ttl=64 time=0.571 ms

We need to do something similar on the other node so it can run VMs as well. It is a tiny bit simpler because there wont be any DHCP there however, and remembering that you need to change 35.223.115.132 to the IP of your first node:

sudo ip link set down dev vxlan0
sudo ip link del vxlan0

sudo ip link add vxlan0 type vxlan id 42 dev eth0 dstport 0
sudo  bridge fdb append to 00:00:00:00:00:00 dst 35.223.115.132 dev vxlan0
sudo ip link add br-vxlan0 type bridge
sudo ip link set vxlan0 master br-vxlan0
sudo ip link set vxlan0 up
sudo ip link set br-vxlan0 up

Note that now we can’t do a ping test because the second VM no longer consumes an IP for the base OS.

Now we can start the docker container with dhcpd listening on dhcp-vxlan0:

sudo docker run -it --rm --init --net host -v /srv/shakenfist/dhcp:/data networkboot/dhcpd dhcp-vxlan0

This runs dhcpd interactively so we can see what happens. Now try starting a VM on the other node:

sudo python3 daemon.py

You can watch the VM booting using the “virsh console” command with the name of the vm from “virsh list“. The dhcpd process should show you something like this:

sudo docker run -it --rm --init --net host -v /srv/shakenfist/dhcp:/data networkboot/dhcpd dhcp-vxlan0
Internet Systems Consortium DHCP Server 4.3.5
Copyright 2004-2016 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Config file: /data/dhcpd.conf
Database file: /data/dhcpd.leases
PID file: /var/run/dhcpd.pid
Wrote 0 leases to leases file.
Listening on LPF/dhcp-vxlan0/06:ff:bc:7d:11:e3/192.168.200.0/24
Sending on   LPF/dhcp-vxlan0/06:ff:bc:7d:11:e3/192.168.200.0/24
Sending on   Socket/fallback/fallback-net
Server starting service.
DHCPDISCOVER from ee:95:4d:40:ca:a6 via dhcp-vxlan0
DHCPOFFER on 192.168.200.10 to ee:95:4d:40:ca:a6 (foo) via dhcp-vxlan0
DHCPREQUEST for 192.168.200.10 (192.168.200.1) from ee:95:4d:40:ca:a6 (foo) via dhcp-vxlan0
DHCPACK on 192.168.200.10 to ee:95:4d:40:ca:a6 (foo) via dhcp-vxlan0

You can see here that our new VM got the IP 192.168.200.10 from the DHCP server! It is moments like this when you don’t realise that this blog post took me hours to write that I feel really smart.

If we started a VM on the first node (the same command as for the second node), we’d now have two VMs on a virtual network which had working DHCP and could ping each other. I think that’s enough for one evening.

Setting up VXLAN on Google Compute Engine

So my ultimate goal here is to try out VXLAN between some VMs on instances in Google compute engine, but today I’m just going to get VXLAN working because that took a fair bit longer than I expected. First off, boot your instances — because I will need nested virt later I chose two instances on Google Cloud. Please note that you need to do a bit of a dance to turn on nested virt there. I also chose to use Debian for this experiment:

gcloud compute instances create vx-1 --zone us-central1-b --min-cpu-platform "Intel Haswell" --image nested-vm-image

Now do those standard things you do to all new instances:

sudo apt-get update
sudo apt-get dist-upgrade -y

Now let’s setup VXLAN between the two nodes, with a big nod to this web page. First create a VXLAN interface on each machine (if you care about the port your VXLAN traffic is on being to IANA standards, see the postscript at the end of this):

sudo ip link add vxlan0 type vxlan id 42 dev eth0 dstport 0

Now we need to put the two nodes into a mesh, where 34.70.161.180 is the IP of the node we are not running this command on and the IP address for the second command needs to be different on each machine.

sudo bridge fdb append to 00:00:00:00:00:00 dst 34.70.161.180 dev vxlan0
sudo ip addr add 192.168.200.1/24 dev vxlan0
sudo ip link set up dev vxlan0

I am pretty sure that this style of mesh (all nodes connected) wouldn’t scale past non-trivial sizes, but hey baby steps right? Finally, because we’re using Google Cloud we need to add firewall rules to allow our traffic into the instances:

Note that these rules are a source of confusion for me right now. I wanted (and configured) VXLAN. So why do I need to allow OTV for this to work? I suspect Linux has politely ignored my request and used OTV not VXLAN for my traffic.

We should now be able to ping those newly configured IP addresses from each machine:

ping 192.168.200.2 -c 1
PING 192.168.200.2 (192.168.200.2) 56(84) bytes of data.
64 bytes from 192.168.200.2: icmp_seq=1 ttl=64 time=1.76 ms

Which produces traffic like this on the underlay network:

tcpdump -n -i eth0 host 34.70.161.180
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:01:58.159092 IP 10.128.0.9.59341 > 34.70.161.180.8472: OTV, flags [I] (0x08), overlay 0, instance 42
IP 192.168.200.1 > 192.168.200.2: ICMP echo request, id 20119, seq 1, length 64
09:01:58.160786 IP 34.70.161.180.48471 > 10.128.0.9.8472: OTV, flags [I] (0x08), overlay 0, instance 42
IP 192.168.200.2 > 192.168.200.1: ICMP echo reply, id 20119, seq 1, length 64
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel

Hopefully this is helpful to someone else. Thanks again to Joe Julian for a very helpful post.

Postscript: Dale Shaw pointed out on twitter that I might still be talking VXLAN, just on a weird port. This is supported by this comment I found on the internets: “when VXLAN was first implemented in linux, UDP ports were not specified. Many vendors use 8472, and Linux uses the same port. Later, IANA allocated 4789 as the port. If you need to use the IANA port, you need to specify it with dstport”.

On syncing with Google Contacts

So, I started with a new company a few weeks ago, and one of the things I missed from my previous company was having the entire corporate directory synced onto my phone. Its really handy as an on caller to be able to give people a call when something goes wrong, without having to dig around and find their details.

Back in the good old days at Google the way you got this sort of data onto your phone was to run a script written by one of the guys on the gmail team. The script grabbed the LDAP directory, and pushed it into Google contacts, which you could then sync with your phone. Now I wanted something very similar — especially as the contacts sync stuff with Android is pretty reasonable.

However, I’d never coded with the Google public APIs before, and that turned out to be the hardest part of the problem.

First off I wrote a little script which dumped the corporate directory into a text file. I mostly did this because I wanted other people to be able to run the script in as light weight a manner as possible — for example, if we wanted to roll this out for hundreds of people, then you wouldn’t want to run the LDAP query hundreds of times. The format for my text file is kinda lame to be honest:

Michael Still: {'telephoneNumber': ['+61 123 123 123'], 'ID': ['mikalstill'], 'mail': ['mikal@stillhq.com']}

So, you get the user’s name, then a python dictionary with three keys in it. There isn’t any particular reason for having just three keys, it was just the three fields I thought were most interesting at the time. Note that each field is an array. A simple human readable format like this means that I can also grep through the file if I ever quickly want a user’s details, which is a nice side effect.

The most important thing I learnt here is that the ID field is really important. If you don’t have something you feel you can use there, then you might need to synthesize something — perhaps an ascii representation of the user’s name or something. This is important because I discovered that Google rewrites Unicode characters you ask it to store, so if you do a simple text comparison against the user’s name, then you might get a false negative and end up creating more than one entry for that user. That was particularly a problem for me because there are a fair few people in the company with European accented characters in their names.

The docs for the Google contacts API are ok, although I did have to spend some time randomly searching for examples of some of the things I wanted to do. For example, the docs didn’t have an example of how to store a phone number that I could find. Also, I am a little shocked to discover there is no query interface in contacts for contact name. This seems like a pretty massive oversight to me, but here’s what the docs have to say on the issue:

For more information about query parameters, see the Contacts Data API Reference Guide and the Google Data APIs Reference Guide. In particular, there is no support for full-text queries or locating a contact by email address.

Whatever intern wrote the API should have his ball pit rights revoked until he fixes that. After that it was all gravy. Here’s the code: http://www.stillhq.com/svn/trunk/google-contacts/pushdirectory.py.

I note that there is an enterprise shared contacts API (see here), but you have to be a premiere customer for it to work.

Getting Google Talk working with PyXMPP

Jacek Konieczny has written the wholly fantabulous PyXMPP, which implements Jabber clients and servers in Python. Now, Google Talk is a Jabber server, but it needs TLS support before it works. The code is all there, but the echobot example in the download (look in the examples directory) doesn’t show you how. It’s not that hard though — here’s the patch I needed to make it work:

--- echobot.py  2005-12-26 07:25:55.000000000 -0800
+++ echobot2.py 2006-10-25 04:25:02.000000000 -0700
@@ -13,6 +13,7 @@

 from pyxmpp.all import JID,Iq,Presence,Message,StreamError
 from pyxmpp.jabber.client import JabberClient
+from pyxmpp import streamtls

 class Client(JabberClient):
     """Simple bot (client) example. Uses `pyxmpp.jabber.client.JabberClient`
@@ -28,8 +29,12 @@

         # setup client with provided connection information
         # and identity data
+
+        tls = streamtls.TLSSettings(require=True, verify_peer=False)
+        auth = ['sasl:PLAIN']
         JabberClient.__init__(self, jid, password,
-                disco_name="PyXMPP example: echo bot", disco_type="bot")
+                disco_name="PyXMPP example: echo bot", disco_type="bot",
+                tls_settings=tls, auth_methods=auth)

         # register features to be announced via Service Discovery
         self.disco_info.add_feature("jabber:iq:version")

That makes the __init__ method for the client:

def __init__(self, jid, password):

    # if bare JID is provided add a resource -- it is required
    if not jid.resource:
        jid=JID(jid.node, jid.domain, "Echobot")

    # setup client with provided connection information
    # and identity data

    tls = streamtls.TLSSettings(require=True, verify_peer=False)
    auth = ['sasl:PLAIN']
    JabberClient.__init__(self, jid, password,
            disco_name="PyXMPP example: echo bot", disco_type="bot",
            tls_settings=tls, auth_methods=auth)

    # register features to be announced via Service Discovery
    self.disco_info.add_feature("jabber:iq:version")

Now the client works with a gtalk login:

$ ./echobot2.py username@gmail.com supersecretthingie
creating client...
connecting...
*** State changed: resolving srv (u'gmail.com', 'xmpp-client') ***
*** State changed: resolving 'talk.l.google.com.' ***
*** State changed: connecting ('72.14.253.125', 5222) ***
*** State changed: connected ('72.14.253.125', 5222) ***
looping...
*** State changed: tls connecting  ***
*** State changed: tls connected  ***
*** State changed: fully connected  ***
*** State changed: authenticated  ***
*** State changed: binding u'Echobot' ***
*** State changed: authorized  ***
mikalstill@gmail.com/Gaim6734F991 has become available
mikalstill@gmail.com/GaimD2ECF56B has become available(away): I'm not at my
desk at work at the moment. This is probably because I'm at a meeting or
racing electric scooters. If you IM me I will see the message when I get back.
My roster:
mikalstill@gmail.com "" subscription=both groups=
Message from mikalstill@gmail.com/Gaim6734F991 received. Body: "Hello there". Type: "chat".
disconnecting...
exiting...
$

Too easy.

Update: mbot is a Google Talk bot engine built on top of this.

Looking for Women studying computing in Australia

I was in an unrelated meeting at work today, and it came up that the first annual Anti Borg scholarship is closing it’s application window in a few days. I thought it was worth mentioning here, in case there are people who are interested in applying. The basic deal is:

Dr. Anita Borg (1949 – 2003) devoted her adult life to revolutionizing the way we think about technology and dismantling barriers that keep women and minorities from entering computing and technology fields. Her combination of technical expertise and fearless vision continues to inspire and motivate countless women to become active participants and leaders in creating technology.

As part of Google’s ongoing commitment to furthering Anita’s vision, we are pleased to announce the 2006 Google Australia Anita Borg Scholarship. Through the scholarship, we would like to encourage women to excel in computing and technology and become active role models and leaders.

Scholarships will be awarded based on the strength of candidates’ academic background and demonstrated leadership. A group of female undergraduate and postgraduate student finalists will be chosen from the applicant pool. The scholarship recipients, selected from the finalists, will each receive a $5,000 AUD scholarship for the 2007 academic year.

All finalists will be invited to visit Google Sydney in November 2006 for a networking retreat. It will include workshops with a series of speakers, breakout sessions and social activities. The visit is meant to be an opportunity for all finalists to meet and share their experiences.

If that sounds interesting to you and you’re a lady studying computing or a related field, then you should checkout the announcement page before it’s too late.

Mountain View WiFi

I’m sitting in a park in Mountain View at the moment, and the Mountain View Google WiFi works just fine. That’s totally cool. It makes me think about all those random crazy projects I can now implement where a computer in my car needs an internet connection to work right. Hmmmmmm… I guess that’s another good use for a slug.

Holy crap!

These guys really did deliver 1,000 pizzas to Google’s Mountain View campus. We even ate them. I guess that’s one way of getting publicity…

Kinderplex

The kids are the in the Kinderplex today (the Google-only child care center). It’s the first time I’ve dropped Matthew off for care and he hasn’t gone ape — I think it’s because he loved the center so much… There are computers in every room, and it’s probably the nicest laid out and supplied child care center I have ever seen. It makes me wonder if I should be sending Andrew there next year for pre-school.