Google – Made by Mikal

Exporting volumes from Cinder and re-creating COW layers

Post author:mikal
Post published:April 14, 2020
Post category:Google Google Compute Engine OpenStack

Today I wandered into a bit of a rat hole discovering how to export data from OpenStack Cinder volumes when you don't have admin permissions, and I thought it was worth documenting here so I remember it for next time. Let's assume that you have a Cinder volume named "child1", which is a 64gb volume originally cloned from "parent1". parent1 is a 7.9gb VMDK, but the only way I can find to extract child1 is to convert it to a glance image and then download the entire volume as a raw. Something like this: $ cinder upload-to-image $child1 "extract:$child1" Where $child1 is the UUID of the Cinder volume. You then need to find the UUID of the image in Glance, which the Cinder upload-to-image command will have told you, but you can also find by searching Glance for your image named "extract:$child1": $ glance image-list | grep "extract:$cinder_uuid" You now need to watch that Glance image until the status of the image is "active". It will go through a series of steps with names like "queued", and "uploading" first. Now you can download the image from Glance: $ glance image-download --file images/$child1.raw --progress $glance_uuid And then delete the intermediate glance…

Configuring load balancing and location headers on Google Cloud

Post author:mikal
Post published:March 4, 2020
Post category:Google Google Compute Engine

I have a need at the moment to know where my users are in the world. This helps me to identify what compute resources to serve their request with in order to reduce the latency they experience. So how do you do that thing with Google Cloud? The first step is to setup a series of test backends to send traffic to. I built three regions: Sydney; London; and Los Angeles. It turns out in hindsight that wasn't actually nessesary though -- this would work with a single backend just as well. For my backends I chose a minimal Ubuntu install, running this simple backend HTTP service. I had some initial trouble finding a single page which walked through the setup of the Google Cloud load balancer to do what I wanted, which is the main reason for writing this post. The steps are: Create your test instances and configure the backend on them. I ended up with a setup like this: Next setup instance groups to contain these instances. I chose unmanaged instance groups (that is, I don't want autoscaling). You need to create one per region. But wait! There's one more layer of abstraction. We need a backend…

Setting up VXLAN between nested virt VMs on Google Compute Engine

Post author:mikal
Post published:January 6, 2020
Post category:Google Google Compute Engine

I wanted to play with a VXLAN mesh between VMs on more than one hypervisor node, but the setup for VXLAN ended up being a separate post because it was a bit long. Read that post first if you want to follow the instructions here. Now that we have a working VXLAN mesh between our two nodes we can move on to installing libvirt (which is called libvirt-daemon-system on Debian, not libvirt-bin as on Ubuntu): sudo apt-get install -y qemu-kvm libvirt-daemon-system sudo virsh net-start default sudo virsh net-autostart --network default I'm going to use a little python helper to launch my VMs, so I need some other dependancies as well: sudo apt-get install -y python3-pip pkg-config libvirt-dev git git clone https://github.com/mikalstill/shakenfist cd shakenfist git checkout 6bfac153d249752b27d224ad9d079095b640498e sudo mkdir /srv/shakenfist sudo cp template.debian.xml /srv/shakenfist/template.xml sudo pip3 install -r requirements.txt Let's launch a quick test VM to make sure the helper works: sudo python3 daemon.py sudo virsh list You can destroy that VM for now, it was just testing the install. sudo virsh destroy ...name... Next we need to tweak the template that shakenfist is using to start instances so that it uses the bridge for networking (that template is the one…

Setting up VXLAN on Google Compute Engine

Post author:mikal
Post published:January 5, 2020
Post category:Google Google Compute Engine

So my ultimate goal here is to try out VXLAN between some VMs on instances in Google compute engine, but today I'm just going to get VXLAN working because that took a fair bit longer than I expected. First off, boot your instances -- because I will need nested virt later I chose two instances on Google Cloud. Please note that you need to do a bit of a dance to turn on nested virt there. I also chose to use Debian for this experiment: gcloud compute instances create vx-1 --zone us-central1-b --min-cpu-platform "Intel Haswell" --image nested-vm-image Now do those standard things you do to all new instances: sudo apt-get update sudo apt-get dist-upgrade -y Now let's setup VXLAN between the two nodes, with a big nod to this web page. First create a VXLAN interface on each machine (if you care about the port your VXLAN traffic is on being to IANA standards, see the postscript at the end of this): sudo ip link add vxlan0 type vxlan id 42 dev eth0 dstport 0 Now we need to put the two nodes into a mesh, where 34.70.161.180 is the IP of the node we are not running this command on…

On syncing with Google Contacts

Post author:mikal
Post published:December 16, 2011
Post category:Google

So, I started with a new company a few weeks ago, and one of the things I missed from my previous company was having the entire corporate directory synced onto my phone. Its really handy as an on caller to be able to give people a call when something goes wrong, without having to dig around and find their details. Back in the good old days at Google the way you got this sort of data onto your phone was to run a script written by one of the guys on the gmail team. The script grabbed the LDAP directory, and pushed it into Google contacts, which you could then sync with your phone. Now I wanted something very similar -- especially as the contacts sync stuff with Android is pretty reasonable. However, I'd never coded with the Google public APIs before, and that turned out to be the hardest part of the problem. First off I wrote a little script which dumped the corporate directory into a text file. I mostly did this because I wanted other people to be able to run the script in as light weight a manner as possible -- for example, if we wanted…

Getting Google Talk working with PyXMPP

Post author:mikal
Post published:November 20, 2006
Post category:Google

Jacek Konieczny has written the wholly fantabulous PyXMPP, which implements Jabber clients and servers in Python. Now, Google Talk is a Jabber server, but it needs TLS support before it works. The code is all there, but the echobot example in the download (look in the examples directory) doesn't show you how. It's not that hard though -- here's the patch I needed to make it work: --- echobot.py 2005-12-26 07:25:55.000000000 -0800 +++ echobot2.py 2006-10-25 04:25:02.000000000 -0700 @@ -13,6 +13,7 @@ from pyxmpp.all import JID,Iq,Presence,Message,StreamError from pyxmpp.jabber.client import JabberClient +from pyxmpp import streamtls class Client(JabberClient): """Simple bot (client) example. Uses `pyxmpp.jabber.client.JabberClient` @@ -28,8 +29,12 @@ # setup client with provided connection information # and identity data + + tls = streamtls.TLSSettings(require=True, verify_peer=False) + auth = ['sasl:PLAIN'] JabberClient.__init__(self, jid, password, - disco_name="PyXMPP example: echo bot", disco_type="bot") + disco_name="PyXMPP example: echo bot", disco_type="bot", + tls_settings=tls, auth_methods=auth) # register features to be announced via Service Discovery self.disco_info.add_feature("jabber:iq:version") That makes the __init__ method for the client: def __init__(self, jid, password): # if bare JID is provided add a resource -- it is required if not jid.resource: jid=JID(jid.node, jid.domain, "Echobot") # setup client with provided connection information # and identity data tls = streamtls.TLSSettings(require=True, verify_peer=False)…

Looking for Women studying computing in Australia

Post author:mikal
Post published:September 7, 2006
Post category:Google

I was in an unrelated meeting at work today, and it came up that the first annual Anti Borg scholarship is closing it's application window in a few days. I thought it was worth mentioning here, in case there are people who are interested in applying. The basic deal is: Dr. Anita Borg (1949 - 2003) devoted her adult life to revolutionizing the way we think about technology and dismantling barriers that keep women and minorities from entering computing and technology fields. Her combination of technical expertise and fearless vision continues to inspire and motivate countless women to become active participants and leaders in creating technology. As part of Google's ongoing commitment to furthering Anita's vision, we are pleased to announce the 2006 Google Australia Anita Borg Scholarship. Through the scholarship, we would like to encourage women to excel in computing and technology and become active role models and leaders. Scholarships will be awarded based on the strength of candidates' academic background and demonstrated leadership. A group of female undergraduate and postgraduate student finalists will be chosen from the applicant pool. The scholarship recipients, selected from the finalists, will each receive a $5,000 AUD scholarship for the 2007 academic year.…

Mountain View WiFi

Post author:mikal
Post published:August 21, 2006
Post category:Google

I'm sitting in a park in Mountain View at the moment, and the Mountain View Google WiFi works just fine. That's totally cool. It makes me think about all those random crazy projects I can now implement where a computer in my car needs an internet connection to work right. Hmmmmmm... I guess that's another good use for a slug.

Holy crap!

Post author:mikal
Post published:June 28, 2006
Post category:Google

These guys really did deliver 1,000 pizzas to Google's Mountain View campus. We even ate them. I guess that's one way of getting publicity...

Kinderplex

Post author:mikal
Post published:February 16, 2006
Post category:Google

The kids are the in the Kinderplex today (the Google-only child care center). It's the first time I've dropped Matthew off for care and he hasn't gone ape -- I think it's because he loved the center so much... There are computers in every room, and it's probably the nicest laid out and supplied child care center I have ever seen. It makes me wonder if I should be sending Andrew there next year for pre-school.