Made by Mikal – Page 10

Linux bridges have their MTU overwritten when you add an interface

Post author:mikal
Post published:September 7, 2021
Post category:Linux

I discovered last night that network bridges on linux have their Maximum Transmission Unit (MTU) overwritten by whatever is the MTU value of the most recent interface added to the bridge. This is bad. Very bad. Specifically this is bad because MTU matters for accurately describing the capabilities of the network path the packets will travel on, so it shouldn't be clobbered willy nilly. Here's an example of the behaviour: # ip link add egr-br-ens1f0 mtu 1500 type bridge # ip link show dev egr-br-ens1f0 3: egr-br-ens1f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 7e:33:1b:30:d8:00 brd ff:ff:ff:ff:ff:ff # ip link add egr-eaa64a-o mtu 8950 type veth peer name egr-eaa64a-i # ip link show dev egr-br-ens1f0 3: egr-br-ens1f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 7e:33:1b:30:d8:00 brd ff:ff:ff:ff:ff:ff # brctl addif egr-br-ens1f0 egr-eaa64a-o # ip link show dev egr-br-ens1f0 3: egr-br-ens1f0: <BROADCAST,MULTICAST> mtu 8950 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether da:82:cf:34:13:60 brd ff:ff:ff:ff:ff:ff So you can see here that the bridge had an MTU of 1,500 bytes. We create a veth pair with an MTU of 8,950 bytes and add it to…

Manipulating Docker images without Docker installed

Post author:mikal
Post published:March 27, 2021
Post category:Containers Docker Occy Strap

Recently I've been playing a bit more with Docker images and Docker image repositories. I had in the past written a quick hack to let me extract files from a Docker image, but I wanted to do something a little more mature than that. For example, sometimes you want to download an image from a Docker image repository without using Docker. Naively if you had Docker, you'd do something like this: docker pull busybox docker save busybox However, that assumes that you have Docker installed on the machine downloading the images, and that's sometimes not possible for security reasons. The most obvious example I can think of is airgapped secure environments where you need to walk the data between two networks, and the unclassified network machine doesn't allow administrator access to install Docker. So I wrote a little tool to do image manipulation for me. The tool is called Occy Strap, is written in python, and is available on pypi. That means installing it is relatively simple: python3 -m venv ~/virtualenvs/occystrap . ~/virtualenvs/occystrap/bin/activate pip install occystrap Which doesn't require administrator permissions. There are then a few things we can do with Occy Strap. Downloading an image from a repository and…

Complexity Arrangements for Sustained Innovation: Lessons From 3M Corporation

Post author:mikal
Post published:March 26, 2021
Post category:Management

This is the second business paper I've read this week while reading along with my son's university studies. The first is discussed here if you're interested. This paper is better written, but more academic in its style. This ironically makes it harder to read, because its grammar style is more complicated and harder to parse. The take aways for me from this paper is that 3M is good at encouraging serendipity and opportune moments that create innovation. This is similar to Google's attempts to build internal peer networks and deliberate lack of structure. In 3M's case its partially expressed as 15% time, which is similar to Google's 20% time. Specifically, "eureka moments" cannot be planned or scheduled, but require prior engagement. chance favors only the prepared mind -- Pasteur 3M has a variety of methods for encouraging peer networks, including technology fairs, "bootlegging" (borrowing idle resources from other teams), innovation grants, and so on. At the same time, 3M tries to keep at least a partial focus on events driving by schedules. The concept of time is important here -- there is a "time to wait" (we are ahead of the market); "a time in between" (15% time); and "a…

A corporate system for continuous innovation: The case of Google Inc

Post author:mikal
Post published:March 26, 2021
Post category:Management

So, one of my kids is studying some business units at university and was assigned this paper to read. I thought it looked interesting, so I gave it a read as well. While not being particularly well written in terms of style, this is an approachable introduction to the culture and values of Google and how they play into Google’s continued ability to innovate. The paper identifies seven important attributes of the company's culture that promote innovation, as ranked by the interviewed employees: The culture is innovation oriented. They put a lot of effort into selecting individuals who will fit well with the culture at hiring time. Leaders are seen as performing a facilitiation role, not a directive one. The organizational structure is loosely defined. OKRs and aligned performance incentives. A culture of organizational learning through postmortems and building internal social networks. Learning is considered a peer to peer activity that is not heavily structured. External interaction -- especially in the form of aggressive acquisition of skills and technologies in areas Google feels they are struggling in. Additionally, they identify eight habits of a good leader: A good coach. Empoyer your team and don't micro-manage. Express interest in employees' success…

Shaken Fist v0.4.2

Post author:mikal
Post published:February 7, 2021
Post category:Shaken Fist

Shaken Fist v0.4.2 snuck out yesterday as part of shooting this tutorial video. That's because I really wanted to demonstrate floating IPs, which I only recently got working nicely. Overall in v0.4.2 we: Improved CI for image API calls. Improved upgrade CI testing. Improved network state tracking. Floating IPs now work, and have covering CI. shakenfist#257 Resolve leaks of floating IPs from both direct use and NAT gateways. shakenfist#256 Resolve leaks of IPManagers on network delete. shakenfist#675 Use system packages for ansible during install.

Starting your first instance on Shaken Fist (a video tutorial)

Post author:mikal
Post published:February 7, 2021
Post category:Shaken Fist

As a bit of an experiment, I've made this quick and dirty "vlog" style tutorial video to show you how to install Shaken Fist on a single machine and boot your first instance. I demonstrate how to install, setup your first virtual network, start the instance, inspect events that the instance has experienced, and then log in. Let me know if you think its useful.

Books read in January 2021

Post author:mikal
Post published:January 31, 2021
Post category:Book

Its been 10 years since I've read enough to write one of these summary posts... Which I guess means something. This month I've been thinking a lot about systems design and how to avoid Second Systems effect while growing a product, which guided my reading choices a fair bit. A fair bit of that reading has been in the form of blog posts and twitter threads, so I am going to start including those in these listings of things I've read. Social media posts of note: A better Kubernetes from the ground up Systems Design Explains the World, Volume 1 This twitter thread about why Google Cloud isn't winning in the cloud platform wars Stevey's Google Platforms Rant OKRs -- Secrets to Success Goals Gone Wild Books: The Mythical Man-Month Bolos 1: Honor of the Regiment (re-read, in the hope that learning about artificially intelligent super tanks killing aliens would help me with motivational techniques for my team). Bolos 2: The Unconquerable (another re-read).

Shaken Fist 0.4.1

Post author:mikal
Post published:January 26, 2021
Post category:Shaken Fist

I don't blog about every Shaken Fist release here, but I do feel like the 0.4 release (and the subsequent minor bug fix release 0.4.1) are a pretty big deal in the life of the project. The focus of the v0.4 series is reliability -- we've used behaviour in the continuous integration pipeline as a proxy for that, but it should be a significant improvement in the real world as well. This has included: much more extensive continuous integration coverage, including several new jobs. checksumming image downloads, and retrying images where the checksum fails. reworked locking. etcd reliability improvements. refactoring instances and networks to a new "non-volatile" object model where only immutable values are cached. images now track a state much like instances and networks. a reworked state model for instances, where its clearer why an instance ended up in an error state. This is documented in our developer docs. In terms of new features, we also added: a network ping API, which will emit ICMP ping packets on the network node onto your virtual network. We use this in testing to ensure instances booted and ended up online. networks are now checked to ensure that they have a reasonable minimum…

Goals Gone Wild

Post author:mikal
Post published:January 25, 2021
Post category:Management

In 2009 Harvard Business School published a draft paper entitled "Goals Gone Wild", and its abstract is quite concerning. For example: "We identify specific side effects associated with goal setting, including a narrow focus that neglects non-goal areas, a rise in unethical behavior, distorted risk preferences, corrosion of organizational culture, and reduced intrinsic motivation." Are we doomed? Is all goal setting harmful? Interestingly, I came across this paper while reading Measure What Matters, which argues the exact opposite point -- that OKRs provide a meaningful way to improve the productivity of an organization. The paper starts by listing a series of examples of goal setting gone wrong: Sears' auto repair in the early 1900s over charging customers to meet hourly billable goals; Enron's sales targets based solely on volume and revenue and not profit; and Ford Motor Company's goal of shipping a car at a specific target price point which resulted in significant safety failures. The paper then provides specific examples of how goals can go wrong: By being too specific and causing other important features of a task to be ignored -- for example shipping on a specific deadline but ignoring testing adequately to achieve that deadline. By being…

A super simple sourdough loaf

Post author:mikal
Post published:January 15, 2021
Post category:Cooking

This is the fourth in a series of posts documenting my adventures in making bread during the COVID-19 shutdown. This post has been a while coming, but my sister in law was interested in the sourdough loaf last night, so I figured I should finally document my process. First off you need to have a sourdough starter, which I wrote up in a previous post. I am sure less cheaty ways will work too, but the cheating was where it was at for me. Then, you basically follow the process I use for my super simple non-breadmaker loaf, but tweaked a little to use the starter. For the loaf itself: 2 cups of bakers flour (not plain white flour) 1 tea spoon of salt 2 cups of the sourdough starter 1 cup water Similarly to the super simple loaf, you want the dough to be a bit tacky when mixed -- it gets runnier as the yeast does its thing, so it will be too runny if it doesn't start out tacky. I then just leave it on the kitchen bench under a cover for the day. In the evening its baked like the super simple loaf -- heat a…