Training isn’t a work perk, its essential operational risk mitigation

  • Post author:
  • Post category:Work

I’ve been thinking a bit about training at work recently, largely in the context of having spent the last twenty years working for US technology companies. I think effectively all of these companies made a pretty big mistake — they viewed training of employees as a perk much like vacation, book budgets, or t-shirts. They advertise their training programs as part of their recruitment process, and just like other perks they’re cut when times get a bit grim. However, that’s not actually why employers should train their people. We train people so that they have the skills they need to do their jobs — especially when things get real and aren’t working out to plan.

There are definitely industries who have good examples of this sort of risk reduction training done well — airlines and the military both engage in regular training activities that ensure that when things get exciting the people know what to do. This includes leaders being trained on how to make decisions that are likely to result in the desired outcomes.

(more…)

Continue ReadingTraining isn’t a work perk, its essential operational risk mitigation

I really wanted to like etcd, but Andy Pavlo was right

Andy Pavlo of the CMU Database Group is well known for saying that while NoSQL databases acquire cyclical popularity, all databases eventually iterate back to a SQL interface — it happened with MongoDB and Google’s BigTable for example.

I think I have hit that point with etcd. Initially I ported from MySQL to etcd because I really wanted the inexpensive distributed locking and being able to watch values. However, I never actually watch values in my code any more, and I now spend a huge amount of my time maintaining what my code calls “caches”, but which I can now see are just poorly implemented secondary indexes. The straw that broke the camel’s back was https://github.com/etcd-io/etcd/issues/9043, which changed etcd’s defaults to only being able to return 1.5mb in a RPC request.

I therefore think it might be time for me to port back to a real SQL database, perhaps keeping etcd to manage distributed locks. Perhaps.

(more…)

Continue ReadingI really wanted to like etcd, but Andy Pavlo was right

Databases built on object stores are officially interesting

  • Post author:
  • Post category:Databases

…even if none of my friends seem to think so.

I’ve been off on a bit of a tangent recently. Its a slow burn tangent, that I am pretty sure was kicked off by this Geek Narrator podcast episode about the design of Turbo Puffer with Simon Eskildsen:

The basic idea is that you can build very large scale database systems using only the primitives provided by an object store such as Amazon S3. Now, the performance might also suck, but you can alleviate some of that with a good caching layer and in return you get massive scale. This first video caused me to discover the work of Andy Pavlo, who was interviewed by the same podcast:

(more…)

Continue ReadingDatabases built on object stores are officially interesting

The simplest boot target for the Kerbside SPICE VDI proxy CI

For the last couple of years I have been working on a SPICE protocol native proxy called Kerbside. The basic idea is to be able to provide SPICE Virtual Desktop Interface (VDI) consoles to users from cloud platforms such as Shaken Fist, OpenStack, or oVirt. Think Citrix, but for Open Source cloud platforms. SPICE is attractive here because it has some features that other more common VDI protocols like VNC don’t have — good cut and paste support, USB device pass-through, multiple monitor support, and so on. RDP has these, but RDP was not a supported VDI protocol when using qemu on Linux with KVM until incredibly recently — literally the last couple of months.

(In terms of clouds that Kerbside supports, I think it would be relatively trivial to also support Proxmox, KubeVirt, or a list of static manually created virtual machines, but there’s only so many things one Mikal can do at once…)

Some of these cloud platforms have supported SPICE consoles for a while, but generally with warts. OpenStack for example only exposes them as HTML5 transcoded sessions with reduced functionality. oVirt exposes them via a “proxy” which is just squid (or equivalent), but its fairly dumb — it exposes the underlying hypervisor details to the client for example. I thought I could do better than that.

(more…)

Continue ReadingThe simplest boot target for the Kerbside SPICE VDI proxy CI

The Idea Factory

  • Post author:
  • Post category:Book

This book is a history of the Bell Labs run by AT&T for much of the 20th century. These are the labs which produced many of the things I use day to day -- Unix and the C programming language for example, although this book focuses on other people present at the lab, and a bit earlier than the Unix people. Unix, a history and a memoir for example is set in the same location but later in time. One interesting point the book makes early is that the America of the early 20th century wasn't super into scientists, it was much more about engineers. So for example Edison was an engineer whose super power was systematically grinding through a problem space looking for solutions to a problem, but not necessarily actually understanding the mechanism that caused the solution to work. A really good example, although not one of Edison's, is adding lead to fuel to stop engine knocking and wear -- they literally walked the periodic table until they found an element that worked. I am left wondering how much of this failure to understand the underlying mechanism was a contributor to the longer term environmental and health implications…

Continue ReadingThe Idea Factory

virtio-vsock: python examples of running the server in the guest

I've been using virtio-serial for communications between Linux hypervisors and guest virtual machines for ages. Lots of other people do it to -- the qemu guest agent for example is implemented like this. In fact, I think that's where I got my original thoughts on the matter from. However, virtio-serial is actually fairly terrible to write against as a programming model, because you're left to do all the multiplexing of various requests down the channel and surely there's something better? Well... There is! virtio-vsock is basically the same concept, except it uses the socket interface. You can have more than one connection open and the sockets layer handles multiplexing by magic. This massively simplifies the programming model for supporting concurrent users down the channel. So that's actually pretty cool. I should credit Kata Containers with noticing this quality of life improvement nearly a decade before I did, but I get there in the end. The virtio-vsock model is only a little bit weird. The "address" for the guest virtual machine is a "CID" (Context ID). The hypervisor process is always at CID 0, CID 1 is reserved and unused, and CID 2 is any process on the host which is not…

Continue Readingvirtio-vsock: python examples of running the server in the guest

Fugitive Telemetry

  • Post author:
  • Post category:Book

This is the fifth murderbot book and it's a fun read just like the rest. Unfortunately, it's also really short just like most of the others and I find that the story is therefore just a bit simple and two dimensional. It is nice that the story isn't just a repeat of previous entries in the series, although I would say that this one is relatively free standing in that it doesn't progress the overall story arc. That said, no regrets reading this one.

Continue ReadingFugitive Telemetry

Children of Memory

  • Post author:
  • Post category:Book

This is the third book in this series, coming after Children of Time and Children of Ruin. While I really liked the first of the books in the series, the second felt weaker. While this one doesn't review as well as the second I think it's actually a stronger book. Whilst sometimes a bit repetitive I think the ideas presented here are novel, and the book does a good job of finding a new way of discussing the tensions that refugees and mass immigration create for societies. This book is also an interesting combination of science fiction and fantasy -- the familiar territory of a failing colonization ship sent out on a hope and a prayer, and then a fantasy story about a little girl trying to save her family and a group of strangers come to town. Overall, I enjoyed this book.

Continue ReadingChildren of Memory

Taming Silicon Valley

  • Post author:
  • Post category:Book

The similarities and contrast between this book and AI Snake Oil are striking. For example, AI Snake Oil describes generative AI as something which largely works but is sometimes wrong, whereas this book is very concerned about how they've been rushed out the door in the wake of the unexpected popularity of ChatGPT despite clear issues with hallucinations and unacceptable content generation. Yet the books agree on many things too -- the widespread use of creators' content without permission, weaponization of generative AI political misinformation, the dangers of deep fakes, and the lack of any form of factual verification (or understanding of the world at all) in the statistical approaches used to generate the content. Big tech has no answer for these "negative externalities" that they are enabling and would really rather we all pretend they're not a thing. This book pushes much harder on the issue of how unregulated big tech is, and how it is repeatedly allowed to cause harm to society in returns for profits. It will be interesting to see if any regulation with teeth is created in this space. I find the assertion made in this book that large language models should not be open…

Continue ReadingTaming Silicon Valley

Network Effect

  • Post author:
  • Post category:Book

I'm not really sure why, but I found it harder to get going on this book than the others in the series. It might have been that I was also reading a particularly good non-fiction book at the same time, or it might have been that the premise for these books is starting to wear a bit thin. I'm unsure. That said, while the start of the book covers familiar territory, the overall story rapid diverges into new things and I found it quite readable once I build up some momentum. In the end, I enjoyed this book and would definitely read it again sometime.

Continue ReadingNetwork Effect

End of content

No more pages to load