shakenfist

I really wanted to like etcd, but Andy Pavlo was right

Andy Pavlo of the CMU Database Group is well known for saying that while NoSQL databases acquire cyclical popularity, all databases eventually iterate back to a SQL interface — it happened with MongoDB and Google’s BigTable for example.

I think I have hit that point with etcd. Initially I ported from MySQL to etcd because I really wanted the inexpensive distributed locking and being able to watch values. However, I never actually watch values in my code any more, and I now spend a huge amount of my time maintaining what my code calls “caches”, but which I can now see are just poorly implemented secondary indexes. The straw that broke the camel’s back was https://github.com/etcd-io/etcd/issues/9043, which changed etcd’s defaults to only being able to return 1.5mb in a RPC request.

I therefore think it might be time for me to port back to a real SQL database, perhaps keeping etcd to manage distributed locks. Perhaps.

(more…)

Continue ReadingI really wanted to like etcd, but Andy Pavlo was right

The simplest boot target for the Kerbside SPICE VDI proxy CI

For the last couple of years I have been working on a SPICE protocol native proxy called Kerbside. The basic idea is to be able to provide SPICE Virtual Desktop Interface (VDI) consoles to users from cloud platforms such as Shaken Fist, OpenStack, or oVirt. Think Citrix, but for Open Source cloud platforms. SPICE is attractive here because it has some features that other more common VDI protocols like VNC don’t have — good cut and paste support, USB device pass-through, multiple monitor support, and so on. RDP has these, but RDP was not a supported VDI protocol when using qemu on Linux with KVM until incredibly recently — literally the last couple of months.

(In terms of clouds that Kerbside supports, I think it would be relatively trivial to also support Proxmox, KubeVirt, or a list of static manually created virtual machines, but there’s only so many things one Mikal can do at once…)

Some of these cloud platforms have supported SPICE consoles for a while, but generally with warts. OpenStack for example only exposes them as HTML5 transcoded sessions with reduced functionality. oVirt exposes them via a “proxy” which is just squid (or equivalent), but its fairly dumb — it exposes the underlying hypervisor details to the client for example. I thought I could do better than that.

(more…)

Continue ReadingThe simplest boot target for the Kerbside SPICE VDI proxy CI

On GitHub merge queues, reliability, and multiplexed serial connections

Assuming anyone was paying attention, which I suspect they are not, they would have noticed that there are a lot of queued up pull requests for Shaken Fist right now. There are a couple of factors leading to that — there are now several bots which do automated pull requests for code hygiene purposes; and a couple of months ago I decided to give GitHub’s new merge queue functionality a go in order to keep the CI resource requirements for Shaken Fist under control. All CI runs on four machines in my home office, and there were periods of time where the testing backlog would be more than 24 hours long. I can’t simply buy more hardware, and I didn’t really want to test things less.

The basic idea of GitHub merge queues is that you have a quick set of initial tests which determine if the pull request smells, and then only run the full test suite on non-stinky code which a human has signed off on. Once the human signs off, the code will only merge if the full suite passes, and GitHub manages a queue of merge attempts to keep that reasonable.

(more…)

Continue ReadingOn GitHub merge queues, reliability, and multiplexed serial connections

FastCDC, puzzlefs, and de-duplicating container and VM images

Since about 2017, a group at Cisco has been working on an “OCI native operating system” under the title “project machine”, which is a terrible project name. I note that most of the people publicly involved in the project according to github commits no longer work at Cisco, so I cannot vouch for the health of the overall project. That said, they did come up with some interesting ideas along the way and given its a quiet time of year I figured I could do some reading.

(more…)

Continue ReadingFastCDC, puzzlefs, and de-duplicating container and VM images

Exploring more efficient remote large file storage

My primary personal project is a thing called Shaken Fist these days — it is an infrastructure as a service cloud akin to OpenStack Compute, but smaller and simpler. Shaken Fist doesn’t have an equivalent to the OpenStack Image service, instead letting your describe your instance images by a standard URL. One of the things Shaken Fist does to be easier to use is it maintains an official repository of common images, which allows users to refer to those images with a shorthand syntax instead of a complete URL. The images also contain small customizations (mainly including the Shaken Fist in-guest agent), which means I can’t just use the official upstream cloud images like OpenStack does.

The images were stored at DreamHost until this week, when a robot decided that they looked like offline backups, despite being served to the Internet via HTTP and being used regularly (although admittedly not frequently). DreamHost unilaterally decided to delete the web site, so now I am looking for new image hosting services, and thinking about better ways to build an image store.

(Oh, and recommending to anyone who asks that they consider using someone less capricious than DreamHost for their hosting needs).

(more…)

Continue ReadingExploring more efficient remote large file storage

Ansible 7.0 onwards requires blocking IO from stdin, stdout, and stderr

  • Post author:
  • Post category:Ansible

Shaken Fist CI started failing this afternoon with this message logged: ERROR: Ansible requires blocking IO on stdin/stdout/stderr. Non-blocking file handles detected: <stdout> Specifically this was happening when using ansible-galaxy to install some requirements, but the check is a more generic check than that was implemented by this ansible pull request, which appears to have been released with ansible-core 2.14 on November 8. That sat around until today, when ansible 7.0.0 was released and broke CI for me. To be completely honest I'm not sure what's happening here -- somewhere in GitHub actions calling a shell script that calls ansible-galaxy the stdout file descriptor gets set to non-blocking and everything breaks. I'm unsure exactly where because its a pain to track down. That said, Jack came to the rescue with this gem: ansible-galaxy install andrewrothstein.etcd-cluster | cat - Which unblocks me. It will be interesting to see if other people encounter problems with this change.

Continue ReadingAnsible 7.0 onwards requires blocking IO from stdin, stdout, and stderr

All python packages require a pyproject.toml with modern pip

So last night Shaken Fist CI jobs started failing with errors like this (editted lightly for clarity): Building wheels for collected packages: shakenfist-ci Building wheel for shakenfist-ci (setup.py): started Building wheel for shakenfist-ci (setup.py): finished with status 'error' error: subprocess-exited-with-error × python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [86 lines of output] ... ...setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. setuptools.SetuptoolsDeprecationWarning, installing to build/bdist.linux-x86_64/wheel running install ... warning: install_lib: byte-compiling is disabled, skipping. running install_egg_info Copying shakenfist_ci.egg-info to build/bdist.linux-x86_64/wheel/shakenfist_ci-0.0.1.dev2544-py3.7.egg-info running install_scripts error: invalid command 'bdist_wininst' [end of output] This was pretty concerning. I know that a setup.py / setup.cfg style install is a little old school, but it was unexpected that it broke entirely. At first I thought I'd have to convert to poetry to unblock this, but Chet helpfully pointed out that this is as simple as adding a pyproject.toml file to the directory which contains your setup.py and setup.cfg. The basic issue is that a modern pip doesn't assume that you're going to use setuptools, so you need to tell it that you're doing that in pyproject.toml. Then you're unblocked. So, just create a file named…

Continue ReadingAll python packages require a pyproject.toml with modern pip

Debian 10 buster bcrypt pip install breakage

So, as of today by Shaken Fist CI jobs for Debian 10 are failing to install bcrypt, with an error that looks like this: Running setup.py install for bcrypt: started Running setup.py install for bcrypt: finished with status 'error' [ ... snip ... ] running build_rust =============================DEBUG ASSISTANCE============================= If you are seeing a compilation error please try the following steps to successfully install bcrypt: 1) Upgrade to the latest pip and try again. This will fix errors for most users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip 2) Ensure you have a recent Rust toolchain installed. bcrypt requires rustc >= 1.56.0. Python: 3.7.3 platform: Linux-4.19.0-21-amd64-x86_64-with-debian-10.12 pip: 18.1 setuptools: 65.2.0 setuptools_rust: 1.5.1 rustc: n/a =============================DEBUG ASSISTANCE============================= I'm not really interested in debating why installing a python package requires a rust compiler, that has been dicussed elsewhere. This specific breakage has been caused by bcrypt releasing 4.0.0, which has this in the changelog: "bcrypt is now implemented in Rust. Users building from source will need to have a Rust compiler available. Nothing will change for users downloading wheels." Unfortunately, you can't just install rustc with apt, as it is both quite big (350mb), and too old (version 1.41.1 versus the required 1.56.0 or better). I also couldn't…

Continue ReadingDebian 10 buster bcrypt pip install breakage

Shaken Fist v0.4.2

Shaken Fist v0.4.2 snuck out yesterday as part of shooting this tutorial video. That's because I really wanted to demonstrate floating IPs, which I only recently got working nicely. Overall in v0.4.2 we: Improved CI for image API calls. Improved upgrade CI testing. Improved network state tracking. Floating IPs now work, and have covering CI. shakenfist#257 Resolve leaks of floating IPs from both direct use and NAT gateways. shakenfist#256 Resolve leaks of IPManagers on network delete. shakenfist#675 Use system packages for ansible during install.

Continue ReadingShaken Fist v0.4.2

Starting your first instance on Shaken Fist (a video tutorial)

As a bit of an experiment, I've made this quick and dirty "vlog" style tutorial video to show you how to install Shaken Fist on a single machine and boot your first instance. I demonstrate how to install, setup your first virtual network, start the instance, inspect events that the instance has experienced, and then log in. Let me know if you think its useful.

Continue ReadingStarting your first instance on Shaken Fist (a video tutorial)

End of content

No more pages to load

Older Posts