Recent Posts

Exploring more efficient remote large file storage

My primary personal project is a thing called Shaken Fist these days — it is an infrastructure as a service cloud akin to OpenStack Compute, but smaller and simpler. Shaken Fist doesn’t have an equivalent to the OpenStack Image service, instead letting your describe your instance images by a standard URL. One of the things Shaken Fist does to be easier to use is it maintains an official repository of common images, which allows users to refer to those images with a shorthand syntax instead of a complete URL. The images also contain small customizations (mainly including the Shaken Fist in-guest agent), which means I can’t just use the official upstream cloud images like OpenStack does.

The images were stored at DreamHost until this week, when a robot decided that they looked like offline backups, despite being served to the Internet via HTTP and being used regularly (although admittedly not frequently). DreamHost unilaterally decided to delete the web site, so now I am looking for new image hosting services, and thinking about better ways to build an image store.

(Oh, and recommending to anyone who asks that they consider using someone less capricious than DreamHost for their hosting needs).

(more…)

Continue ReadingExploring more efficient remote large file storage

This is going to hurt

  • Post author:
  • Post category:Book

This book is lots of things: honest, funny, and ultimately heart breaking. I don't remember how I came across it, but its a good read for when travelling as the diary format means you can put it down whenever you need to do something else. I'm left wondering how the Australian medical system compares to the NHS -- I know we have more patient choice and flexibility -- but I wonder what its like for those working within the system. Either way I definitely recommend this book.

Continue ReadingThis is going to hurt

Ansible 7.0 onwards requires blocking IO from stdin, stdout, and stderr

  • Post author:
  • Post category:Ansible

Shaken Fist CI started failing this afternoon with this message logged: ERROR: Ansible requires blocking IO on stdin/stdout/stderr. Non-blocking file handles detected: <stdout> Specifically this was happening when using ansible-galaxy to install some requirements, but the check is a more generic check than that was implemented by this ansible pull request, which appears to have been released with ansible-core 2.14 on November 8. That sat around until today, when ansible 7.0.0 was released and broke CI for me. To be completely honest I'm not sure what's happening here -- somewhere in GitHub actions calling a shell script that calls ansible-galaxy the stdout file descriptor gets set to non-blocking and everything breaks. I'm unsure exactly where because its a pain to track down. That said, Jack came to the rescue with this gem: ansible-galaxy install andrewrothstein.etcd-cluster | cat - Which unblocks me. It will be interesting to see if other people encounter problems with this change.

Continue ReadingAnsible 7.0 onwards requires blocking IO from stdin, stdout, and stderr

Unix: a history and a memoir

  • Post author:
  • Post category:Book

It was a bit surprising to me that Brian Kernighan self-published a book about Unix history with Kindle Direct publishing, but given how many other books he's published he must have his reasons for not using traditional channels for this one. The book is an engaging read, with quotes which still seem timely today popping up every so often. Certainly the decision to self-publish does not appear to have been because of a lack of effort put into the book. An example of a quote I think is still relevant today: "Stable funding was a crucial factor for research. It meant that AT&T could take a long-term view and Bell Labs researchers had the freedom to explore areas that might not have a near-term payoff and perhaps never would. That's a contrast with today's world, in which planning often seems to look ahead only a few months, and much effort is spent on speculating on financial results for the next quarter." (page 7). Kernighan covers his own early career and the general functioning of Bell Labs, before starting to delve into the history of Unix. Describing at a high level early batch processing systems and then Multics, Kernighan describes how…

Continue ReadingUnix: a history and a memoir

The Kaiju Preservation Society

  • Post author:
  • Post category:Book

This is a classic Scalzi book -- novel, fun, a little bit funny, and in sync with the time it was created in. A story set around a product manager laid off during the pandemic, and then presented with an... unusal employment option, the book moves quickly and in a way which keeps you engaged. Yet another Scalzi book I really enjoyed.

Continue ReadingThe Kaiju Preservation Society

The Three-Body Problem

  • Post author:
  • Post category:Book

I'm torn about this book -- the premise is interesting, the world is novel, and the book is well written. The book has a strong environmental theme, with a focus on the environmental impact of Chinese economic development during Mao's cultural revolution. However, despite all that the book didn't "grab" me. I think perhaps its because there is a lot of effort spent describing things which ultimately don't really matter -- like weather or not the desktop PC being used by one of the characters is the current model or not. Or perhaps its because I didn't actually like any of the characters -- none of them is what I would call a nice person. Or perhaps this is an artifact of the book having been translated from Chinese, and perhaps different stylisting expectations or some such? Either way, I don't think I'll finish this trilogy.

Continue ReadingThe Three-Body Problem

All python packages require a pyproject.toml with modern pip

So last night Shaken Fist CI jobs started failing with errors like this (editted lightly for clarity): Building wheels for collected packages: shakenfist-ci Building wheel for shakenfist-ci (setup.py): started Building wheel for shakenfist-ci (setup.py): finished with status 'error' error: subprocess-exited-with-error × python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [86 lines of output] ... ...setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. setuptools.SetuptoolsDeprecationWarning, installing to build/bdist.linux-x86_64/wheel running install ... warning: install_lib: byte-compiling is disabled, skipping. running install_egg_info Copying shakenfist_ci.egg-info to build/bdist.linux-x86_64/wheel/shakenfist_ci-0.0.1.dev2544-py3.7.egg-info running install_scripts error: invalid command 'bdist_wininst' [end of output] This was pretty concerning. I know that a setup.py / setup.cfg style install is a little old school, but it was unexpected that it broke entirely. At first I thought I'd have to convert to poetry to unblock this, but Chet helpfully pointed out that this is as simple as adding a pyproject.toml file to the directory which contains your setup.py and setup.cfg. The basic issue is that a modern pip doesn't assume that you're going to use setuptools, so you need to tell it that you're doing that in pyproject.toml. Then you're unblocked. So, just create a file named…

Continue ReadingAll python packages require a pyproject.toml with modern pip

Debian 10 buster bcrypt pip install breakage

So, as of today by Shaken Fist CI jobs for Debian 10 are failing to install bcrypt, with an error that looks like this: Running setup.py install for bcrypt: started Running setup.py install for bcrypt: finished with status 'error' [ ... snip ... ] running build_rust =============================DEBUG ASSISTANCE============================= If you are seeing a compilation error please try the following steps to successfully install bcrypt: 1) Upgrade to the latest pip and try again. This will fix errors for most users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip 2) Ensure you have a recent Rust toolchain installed. bcrypt requires rustc >= 1.56.0. Python: 3.7.3 platform: Linux-4.19.0-21-amd64-x86_64-with-debian-10.12 pip: 18.1 setuptools: 65.2.0 setuptools_rust: 1.5.1 rustc: n/a =============================DEBUG ASSISTANCE============================= I'm not really interested in debating why installing a python package requires a rust compiler, that has been dicussed elsewhere. This specific breakage has been caused by bcrypt releasing 4.0.0, which has this in the changelog: "bcrypt is now implemented in Rust. Users building from source will need to have a Rust compiler available. Nothing will change for users downloading wheels." Unfortunately, you can't just install rustc with apt, as it is both quite big (350mb), and too old (version 1.41.1 versus the required 1.56.0 or better). I also couldn't…

Continue ReadingDebian 10 buster bcrypt pip install breakage

postgres_log_dir error while installing Pulp 3.20

I’m new to pulp and am installing based on the Ansible roles as documented at Getting started - Pulp Installer. During the install, I get this error: TASK [geerlingguy.postgresql : Define postgresql_log_dir.] ************ fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'log_directory'\n\nThe error appears to be in '/root/.ansible/roles/geerlingguy.postgresql/tasks/variables.yml': line 58, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Define postgresql_log_dir.\n ^ here\n"} This seems to be that a newer version of geerlingguy.postgresql has been released (version 3.4) which requires some changes to the Pulp Ansible installer. Those changes have been made in this pull request, which has been merged but not yet released. For others blocked on this, you can instead lock the version of geerlingguy.postgresql to something older, which will then allow the install to work: ansible-galaxy install geerlingguy.postgresql,3.3.1 ansible-galaxy collection install pulp.pulp_installer

Continue Readingpostgres_log_dir error while installing Pulp 3.20

Interpreting whiteout files in Docker image layers

I've been playing again with Docker images and their internal layers a little more over the last week -- you can see some of my previous adventures at Manipulating Docker images without Docker installed. The general thrust of these adventures is understanding the format and how to manipulate it by building a tool called Occy Strap which can manipulate the format in useful ways. My eventual goal there is to be able to build OCI compliant image bundles and then have a container runtime like runc execute them, and I must say I am getting a lot closer. This time I was interested in the exact mechanisms used by whiteout files in those layers and how that interacts with Linux kernel overlay filesystem types. Firstly, what is a whiteout file? Well, when you delete a file or directory from a lower layer in the Docker image, it doesn't actually get removed from that lower layer, as layers are immutable. Instead, the uppermost layer records that the file or directory has been removed, and it is therefore no longer visible in the Docker image that the container sees. This has obvious security implications if you delete a file like a password you…

Continue ReadingInterpreting whiteout files in Docker image layers

End of content

No more pages to load

Newer Posts
Older Posts