Author: mikal

Using the openstacksdk with authentication arguments

I wanted to authenticate against OpenStack recently, and had a lot of trouble finding documentation about how to authenticate just by passing arguments (as opposed to by using clouds.yaml or environment variables). Now that I have a working incantation, I figure I should write it down so I can find it again. Its also disappointing the OpenStack documentation doesn’t appear to cover this particularly well…

from keystoneauth1.identity import v3
from keystoneauth1 import session
from openstack import connection


auth = v3.Password(
    auth_url='http://kolla.home.stillhq.com:5000',
    username='admin',
    password='...',
    project_name='admin',
    user_domain_id='default',
    project_domain_id='default')
sess = session.Session(auth=auth)

conn = connection.Connection(session=sess)

print([x.name for x in conn.list_servers()])

This code will authenticate using the arguments provided, and then list all the servers (instances) visible to that user. You’re welcome.

Fetching the most recent GitHub actions runner version

One of the struggles I have with running self-hosted GitHub actions runners is that GitHub releases new versions of the runner quite often and I don’t notice. That’s fine as long as you ignore the scary warnings on action output, until they drop support for whatever random old runner you’re using. They did just that to me this week. The best bit was that the “old runner” was only a month old!

I was left wondering if I could automate this. The answer is thankfully yes.

Specifically, I wanted to automate it with a GitHub action which downloads the runner and puts it into the self-hosted runner image. That looks like this:

- name: Install the github command line
  run: |
    sudo apt update
    sudo apt install -y curl

    curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg
    sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg
    echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null
    sudo apt update
    sudo apt install -y gh

- name: Lookup latest version of the GitHub actions runner
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  run: |
    actions_url=$(gh release view --repo actions/runner --json assets | \
        jq -r '.assets[].url | select (contains("linux-x64-2")) | select (test("[0-9].tar.gz$"))')
    echo "GITHUB_ACTIONS_URL=$actions_url" >> $GITHUB_ENV

- name: Cache github actions runner
  run: |
    curl -o /srv/ci/github-actions-runner.tar.gz ${GITHUB_ACTIONS_URL}

For my setup, this runs in an action which builds a new virtual machine image for the github runners each night. That third step downloads the runner image, and caches it to the virtual machine’s disk. The virtual machine then installs the github actions running on boot for each ephemeral worker.

Turnover of Companies in OpenStack: Prevalence and Rationale

This paper examines the withdrawal behaviour of corporate contributors to OpenStack, which seems particularly relevant given most contributions in OpenStack are corporately supported, and corporate engagement is declining over time. Its also directly relevant to my own experiences contributing to the project, so seemed like a thing I should read.

One interesting aspect of the study is how they define withdrawal from contributions. For each company, they calculate an individual frequency of contribution, and then use that to determine if the company is still making contributions. That is, of a company only ever contributed once a year, we must wait at least a year to know that they have indeed stopped contributing.

The paper finds that in more recent OpenStack releases, more companies are leaving contributions than joining. The authors assert that in general engaged developers are now less experienced than previously, which presents risks in terms of developer effectiveness as well as code quality. However, the paper does note that companies with smaller contributions are more likely to disengage than “sustaining companies”, however that’s largely because there are a huge number of companies contributing only one developer who makes a small number of commits.

Unsurprisingly, the paper notes that companies which contribute more are more likely remain as contributors — both because of momentum, but also because they’re more likely to have a say in the roadmap direction of the project and therefore whether it fits their needs or priorities. They use some loaded words like “dominated by a small number of contributors”, but I don’t think that’s really helpful given that other companies could choose to contribute if they wanted to. I think some of this behaviour is what I would call “rent seeking” — players who contribute little but think that the project somehow owes them changes to make their commercialisation successful. The researchers also note an additional factor here — OpenStack isn’t well suited to small environments, so larger organizations are more likely to have a successful deployment and therefore stay as contributors.

Overall I’d describe this paper as not particularly groundbreaking, but perhaps useful when trying to decide what behaviour to encourage in an Open Source community in order to make a project sustainable.

On-demand Container Loading in AWS Lambda

My team at work now has a daily personal learning time called “egg time” — its a slightly silly story involving a manager who was good at taking some time to learn things each day, and an egg shaped chair.

Today I decided that I should read this paper about container image loading in AWS lambda, as recommended by Robert Collins on LinkedIn. The paper details the work they had to do to transition from all Lambda functions being packaged as relatively small zip files (250mb), to relatively large Docker containers (10gb+) while maintaining their aggressive target cold-start time of 50ms.

Continue reading “On-demand Container Loading in AWS Lambda”

Holman CLXRGB60 RGB WiFi garden light controllers and tasmota

Today I went forth to Bunnings in the rain to purchase a Holman CLXRGB60 RGB garden light controller so that I too could have fancy lighting in my garden and impress all those guests I never have over. I had been given hope by the Blakadder site that I would be able to flash tasmota onto the controller so it integrated with my Home Assistant home automation.

Unfortunately, it was not to be. Despite the device being TYWE3L based, the warning on the blakaddr site was correct, and this is a next-gen Tuya device where the crypto hasn’t been broken yet. Then again, I couldn’t even get this device to pair in the Holman app, so it clearly hates me.

This unfortunately means the excellent instructions from Jon Oxer were unforunately not helpful today. I think there is a theoretical option here to flash using the serial pins on the board, ala this guide. Also, it means my hair got wet for nothing.

So as to take revenge for my wet hair I have decided to pivot. The Holman lights seem quite well made, but they’re just 12 volt RGB PWM devices. So I can use their lights and build my own controller — although I need to ponder how to drive high current PWM I suppose. I thought it would therefore be useful to document the pinout on the Holman connector before I return the controller.

Labelled RGB Holman Garden Light 12v pinout

I hope this is helpful to someone else as well. I wonder what this connector is called?

Minor questions in Linux file semantics

I’ve known for a long time that if you delete a file on Unix / Linux but that file is open somewhere, the blocks used by the file aren’t freed until that user closes the file (or is terminated), but I was left wondering about some other edge cases.

Shaken Fist has a distributed blob store. It also has a cache of images that virtual machines are using. If the blob store and the image cache are on the same filesystem, sometimes the image cache entry can be a hard link to an entry in the blob store (for example, if the entry in the blob store doesn’t need to be transcoded before use by the virtual machine). However, if they are on different file systems, I instead use a symbolic link.

This raises questions — what happens if you rename a file which is open for writing in a program? What happens if you change a symbolic link to point somewhere else while it is open? I suspect in both cases the right thing happens, but I decided I should test these theories out.

Continue reading “Minor questions in Linux file semantics”

Malware Analyst’s Cookbook and DVD

Another technical book, this time because my employer lets me buy random technical books as long as I pinky swear to read them and this one sounded interesting and got good reviews.

First off, the book is a bit dated given its from 2011 — there are lots of references to Ubuntu 10.10 for example and they say to avoid Python 3, which has its historical charm. This is unfortunate given the first section of the book talks about setting up honeypots to collect malware to examine, but Dionaea for example had its last commit in 2021. I am left wondering if there are more modern honey pot systems that people use these days.

Secondly the book is definitely a cookbook and that’s on me for not noticing this about the book before buying it — its a series of recipes / scripts that do interesting things with malware. That said, it isn’t really teaching a cohesive set of skills, its more of a series of stepping stones along the path you might follow. I think that’s an unintended piece of important learning — books with “cookbook” or “recipes” in their title probably aren’t very good as an overview of a topic area. My bad.

That said, some parts of the book are very good — the discussion of whois, DNS, and Real Time Black Lists (RTBLs) is helpful and less focussed on providing scripts you could run. The discussion of how to log changes to a Windows system, detect attempts to hide files in NTFS filesystems, and detect changes to registry hives were interesting in an abstract way, but perhaps obvious to someone who actually uses Windows.

Overall, I’m a bit disappointed in this book and it will be exhiled to a shelf at the office as a punishment.

Malware Analyst's Cookbook and DVD Book Cover Malware Analyst's Cookbook and DVD
Michael Ligh, Steven Adair, Blake Hartstein, Matthew Richard,
Computers
John Wiley & Sons
November 2, 2010
747

A computer forensics "how-to" for fighting malicious code and analyzing incidents. With our ever-increasing reliance on computers comes an ever-growing risk of malware. Security professionals will find plenty of solutions in this book to the problems posed by viruses, Trojan horses, worms, spyware, rootkits, adware, and other invasive software. Written by well-known malware experts, this guide reveals solutions to numerous problems and includes a DVD of custom programs and tools that illustrate the concepts, enhancing your skills. Security professionals face a constant battle against malicious software; this practical manual will improve your analytical capabilities and provide dozens of valuable and innovative solutions Covers classifying malware, packing and unpacking, dynamic malware analysis, decoding and decrypting, rootkit detection, memory forensics, open source malware research, and much more.

The BeyondCorp papers

Google’s BeyondCorp effort would probably be what we would now call Zero Trust, although I am surprised by how little name recognition BeyondCorp has when I talk to security people about Zero Trust. Perhaps there are subtle differences between the two, but if there are they aren’t obvious to me. I find myself reading the relevant Usenix papers for BeyondCorp, so I figure I’ll post a summary of what I got from each paper here.

The earliest of these papers are quite old now (2014), especially for something the rest of the industry is only starting to talk a lot about at the moment. I wonder if there is a viable business model in watching what papers megacorps like Google publish, and the implementing them as commercialized products before the rest of the market catches on?

Either way, here’s a summary of the various papers from the perspective of an interested bystander…

Continue reading “The BeyondCorp papers”

Cisco CyberOps Associate: Official Cert Guide

I don’t think I’ve really reviewed a technical book here before, but I read the thing so I guess I should. This book is the certification guide for a “Cisco CyberOps Associate” certification, which is what they now call the CCNA Security qualification. Its a relatively junior certification, qualifying you to be a level one operator in a Security Operations Centre (SOC).

I read this book because I took a Cisco NetAcad course for the associated certification in the second half of 2022 (although it has continued to be a thing I plug away at in 2023). That was mainly motivated by a desire to more about a field that is clearly important, but hasn’t been core to my personal career.

This book is reasonably well written and readable — I’d read a chapter in the evening after work and its wasn’t a huge chore to churn though. I certainly learned things along the way, even if the certification seems to suffer from a desire to have everyone rote learn a lot of acronyms, which seems like a common ailment in the industry (AWS Certified Cloud Practitioner, I’m looking at you).

My main critism is of the qualification itself, which is that it is quite Cisco centric — almost all examples of the implementation of a technology are a Cisco product, which is great if you’re trying to demonstrate the depth of Cisco’s portfolio, but isn’t great if you’re competing with less vendor centric certification options. This is in contrast to the CCNA content, which feels more vendor neutral to me because its more fundamental.

That said, this book wasn’t a waste of my time and I learned stuff — which I guess is mission accomplished for a technical book?

Cisco Cyberops Associate Cbrops 200-201 Official Cert Guide Book Cover Cisco Cyberops Associate Cbrops 200-201 Official Cert Guide
Omar Santos
Computers
Cisco Press
August 6, 2020
900

Modern organizations rely on Security Operations Center (SOC) teams to vigilantly watch security systems, rapidly detect breaches, and respond quickly and effectively. To succeed, SOCs desperately need more qualified cybersecurity professionals. Cisco's new Cisco Certified CyberOps Associate certification prepares candidates to begin a career working as associate-level cybersecurity analysts within SOCs.

Exploring more efficient remote large file storage

My primary personal project is a thing called Shaken Fist these days — it is an infrastructure as a service cloud akin to OpenStack Compute, but smaller and simpler. Shaken Fist doesn’t have an equivalent to the OpenStack Image service, instead letting your describe your instance images by a standard URL. One of the things Shaken Fist does to be easier to use is it maintains an official repository of common images, which allows users to refer to those images with a shorthand syntax instead of a complete URL. The images also contain small customizations (mainly including the Shaken Fist in-guest agent), which means I can’t just use the official upstream cloud images like OpenStack does.

The images were stored at DreamHost until this week, when a robot decided that they looked like offline backups, despite being served to the Internet via HTTP and being used regularly (although admittedly not frequently). DreamHost unilaterally decided to delete the web site, so now I am looking for new image hosting services, and thinking about better ways to build an image store.

(Oh, and recommending to anyone who asks that they consider using someone less capricious than DreamHost for their hosting needs).

Continue reading “Exploring more efficient remote large file storage”