Minor questions in Linux file semantics

I’ve known for a long time that if you delete a file on Unix / Linux but that file is open somewhere, the blocks used by the file aren’t freed until that user closes the file (or is terminated), but I was left wondering about some other edge cases.

Shaken Fist has a distributed blob store. It also has a cache of images that virtual machines are using. If the blob store and the image cache are on the same filesystem, sometimes the image cache entry can be a hard link to an entry in the blob store (for example, if the entry in the blob store doesn’t need to be transcoded before use by the virtual machine). However, if they are on different file systems, I instead use a symbolic link.

This raises questions — what happens if you rename a file which is open for writing in a program? What happens if you change a symbolic link to point somewhere else while it is open? I suspect in both cases the right thing happens, but I decided I should test these theories out.

Continue reading “Minor questions in Linux file semantics”

Malware Analyst’s Cookbook and DVD

Another technical book, this time because my employer lets me buy random technical books as long as I pinky swear to read them and this one sounded interesting and got good reviews.

First off, the book is a bit dated given its from 2011 — there are lots of references to Ubuntu 10.10 for example and they say to avoid Python 3, which has its historical charm. This is unfortunate given the first section of the book talks about setting up honeypots to collect malware to examine, but Dionaea for example had its last commit in 2021. I am left wondering if there are more modern honey pot systems that people use these days.

Secondly the book is definitely a cookbook and that’s on me for not noticing this about the book before buying it — its a series of recipes / scripts that do interesting things with malware. That said, it isn’t really teaching a cohesive set of skills, its more of a series of stepping stones along the path you might follow. I think that’s an unintended piece of important learning — books with “cookbook” or “recipes” in their title probably aren’t very good as an overview of a topic area. My bad.

That said, some parts of the book are very good — the discussion of whois, DNS, and Real Time Black Lists (RTBLs) is helpful and less focussed on providing scripts you could run. The discussion of how to log changes to a Windows system, detect attempts to hide files in NTFS filesystems, and detect changes to registry hives were interesting in an abstract way, but perhaps obvious to someone who actually uses Windows.

Overall, I’m a bit disappointed in this book and it will be exhiled to a shelf at the office as a punishment.

Malware Analyst's Cookbook and DVD Book Cover Malware Analyst's Cookbook and DVD
Michael Ligh, Steven Adair, Blake Hartstein, Matthew Richard,
Computers
John Wiley & Sons
November 2, 2010
747

A computer forensics "how-to" for fighting malicious code and analyzing incidents. With our ever-increasing reliance on computers comes an ever-growing risk of malware. Security professionals will find plenty of solutions in this book to the problems posed by viruses, Trojan horses, worms, spyware, rootkits, adware, and other invasive software. Written by well-known malware experts, this guide reveals solutions to numerous problems and includes a DVD of custom programs and tools that illustrate the concepts, enhancing your skills. Security professionals face a constant battle against malicious software; this practical manual will improve your analytical capabilities and provide dozens of valuable and innovative solutions Covers classifying malware, packing and unpacking, dynamic malware analysis, decoding and decrypting, rootkit detection, memory forensics, open source malware research, and much more.

The BeyondCorp papers

Google’s BeyondCorp effort would probably be what we would now call Zero Trust, although I am surprised by how little name recognition BeyondCorp has when I talk to security people about Zero Trust. Perhaps there are subtle differences between the two, but if there are they aren’t obvious to me. I find myself reading the relevant Usenix papers for BeyondCorp, so I figure I’ll post a summary of what I got from each paper here.

The earliest of these papers are quite old now (2014), especially for something the rest of the industry is only starting to talk a lot about at the moment. I wonder if there is a viable business model in watching what papers megacorps like Google publish, and the implementing them as commercialized products before the rest of the market catches on?

Either way, here’s a summary of the various papers from the perspective of an interested bystander…

Continue reading “The BeyondCorp papers”

Cisco CyberOps Associate: Official Cert Guide

I don’t think I’ve really reviewed a technical book here before, but I read the thing so I guess I should. This book is the certification guide for a “Cisco CyberOps Associate” certification, which is what they now call the CCNA Security qualification. Its a relatively junior certification, qualifying you to be a level one operator in a Security Operations Centre (SOC).

I read this book because I took a Cisco NetAcad course for the associated certification in the second half of 2022 (although it has continued to be a thing I plug away at in 2023). That was mainly motivated by a desire to more about a field that is clearly important, but hasn’t been core to my personal career.

This book is reasonably well written and readable — I’d read a chapter in the evening after work and its wasn’t a huge chore to churn though. I certainly learned things along the way, even if the certification seems to suffer from a desire to have everyone rote learn a lot of acronyms, which seems like a common ailment in the industry (AWS Certified Cloud Practitioner, I’m looking at you).

My main critism is of the qualification itself, which is that it is quite Cisco centric — almost all examples of the implementation of a technology are a Cisco product, which is great if you’re trying to demonstrate the depth of Cisco’s portfolio, but isn’t great if you’re competing with less vendor centric certification options. This is in contrast to the CCNA content, which feels more vendor neutral to me because its more fundamental.

That said, this book wasn’t a waste of my time and I learned stuff — which I guess is mission accomplished for a technical book?

Cisco Cyberops Associate Cbrops 200-201 Official Cert Guide Book Cover Cisco Cyberops Associate Cbrops 200-201 Official Cert Guide
Omar Santos
Computers
Cisco Press
August 6, 2020
900

Modern organizations rely on Security Operations Center (SOC) teams to vigilantly watch security systems, rapidly detect breaches, and respond quickly and effectively. To succeed, SOCs desperately need more qualified cybersecurity professionals. Cisco's new Cisco Certified CyberOps Associate certification prepares candidates to begin a career working as associate-level cybersecurity analysts within SOCs.

Exploring more efficient remote large file storage

My primary personal project is a thing called Shaken Fist these days — it is an infrastructure as a service cloud akin to OpenStack Compute, but smaller and simpler. Shaken Fist doesn’t have an equivalent to the OpenStack Image service, instead letting your describe your instance images by a standard URL. One of the things Shaken Fist does to be easier to use is it maintains an official repository of common images, which allows users to refer to those images with a shorthand syntax instead of a complete URL. The images also contain small customizations (mainly including the Shaken Fist in-guest agent), which means I can’t just use the official upstream cloud images like OpenStack does.

The images were stored at DreamHost until this week, when a robot decided that they looked like offline backups, despite being served to the Internet via HTTP and being used regularly (although admittedly not frequently). DreamHost unilaterally decided to delete the web site, so now I am looking for new image hosting services, and thinking about better ways to build an image store.

(Oh, and recommending to anyone who asks that they consider using someone less capricious than DreamHost for their hosting needs).

Continue reading “Exploring more efficient remote large file storage”

This is going to hurt

This book is lots of things: honest, funny, and ultimately heart breaking. I don’t remember how I came across it, but its a good read for when travelling as the diary format means you can put it down whenever you need to do something else.

I’m left wondering how the Australian medical system compares to the NHS — I know we have more patient choice and flexibility — but I wonder what its like for those working within the system.

Either way I definitely recommend this book.

This is Going to Hurt Book Cover This is Going to Hurt
Adam Kay
Biography & Autobiography
Pan Macmillan
2018
279

As soon as Adam Kay set foot on a hospital ward for the first time, he realised there's quite a lot they don't teach you at medical school ... His diaries from the NHS front line - scribbled in secret after long nights, endless days and missed weekends - are hilarious, horrifying and heartbreaking by turns. This Is Going to Hurt is everything you wanted to know about being a junior doctor, and more than a few things you really didn't. And yes, it may leave a scar.

Ansible 7.0 onwards requires blocking IO from stdin, stdout, and stderr

Shaken Fist CI started failing this afternoon with this message logged:

ERROR: Ansible requires blocking IO on stdin/stdout/stderr.
Non-blocking file handles detected: <stdout>

Specifically this was happening when using ansible-galaxy to install some requirements, but the check is a more generic check than that was implemented by this ansible pull request, which appears to have been released with ansible-core 2.14 on November 8. That sat around until today, when ansible 7.0.0 was released and broke CI for me.

To be completely honest I’m not sure what’s happening here — somewhere in GitHub actions calling a shell script that calls ansible-galaxy the stdout file descriptor gets set to non-blocking and everything breaks. I’m unsure exactly where because its a pain to track down.

That said, Jack came to the rescue with this gem:

ansible-galaxy install andrewrothstein.etcd-cluster | cat -

Which unblocks me. It will be interesting to see if other people encounter problems with this change.

Unix: a history and a memoir

It was a bit surprising to me that Brian Kernighan self-published a book about Unix history with Kindle Direct publishing, but given how many other books he’s published he must have his reasons for not using traditional channels for this one. The book is an engaging read, with quotes which still seem timely today popping up every so often. Certainly the decision to self-publish does not appear to have been because of a lack of effort put into the book. An example of a quote I think is still relevant today:

“Stable funding was a crucial factor for research. It meant that AT&T could take a long-term view and Bell Labs researchers had the freedom to explore areas that might not have a near-term payoff and perhaps never would. That’s a contrast with today’s world, in which planning often seems to look ahead only a few months, and much effort is spent on speculating on financial results for the next quarter.” (page 7).

Kernighan covers his own early career and the general functioning of Bell Labs, before starting to delve into the history of Unix. Describing at a high level early batch processing systems and then Multics, Kernighan describes how Multics suffered from the second systems effect (see The Mythical Man Month for a good description of that).

Overall, Kernighan describes what must have been an amazing work environment — a grouping of absolute leaders in their field at a time where there was so much fundamental technology to be invented and described. I can’t help but be a little jealous — I’m not sure there is a modern equivalent. Google for example would have had a similar set of circumstances at times in its history, but wasn’t as inclined to share with the greater world like Bell Labs did. Interestingly Eric Schmidt was a summer student at Bell Labs — he re-wrote Lex while there.

Another interesting contribution from Kernighan is the following rule, which I think we’ve lost sight of in a world of large monolithic code bases:

“…a good example of a general rule: if a program writes your code for you, the code will be more correct and reliable than if you write it yourself by hand. If the generator is improved… everyone benefits.” (page 96).

I think this applies to libraries as well — if you’re going to write something which might be useful to others, its much better off in a library where others can get to it than in your own codebase. That said, I think avoiding leftpad seems like a noble goal.

Overall this was an enjoyable book and I recommend it.

Unix Book Cover Unix
Brian W. Kernighan
Operating systems (Computers)
October 18, 2019
198

"The fascinating story of how Unix began and how it took over the world. Brian Kernighan was a member of the original group of Unix developers, the creator of several fundamental Unix programs, and the co-author of classic books like "The C Programming Language" and "The Unix Programming Environment".

The Kaiju Preservation Society

This is a classic Scalzi book — novel, fun, a little bit funny, and in sync with the time it was created in. A story set around a product manager laid off during the pandemic, and then presented with an… unusal employment option, the book moves quickly and in a way which keeps you engaged.

Yet another Scalzi book I really enjoyed.

The Kaiju Preservation Society Book Cover The Kaiju Preservation Society
John Scalzi
Tor
March 17, 2022
336

The Three-Body Problem

I’m torn about this book — the premise is interesting, the world is novel, and the book is well written. The book has a strong environmental theme, with a focus on the environmental impact of Chinese economic development during Mao’s cultural revolution.

However, despite all that the book didn’t “grab” me. I think perhaps its because there is a lot of effort spent describing things which ultimately don’t really matter — like weather or not the desktop PC being used by one of the characters is the current model or not. Or perhaps its because I didn’t actually like any of the characters — none of them is what I would call a nice person. Or perhaps this is an artifact of the book having been translated from Chinese, and perhaps different stylisting expectations or some such?

Either way, I don’t think I’ll finish this trilogy.

The Three-Body Problem Book Cover The Three-Body Problem
Cixin Liu
December 3, 2015
416

1967: Ye Wenjie witnesses Red Guards beat her father to death during China's Cultural Revolution. This singular event will shape not only the rest of her life but also the future of mankind. Four decades later, Beijing police ask nanotech engineer Wang Miao to infiltrate a secretive cabal of scientists after a spate of inexplicable suicides. Wang's investigation will lead him to a mysterious online game and immerse him in a virtual world ruled by the intractable and unpredicatable interaction of its three suns. This is the Three-Body Problem and it is the key to everything: the key to the scientists' deaths, the key to a conspiracy that spans light-years and the key to the extinction-level threat humanity now faces.