Prometheus 2.12, query logging, and startup failures on macos

Share

Prometheus v2.12 added active query logging. The basic idea is that there is a mmaped JSON file that contains all of the queries currently running. If prometheus was to crash, that file would therefore be a list of the queries running at the time of the crash. Overall, not a bad idea.

Some friends had recently added prometheus to their development environments. This is wired up to grafana dashboards for their microservices, and prometheus is configured to store 14 days worth of time series data via a persistent volume from the developer desktops. We did this because it is valuable for the developers to be able to see the history of metrics before and after their changes.

Now we have a developer using macos as their primary development platform, and since prometheus 2.12 it hasn’t worked. Specifically this developer is using parallels to provide the docker virtual machine on his mac. You can summarise the startup for prometheus in the dev environment like this:

$ docker run ...stuff...
...snip...
level=error ts=2019-09-15T02:20:23.520Z caller=query_logger.go:94 component=activeQueryTracker msg="Failed to mmap" file=/prometheus-data/data/queries.active Attemptedsize=20001 err="invalid argument"
panic: Unable to create mmap-ed active query log

goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff9917af38, 0x15, 0x14, 0x2a6b7c0, 0xc00003c7e0, 0x2a6b7c0)
	/app/promql/query_logger.go:112 +0x4d2
main.main()
	/app/cmd/prometheus/main.go:361 +0x52bd

And here’s the underlying problem — because of the way the persistent data is mapped into this container (via parallels sharing in this case), the mmap of the active queries file fails and prometheus fails to start.

In other words, since prometheus 2.12 your prometheus data files have to be stored on a filesystem which supports mmap. Additionally, there is no flag to just disable the active query logger.

So how do we work around this? Well, here’s a horrible workaround — in the data directory that is volume mapped into the container, create a symlink that is to a path that is mmapable inside the docker container, even if that path doesn’t exist outside the container. For example, given that we store the prometheus time series at $CONFIG/prometheus-data:

$ ln -s /tmp/queries.active "$CONFIG/prometheus-data/queries.active"

Note that /tmp/queries.active does not exist on the developer’s mac. Prometheus now starts and its puppies and kittens the whole way down.

Share

The Ghost Brigades (2)

Share

The second time around I think my opinion has changed a little. I found the plot a little hard to believe (perhaps I am scarred by other book’s twee explorations of the motivations of alien species), and overall the book not as good as Old Man’s War. Then again, its far from the worst book I have read this year.

Original post about this book.

[award: nominee prometheus 2007]
[isbn: 0765354063]

Share

Buying Time

Share

This book isn’t Haldeman’s best work (checkout The Forever War, The Forever Peace, or Marsbound for examples of his really good stuff). I found the characters largely unsympathetic, and the plot quite slow. The book is also odd in that it was written in 1989, but is full of stuff you’d expect to see from a Heinlein novel like The Moon Is a Harsh Mistress — global conspiracies, Russian space colonies with vigilante law, that sort of thing.

Interestingly, the plot twist is much more smoothly done than many other Haldeman novels, which is nice.

[award: nominee prometheus 1990]
[isbn: 0380704390]

Share

The Ghost Brigades

Share

This book is a really good sequel, and just as good as Old Man’s War. While some of the characters reappear, the story stands on its own and is quite entertaining. I enjoyed this book a lot. There’s something about bright green genetically engineered super soldiers killing aliens that makes me happy.

[award: nominee prometheus 2007]
[isbn: 0765354063]

Share

The System of the World

Share

The final book of the Baroque Cycle, and it feels like a real achievement to have gone through them all. They have their slow spots, but also excellent action and characters I love. Best of all, this book focuses on the latter two, with only one section of long theoretical dialog (about the nature of God in this case), which was so common in the other books and somethings so hard to follow. This story line was well worth the 2,500 or so pages it took, and the last book was a pleasure to read. It feels like there should be more books in this universe, but I’m not aware of any — perhaps later?

[award: winner prometheus 2005]
[isbn: 0060750863;0060750863]

Share

The Diamond Age

Share

I loved this book. I think its going on my list of all time favorites. The first half of the book is better than the second, but I think that might be because young Nell resonated much more strongly with me than older Nell. I read this book because it came up at a scientific conference where I went to a presentation about 3D fabrication, and its now entirely clear why the presenter thought this was relevant, given 3D nanotech fabrication plays a strong part in this book. Excellent read.

[award: nominee prometheus 1996; winner hugo 1996; nominee nebula_novel 1996]
[isbn: 0553573314]

Share

Mona Lisa Overdrive

Share

This is the book which wraps up the Sprawl series (Burning Chrome, Neuromancer and Count Zero). Its a great book, with several separate story lines which are beautifully molded together by the end of the book. It also wraps up the confusing elements of the various other stories nicely. I really enjoyed it.

[award: nominee nebula_novel 1988; nominee hugo 1989; nominee prometheus 1989]
[isbn: 0553281747]

Share

Cryptonomicon

Share

I read this book on an international trip, and it was a good choice for that. Its long (around 900 pages), but very readable. This is the second time I’ve read the book, and this time its amazing how well the description of Silicon Valley startups matches my experiences there. I love this book.

[award: nominee hugo 2000; nominee prometheus 2000]
[isbn: 0099410672]

Share

Friday

Share

This is the first Heinlein book I have read in a long time — since High School in fact. I read this one simply on an impulse, as the back cover description made it sound interesting. Heinlein isn’t on the list of authors that I am pursuing at the moment, although I might consider changing that.

This book covers a more complicated Earth than the one we have right now, although in some ways its more simple. The main character Friday doesn’t ever seem to have trouble making friends, and portions of the book are just a series of her romantic entanglements. Most of the complexities are political. The story is mostly about a journey, both physical as well as emotional, and interestingly there isn’t a consistent opposing force. I suspect that might be unusual, at least for the stuff I read.

This book was good, even if the constant romantic entanglements seemed extraneous.

[award: winner nebula 1982; winner hugo 1982; winner locus 1983; winner prometheus 1983]
[isbn: 034530988x]

Share