AWS – Made by Mikal

Amazon Linux 2023, DNS, and systemd-resolved — a story of no caching

Post author:mikal
Post published:November 11, 2024
Post category:AmazonLinux AWS

I've learned a few things on an adventure this week, and I figure I should probably write them down. First off, AWS throttles the number of DNS queries you can perform on a VPC. Apparently you're limited to 1,024 packets for Elastic Network Interface (ENI). I am a little unclear on if the limit is per instance ENI, or the ENI on the VPC that is the DNS server. I am also unsure if that's 1,024 request packets, or 1,024 total packets, but either way there is definitely a limit after which you will be throttled. Secondly, AL2023 disables the systemd-resolved DNS caching behaviour, which means its pretty easy to hit that throttling limit. When you google for solutions you'll find re:Post posts recommending dnsmasq, which is a perfectly fine piece of software but not really necessary if you already have systemd-resolved installed on your instance (as you do with AL2023). First off you can verify that you're not caching DNS with a command like this: $ sudo resolvectl status Global Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported resolv.conf mode: uplink Link 2 (eth0) Current Scopes: DNS Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported DNS Servers: 192.168.1.3 The "resolv.conf mode: uplink" here means the "stub…

Providing stable EBS volume device files

Post author:mikal
Post published:June 4, 2024
Post category:AWS

So I had a little adventure at work today and I am sure this is going to come up again. Imagine that you have an AWS instance with more than one EBS volume attached. On modern instance types, the EBS volumes appear as NVMe device files, but the naming of the device files is not stable -- it depends on what PCI device is detected by the kernel first etc. It turns out that providing stable names for the device files is a solved problem though! Specifically, CoreOS has udev rules which use a short script to lookup the EC2 EBS device name from the vendor-specific portion of the NVMe id-ctrl data, and provide an appropriate symlink. This saved me a fair bit of mucking around providing stable UUIDs for EBS volume templates, because we can instead just set the device name in the launch template and then have udev enforce that device name on boot. So that's nice. There is of course no real equivalent for OpenStack, as OpenStack generally uses qemu virtual disks not fake NVMe disks. I should think about that some more sometime. For what its worth, GCE uses the device serial number it seems based…

On-demand Container Loading in AWS Lambda

Post author:mikal
Post published:June 9, 2023
Post category:AWS

My team at work now has a daily personal learning time called “egg time” — its a slightly silly story involving a manager who was good at taking some time to learn things each day, and an egg shaped chair.

Today I decided that I should read this paper about container image loading in AWS lambda, as recommended by Robert Collins on LinkedIn. The paper details the work they had to do to transition from all Lambda functions being packaged as relatively small zip files (250mb), to relatively large Docker containers (10gb+) while maintaining their aggressive target cold-start time of 50ms.

(more…)