Amazon Linux 2023, DNS, and systemd-resolved — a story of no caching
I've learned a few things on an adventure this week, and I figure I should probably write them down. First off, AWS throttles the number of DNS queries you can perform on a VPC. Apparently you're limited to 1,024 packets for Elastic Network Interface (ENI). I am a little unclear on if the limit is per instance ENI, or the ENI on the VPC that is the DNS server. I am also unsure if that's 1,024 request packets, or 1,024 total packets, but either way there is definitely a limit after which you will be throttled. Secondly, AL2023 disables the systemd-resolved DNS caching behaviour, which means its pretty easy to hit that throttling limit. When you google for solutions you'll find re:Post posts recommending dnsmasq, which is a perfectly fine piece of software but not really necessary if you already have systemd-resolved installed on your instance (as you do with AL2023). First off you can verify that you're not caching DNS with a command like this: $ sudo resolvectl status Global Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported resolv.conf mode: uplink Link 2 (eth0) Current Scopes: DNS Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported DNS Servers: 192.168.1.3 The "resolv.conf mode: uplink" here means the "stub…