On 1/15/21 2:50 PM, Tadej Janež wrote:

Hi all!

Hey Tadej

I was recently hit by this and it took me some time to figure out what was going on.

Thanks for sharing and for investigating.

I provisioned a fresh Fedora 33 cloud instance on DigitalOcean. After completing the initial upgrade via 'dnf update' and rebooting the system, DNS resolution stopped working.

I did a little debugging and saw that no DNS servers were configured:

$ resolvectl dns Global: Link 2 (eth0): Link 3 (eth1): $ resolvectl status Global Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported resolv.conf mode: stub

Link 2 (eth0) Current Scopes: LLMNR/IPv4 LLMNR/IPv6 Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (eth1)

I was confused how could that happen since running 'dnf update' on the first boot clearly required DNS resolution to work.

It turns out a recent systemd update, namely systemd-246.7-2.fc33 (https://bodhi.fedoraproject.org/updates/FEDORA-2020-3616681a70), includes this commit: https://src.fedoraproject.org/rpms/systemd/c/ed795fb1fc9a2c20ebcac34bdf7e7c7... which removes the fallback DNS server list (i.e. Cloudflare's and/or Google's DNS servers) due to concerns over DNS requests being personally identifying information (PII) in the sense of GDPR?

In combination with NetworkManager's DNS server handling being disabled via cloud-init:

$ cat /etc/NetworkManager/conf.d/99-cloud-init.conf # Created by cloud-init on instance boot automatically, do not edit. # [main] dns = none

this results in no DNS servers being configured on the system.

The consequence is that Fedora 33 cloud instances are broken after updating the systemd package to version 246.7-2.fc33.

There is a bug in cloud-init where it doesn't seem to properly handle systemd-resolved. It edits /etc/resolv.conf directly (which is already managed by systemd-resolved) and thus the configuration gets lost after the first boot.

https://bugzilla.redhat.com/show_bug.cgi?id=1950770

I just tried to reproduce on DigitalOcean and it seems like maybe DO recently added a workaround for this (i'm not able to reproduce). I see this workaround:

# cat lib/cloud/instances/242253247/boothooks/resolver-fix #!/bin/sh -x # DigitalOcean resolver fixup script [ -f /etc/systemd/resolved.conf ] || exit 0 logger -t DigitalOcean 'writing resolvers to /etc/systemd/resolved.conf' printf "[Resolve]\nDNS=67.207.67.2 67.207.67.3\n" > /etc/systemd/resolved.conf

Can you try again and see if a new f33 instance you start is OK?

(I haven't checked but if Fedora 34 cloud images include this version of systemd, then DNS resolution won't work even on first boot.)

Thanks, I'll look into it.

Please advise on the best steps to fix this issue

Regards, Tadej