This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
== Owner ==
* Name: [[User:thaller|Thomas Haller]] (NetworkManager) * Email: thaller@redhat.com * Name: [[User:dustymabe| Dusty Mabe]] (Fedora CoreOS) * Email: dmabe@redhat.com
== Detailed Description ==
On Fedora, udev by default changes the MAC address of a wide range of software devices. This was introduced by systemd 242 in early 2019 (Fedora 31), when `MACAddressPolicy=` was extended to affect more types of devices.
With `MACAddressPolicy=persistent` udev's aim is to provide a stable MAC address, otherwise the kernel will assign a random one. However, that can cause problems:
Firstly, software devices are always created by some tool that has plans for the device. The tool may not expect that udev is going to change the MAC address and races against that. The best solution for the tool is to set the MAC address when creating an interface. This will prevent udev from changing the MAC address according to the MACAddressPolicy. Otherwise, the tool should wait for udev to initialize the device to avoid the race. In theory, a tool is always advised to wait for udev to initialize the device. However, if it were not for MACAddressPolicy, in common scenarios udev doesn't do anything relevant for software devices to make that necessary.
Secondly, for interface types bridge and bond, an unset MAC address has a special meaning to the kernel and the MAC address of the first port is used. If udev changes the MAC address, that no longer works. Now the generated MAC address is not directly discoverable as it is based on `/etc/machine-id` ([https://www.man7.org/linux/man-pages/man5/machine-id.5.html machine-id(5)]), among other data. Even if there were a tool to easily calculate the MAC address, it could be cumbersome to use it without logging into the machine first. The MAC address can directly affect the assigned IP address, for example when using DHCP. When booting a new virtual machine, the user might know the MAC address of the (virtual) "physical" interfaces. When bonding/bridging those interfaces, the bond/bridge would get one of the well known MAC addresses. `MACAddressPolicy=persistent` interferes with that.
The goal of persistent policy is to provide a stable MAC address. Note that if the tool or user who created the interface would want a certain MAC address, they have all the means to set it already. That applies regardless whether the tool is iproute2, NetworkManager, systemd-networkd. Neither NetworkManager nor systemd-networkd rely on udev's MACAddressPolicy for setting the MAC address. This behavior is mostly useful for plain `ip link add`, but it's unclear which real world user wants this behavior.
Of course, the user is welcome to configure the MAC address in any way they want. Including, dropping a link file that sets `MACAddressPolicy=persistent`. The problem is once udev sets a MAC address, it cannot be unset. Which makes this problematic to do by default.
While Fedora inherited this behavior from upstream systemd, RHEL-9 does not follow this behavior ([https://gitlab.com/redhat/centos-stream/rpms/systemd/-/blob/c8953519504bf2e6... centos9], [https://bugzilla.redhat.com/show_bug.cgi?id=1921094 rh#1921094]). For RHEL-8, this doesn't apply because the systemd there is too old to change the MAC address of most software devices.
This could be either implemented by patching `/usr/lib/systemd/network/99-default.link` to have a different policy, or by dropping a link file with higher priority. In the latter case, that override could be shipped either by udev or even by NetworkManager.
Another option is to change the scope of this proposal to only change to `MACAddressPolicy=none` for the device types where this causes the most issues (bridge/bond/team).
== Feedback ==
This was also discussed on upstream systemd mailing list [https://lists.freedesktop.org/archives/systemd-devel/2022-May/047893.html here]. The upstream systemd maintainers' opinion is that the current udev behavior is desirable, but accepts that distributions (or network stacks such as NetworkManager) may choose to change the default depending on their needs ([https://lists.freedesktop.org/archives/systemd-devel/2022-May/047943.html reference]). The goal here is to find out what the Fedora community thinks is the most appropriate default.
The RHEL-9 bug is [https://bugzilla.redhat.com/show_bug.cgi?id=1921094 [rh#1921094]].
== Benefit to Fedora ==
Pros:
- Consistent behavior with RHEL8 and RHEL9.
- Avoid race of udev and the tool that creates the interface.
- Bridge and bond interfaces can get the MAC addresses from their first port.
In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will get a MAC related to one of its physical NIC devices. In the case of provisioning new systems (especially in a large datacenter) information about the server can be used to configure the network environment (DHCP, Firewall, etc) before the server is ever even powered on. This does mean that you may have to update your network environment configuration if you replace a NIC (con), but that you won't have to update your network environment configuration if you re-install your server (pro), which is what you'd have to do now with `MACAddressPolicy=persistent`.
Cons:
- Deviate from upstream systemd.
It is desirable that RHEL and Fedora behaves similar. A possible outcome could be the current behavior stays and RHEL 10 would change behavior. On the other hand, different distributions (or even Fedora spins) have different uses and needs. Deviating might be fine. In the same vein, there is also a desire to stay close to upstream systemd behavior. But the uses of systemd project go beyond Fedora/RHEL, so deviating here may also be fine.
== Scope ==
* Proposal owners: The main goal of this request is to generate productive discussion and find the desired behavior. The implementation/changes are either way very simple.
* Other developers: Other projects that wish a certain MAC address are welcome to set it for their devices. Including using udev's MACAddressPolicy.
* Release engineering: Not needed for this change.
* Policies and guidelines: N/A (not needed for this Change) * Trademark approval: N/A (not needed for this Change) * Alignment with Objectives:
== Upgrade/compatibility impact ==
After the change, the MAC address for the affected device types changes.
== How To Test ==
1) Create a software device two times, for example `ip link add type bridge`. Note that the MAC address is either stable or random, depending on the `MACAddressPolicy=`.
2) Note that if the software device has the MAC address set initially, udev does not change it (`ip link add address aa:aa:aa:aa:aa:aa type bridge`). That depends on `/sys/class/net/$dev/addr_assign_type`.
3) Create a bridge/bond interface without setting the MAC address. Note that if `MACAddressPolicy=none`, the MAC address is random at first. Note that attaching the first port will update the controller's MAC address. On the other hand, with `MACAddressPolicy=persistent`, the MAC address of the controller is fixed and not inherited from the port.
4) Run
ip monitor link & while : ; do ip link del xxx ip link add name xxx type dummy \ && ip link set xxx addr aa:00:00:00:00:00 \ && ip link show xxx | grep -q aa:00:00:00:00:00 \ || break done
to reproduce the race between a simple tool and udev changing the MAC address.
== User Experience ==
Bond/bridge devices would again get assigned the MAC address of the first NIC added to the device. If we chose to not limit the scope of this change to just bonds/bridges then all software devices would get randomly assigned MAC addresses.
== Dependencies ==
None.
== Contingency Plan ==
If the change is rejected, nothing needs to be done. The change itself will be simple to implement. Contingency deadline: beta freeze Blocks release? No
== Documentation ==
TODO.
== Release Notes ==
On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
Hi!
I already participated in the upstream discussion, so what I write here will be a restatement to some extent, but with a look from the side of Fedora specifically.
The proposal has two variants: 1. just changing the policy to MACAddressPolicy=none or 2. limiting the change to bridge and bond devices.
Re variant 1: MACAddressPolicy=persistent applies to all devices that don't have a hardware address. The proposal as written (blanket MACAddressPolicy=none) would change behaviour for all kinds of devices, incl. e.g. software devices like veths, and cheap hardware devices without a fixed MAC. The proposal doesn't provide any justification for this (except for simplicity of implementation) and this variant seems pretty bad and I'm strongly opposed.
Re variant 2: the proposal limited to brige/bond devices seems much more reasonable. In particular, the case described below of a server (virtualized or not) in a big datacenter is the one case where the benefits of MACAddressPolicy=none are clearly visible. I still don't think it's worth changing the default, but here the cost:benefit ratio is much closer.
== Benefit to Fedora ==
Pros:
Consistent behavior with RHEL8 and RHEL9.
Avoid race of udev and the tool that creates the interface.
The race will happen if the creation is done in a specific way. But at the same time, even the Change proposal describes how to avoid the race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the situation can be summarized as "we have a bunch of 'big' tools that create devices like NetworkManager or systemd-networkd, which already know or can be easily fixed to avoid the race, and a manual tool which can be invoked in a way that avoids the race". Instead of changing the default in udev we could educate people how to invoke it better.
- Bridge and bond interfaces can get the MAC addresses from their first port.
In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will get a MAC related to one of its physical NIC devices. In the case of provisioning new systems (especially in a large datacenter) information about the server can be used to configure the network environment (DHCP, Firewall, etc) before the server is ever even powered on. This does mean that you may have to update your network environment configuration if you replace a NIC (con), but that you won't have to update your network environment configuration if you re-install your server (pro), which is what you'd have to do now with `MACAddressPolicy=persistent`.
Yep. This is *the* case.
Cons:
- Deviate from upstream systemd.
It is also important to mention that Fedora will "deviate" from itself (it's former self). We would be changing a default in place since ~2013 [1].
[1] https://github.com/systemd/systemd/commit/16b9b87aee
It is desirable that RHEL and Fedora behaves similar. A possible outcome could be the current behavior stays and RHEL 10 would change behavior. On the other hand, different distributions (or even Fedora spins) have different uses and needs. Deviating might be fine. In the same vein, there is also a desire to stay close to upstream systemd behavior. But the uses of systemd project go beyond Fedora/RHEL, so deviating here may also be fine.
So: - Variant 1 is not good, variant 2 makes more sense. - The motivating case for v.2 is the "big datacenter" case and race when 'ip' is invoked. - We could improve the tools: 'ip link' could be taught to wait until udev has stopped processing the device, users can be taught to use better invocations. - For "small" users (individual machine admins who just install some server and configure it after turning it on and/or swap network cards between machines), having stable MAC addresses that doesn't depend on the order of device discovery or removal of a single network card seems better. - For "big" users (the datacenter case), changing the policy make sense, but at the same time, those folks can just insert a policy override, they're most likely using some ansible/puppet/cheffy thingy. - RHEL is more of the "big" case, Fedora is more the "small" case. Sometimes RHEL and Fedora have different defaults, sometimes RHEL takes years to follow what Fedora does.
So overall, I think this proposal would make most sense when limited to specific types of hardware (bridge and bond?), and also to specific spins: it's a better fit for CoreOS than Workstation…
But I haven't seen discussion of other approaches: would making 'ip' better be an option? 'udev' already locks some devices when processing using a simple BSD lock. Should we do the same for network devices and just teach 'ip' to do the same? (This would be much simpler and maybe more appealing that teaching it to wait for udev to report that it's finished).
Zbyszek
On Mon, Jun 27, 2022 at 10:10:31AM +0200, Zbigniew Jędrzejewski-Szmek wrote:
On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
Hi!
I already participated in the upstream discussion, so what I write here will be a restatement to some extent, but with a look from the side of Fedora specifically.
The proposal has two variants: 1. just changing the policy to MACAddressPolicy=none or 2. limiting the change to bridge and bond devices.
Re variant 1: MACAddressPolicy=persistent applies to all devices that don't have a hardware address. The proposal as written (blanket MACAddressPolicy=none) would change behaviour for all kinds of devices, incl. e.g. software devices like veths, and cheap hardware devices without a fixed MAC. The proposal doesn't provide any justification for this (except for simplicity of implementation) and this variant seems pretty bad and I'm strongly opposed.
Re variant 2: the proposal limited to brige/bond devices seems much more reasonable. In particular, the case described below of a server (virtualized or not) in a big datacenter is the one case where the benefits of MACAddressPolicy=none are clearly visible. I still don't think it's worth changing the default, but here the cost:benefit ratio is much closer.
== Benefit to Fedora ==
Pros:
Consistent behavior with RHEL8 and RHEL9.
Avoid race of udev and the tool that creates the interface.
The race will happen if the creation is done in a specific way. But at the same time, even the Change proposal describes how to avoid the race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the situation can be summarized as "we have a bunch of 'big' tools that create devices like NetworkManager or systemd-networkd, which already know or can be easily fixed to avoid the race, and a manual tool which can be invoked in a way that avoids the race". Instead of changing the default in udev we could educate people how to invoke it better.
This comes across as blaming every networking tool out there for having their previously correct & working behaviour be broken by a systemd change imposing new requirements on them.
Are there any notable tools which actually follow the usage pattern described, or are we in effect having to fix *everything*, including an uncountable amount of documentation, blogs, examples, most of which will prove impossible to fix in practice.
- Bridge and bond interfaces can get the MAC addresses from their first port.
In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will get a MAC related to one of its physical NIC devices. In the case of provisioning new systems (especially in a large datacenter) information about the server can be used to configure the network environment (DHCP, Firewall, etc) before the server is ever even powered on. This does mean that you may have to update your network environment configuration if you replace a NIC (con), but that you won't have to update your network environment configuration if you re-install your server (pro), which is what you'd have to do now with `MACAddressPolicy=persistent`.
Yep. This is *the* case.
Having the bridge/bond get the MAC addr of tht first physical NIC is pretty compelling from an automation POV, especially when the virtualziation host is doing traffic filtering for the VM. When a mgmt app assigns a MAC & IP pair to a guest, it is not uncommon to apply firewall rules providing anti-spoofing protection for MAC/IP, such that the VM cannot send traffic with other MAC/IP addresses. Libvirt provides this functionality with its 'clean-traffic' network filter, and other virt/cloud mgmt apps do similar.
Cons:
- Deviate from upstream systemd.
This is disappointing, as a great benefit of systemd is the level of consistency it brought to distro behaviour, and I think this change would be useful to all distros (in the context of bridge/bond devices)
It is also important to mention that Fedora will "deviate" from itself (it's former self). We would be changing a default in place since ~2013 [1].
Yes, that is quite unfortunate.
It is desirable that RHEL and Fedora behaves similar. A possible outcome could be the current behavior stays and RHEL 10 would change behavior. On the other hand, different distributions (or even Fedora spins) have different uses and needs. Deviating might be fine. In the same vein, there is also a desire to stay close to upstream systemd behavior. But the uses of systemd project go beyond Fedora/RHEL, so deviating here may also be fine.
So:
- Variant 1 is not good, variant 2 makes more sense.
- The motivating case for v.2 is the "big datacenter" case and race when 'ip' is invoked.
IMHO The more compelling reason is compliance with MAC address filtering applied to VMs.
- We could improve the tools: 'ip link' could be taught to wait until udev has stopped processing the device, users can be taught to use better invocations.
Teaching users is a impractical solution IMHO. The 'ip' tool already has poor usability/learnability IME, such that getting users to learn differences is pretty challenging, plus there's way too much docs out there in the wild that will never be updatable.
- For "small" users (individual machine admins who just install some server and configure it after turning it on and/or swap network cards between machines), having stable MAC addresses that doesn't depend on the order of device discovery or removal of a single network card seems better.
- For "big" users (the datacenter case), changing the policy make sense, but at the same time, those folks can just insert a policy override, they're most likely using some ansible/puppet/cheffy thingy.
- RHEL is more of the "big" case, Fedora is more the "small" case. Sometimes RHEL and Fedora have different defaults, sometimes RHEL takes years to follow what Fedora does.
So overall, I think this proposal would make most sense when limited to specific types of hardware (bridge and bond?), and also to specific spins: it's a better fit for CoreOS than Workstation…
This doesn't feel like a spin-specific problem to me, given it appears useful to both host & guest scenarios alike, and nested VM means host+guest are often the same thing.
But I haven't seen discussion of other approaches: would making 'ip' better be an option? 'udev' already locks some devices when processing using a simple BSD lock. Should we do the same for network devices and just teach 'ip' to do the same? (This would be much simpler and maybe more appealing that teaching it to wait for udev to report that it's finished).
With regards, Daniel
On Mon, Jun 27, 2022 at 09:40:53AM +0100, Daniel P. Berrangé wrote:
On Mon, Jun 27, 2022 at 10:10:31AM +0200, Zbigniew Jędrzejewski-Szmek wrote:
On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
Hi!
I already participated in the upstream discussion, so what I write here will be a restatement to some extent, but with a look from the side of Fedora specifically.
The proposal has two variants: 1. just changing the policy to MACAddressPolicy=none or 2. limiting the change to bridge and bond devices.
Re variant 1: MACAddressPolicy=persistent applies to all devices that don't have a hardware address. The proposal as written (blanket MACAddressPolicy=none) would change behaviour for all kinds of devices, incl. e.g. software devices like veths, and cheap hardware devices without a fixed MAC. The proposal doesn't provide any justification for this (except for simplicity of implementation) and this variant seems pretty bad and I'm strongly opposed.
Re variant 2: the proposal limited to brige/bond devices seems much more reasonable. In particular, the case described below of a server (virtualized or not) in a big datacenter is the one case where the benefits of MACAddressPolicy=none are clearly visible. I still don't think it's worth changing the default, but here the cost:benefit ratio is much closer.
== Benefit to Fedora ==
Pros:
Consistent behavior with RHEL8 and RHEL9.
Avoid race of udev and the tool that creates the interface.
The race will happen if the creation is done in a specific way. But at the same time, even the Change proposal describes how to avoid the race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the situation can be summarized as "we have a bunch of 'big' tools that create devices like NetworkManager or systemd-networkd, which already know or can be easily fixed to avoid the race, and a manual tool which can be invoked in a way that avoids the race". Instead of changing the default in udev we could educate people how to invoke it better.
This comes across as blaming every networking tool out there for having their previously correct & working behaviour be broken by a systemd change imposing new requirements on them.
I was a bit sloppy in my phrasing. NetworkManager and systemd-networkd already do the right thing. 'ip link' is the odd one out. Various other tools (e.g. netplan or Debian's network scripts) don't implement this internally, but call e.g. NetworkManager or 'ip'. So if we changed 'ip', this would go a long way to solving the problem (I'm may be wrong on some details here, please correct me if necessary.)
I'm not "blaming" the tools, I completely understand that they were written a long time ago. But in fact the issue is fairly generic: any software which interacts with devices that udev also touches MUST wait for udev to be done with the device. It's not just the MAC address policy, but also other rules that users may configure locally, sysctl configuration, etc. Without synchronization, one runs into races and errors from the tools when they try to configure things in parallel.
Are there any notable tools which actually follow the usage pattern described, or are we in effect having to fix *everything*, including an uncountable amount of documentation, blogs, examples, most of which will prove impossible to fix in practice.
- Bridge and bond interfaces can get the MAC addresses from their first port.
In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will get a MAC related to one of its physical NIC devices. In the case of provisioning new systems (especially in a large datacenter) information about the server can be used to configure the network environment (DHCP, Firewall, etc) before the server is ever even powered on. This does mean that you may have to update your network environment configuration if you replace a NIC (con), but that you won't have to update your network environment configuration if you re-install your server (pro), which is what you'd have to do now with `MACAddressPolicy=persistent`.
Yep. This is *the* case.
Having the bridge/bond get the MAC addr of tht first physical NIC is pretty compelling from an automation POV, especially when the virtualziation host is doing traffic filtering for the VM. When a mgmt app assigns a MAC & IP pair to a guest, it is not uncommon to apply firewall rules providing anti-spoofing protection for MAC/IP, such that the VM cannot send traffic with other MAC/IP addresses. Libvirt provides this functionality with its 'clean-traffic' network filter, and other virt/cloud mgmt apps do similar.
Cons:
- Deviate from upstream systemd.
This is disappointing, as a great benefit of systemd is the level of consistency it brought to distro behaviour, and I think this change would be useful to all distros (in the context of bridge/bond devices)
That is a good point. If this is done, I too would prefer to change it upstream rather than just downstream in Fedora.
It is also important to mention that Fedora will "deviate" from itself (it's former self). We would be changing a default in place since ~2013 [1].
Yes, that is quite unfortunate.
It is desirable that RHEL and Fedora behaves similar. A possible outcome could be the current behavior stays and RHEL 10 would change behavior. On the other hand, different distributions (or even Fedora spins) have different uses and needs. Deviating might be fine. In the same vein, there is also a desire to stay close to upstream systemd behavior. But the uses of systemd project go beyond Fedora/RHEL, so deviating here may also be fine.
So:
- Variant 1 is not good, variant 2 makes more sense.
- The motivating case for v.2 is the "big datacenter" case and race when 'ip' is invoked.
IMHO The more compelling reason is compliance with MAC address filtering applied to VMs.
FWIW, conditionalizing the policy on ConditionVirtualization=!vm would be trivial to add…
- We could improve the tools: 'ip link' could be taught to wait until udev has stopped processing the device, users can be taught to use better invocations.
Teaching users is a impractical solution IMHO. The 'ip' tool already has poor usability/learnability IME, such that getting users to learn differences is pretty challenging, plus there's way too much docs out there in the wild that will never be updatable.
- For "small" users (individual machine admins who just install some server and configure it after turning it on and/or swap network cards between machines), having stable MAC addresses that doesn't depend on the order of device discovery or removal of a single network card seems better.
- For "big" users (the datacenter case), changing the policy make sense, but at the same time, those folks can just insert a policy override, they're most likely using some ansible/puppet/cheffy thingy.
- RHEL is more of the "big" case, Fedora is more the "small" case. Sometimes RHEL and Fedora have different defaults, sometimes RHEL takes years to follow what Fedora does.
So overall, I think this proposal would make most sense when limited to specific types of hardware (bridge and bond?), and also to specific spins: it's a better fit for CoreOS than Workstation…
This doesn't feel like a spin-specific problem to me, given it appears useful to both host & guest scenarios alike, and nested VM means host+guest are often the same thing.
But I haven't seen discussion of other approaches: would making 'ip' better be an option? 'udev' already locks some devices when processing using a simple BSD lock. Should we do the same for network devices and just teach 'ip' to do the same? (This would be much simpler and maybe more appealing that teaching it to wait for udev to report that it's finished).
Zbyszek
On 6/27/22 07:34, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Jun 27, 2022 at 09:40:53AM +0100, Daniel P. Berrangé wrote:
On Mon, Jun 27, 2022 at 10:10:31AM +0200, Zbigniew Jędrzejewski-Szmek wrote:
On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
Hi!
I already participated in the upstream discussion, so what I write here will be a restatement to some extent, but with a look from the side of Fedora specifically.
The proposal has two variants: 1. just changing the policy to MACAddressPolicy=none or 2. limiting the change to bridge and bond devices.
Re variant 1: MACAddressPolicy=persistent applies to all devices that don't have a hardware address. The proposal as written (blanket MACAddressPolicy=none) would change behaviour for all kinds of devices, incl. e.g. software devices like veths, and cheap hardware devices without a fixed MAC. The proposal doesn't provide any justification for this (except for simplicity of implementation) and this variant seems pretty bad and I'm strongly opposed.
Re variant 2: the proposal limited to brige/bond devices seems much more reasonable. In particular, the case described below of a server (virtualized or not) in a big datacenter is the one case where the benefits of MACAddressPolicy=none are clearly visible. I still don't think it's worth changing the default, but here the cost:benefit ratio is much closer.
== Benefit to Fedora ==
Pros:
Consistent behavior with RHEL8 and RHEL9.
Avoid race of udev and the tool that creates the interface.
The race will happen if the creation is done in a specific way. But at the same time, even the Change proposal describes how to avoid the race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the situation can be summarized as "we have a bunch of 'big' tools that create devices like NetworkManager or systemd-networkd, which already know or can be easily fixed to avoid the race, and a manual tool which can be invoked in a way that avoids the race". Instead of changing the default in udev we could educate people how to invoke it better.
This comes across as blaming every networking tool out there for having their previously correct & working behaviour be broken by a systemd change imposing new requirements on them.
I was a bit sloppy in my phrasing. NetworkManager and systemd-networkd already do the right thing. 'ip link' is the odd one out. Various other tools (e.g. netplan or Debian's network scripts) don't implement this internally, but call e.g. NetworkManager or 'ip'. So if we changed 'ip', this would go a long way to solving the problem (I'm may be wrong on some details here, please correct me if necessary.)
I'm not "blaming" the tools, I completely understand that they were written a long time ago. But in fact the issue is fairly generic: any software which interacts with devices that udev also touches MUST wait for udev to be done with the device. It's not just the MAC address policy, but also other rules that users may configure locally, sysctl configuration, etc. Without synchronization, one runs into races and errors from the tools when they try to configure things in parallel.
Are there any notable tools which actually follow the usage pattern described, or are we in effect having to fix *everything*, including an uncountable amount of documentation, blogs, examples, most of which will prove impossible to fix in practice.
- Bridge and bond interfaces can get the MAC addresses from their first port.
In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will get a MAC related to one of its physical NIC devices. In the case of provisioning new systems (especially in a large datacenter) information about the server can be used to configure the network environment (DHCP, Firewall, etc) before the server is ever even powered on. This does mean that you may have to update your network environment configuration if you replace a NIC (con), but that you won't have to update your network environment configuration if you re-install your server (pro), which is what you'd have to do now with `MACAddressPolicy=persistent`.
Yep. This is *the* case.
Having the bridge/bond get the MAC addr of tht first physical NIC is pretty compelling from an automation POV, especially when the virtualziation host is doing traffic filtering for the VM. When a mgmt app assigns a MAC & IP pair to a guest, it is not uncommon to apply firewall rules providing anti-spoofing protection for MAC/IP, such that the VM cannot send traffic with other MAC/IP addresses. Libvirt provides this functionality with its 'clean-traffic' network filter, and other virt/cloud mgmt apps do similar.
Cons:
- Deviate from upstream systemd.
This is disappointing, as a great benefit of systemd is the level of consistency it brought to distro behaviour, and I think this change would be useful to all distros (in the context of bridge/bond devices)
That is a good point. If this is done, I too would prefer to change it upstream rather than just downstream in Fedora.
Big +1 from me on getting upstream to change so that we are in alignment. I don't know if it was made clear in the upstream discussion, but maybe upstream would be more accepting for just changing the policy for bond/bridge/team devices.
Similarly, we might be able to get the next RHEL to adopt the approach of MACAddressPolicy=persistent for non bond/bridge/team and MACAddressPolicy=none for bond/bridge/team. IOW we may be able to get upstream systemd, Fedora, and downstream (RHEL/CentOS,Alma,Rocky) to all be in alignment.
Dusty
On Mon, 2022-06-27 at 13:34 +0200, Zbigniew Jędrzejewski-Szmek wrote:
I'm not "blaming" the tools, I completely understand that they were written a long time ago. But in fact the issue is fairly generic: any software which interacts with devices that udev also touches MUST wait for udev to be done with the device. It's not just the MAC address policy, but also other rules that users may configure locally, sysctl configuration, etc. Without synchronization, one runs into races and errors from the tools when they try to configure things in parallel.
Yes, any software is well advised to wait for udev.
But if it were not for MACAddressPolicy=, default Fedora would not ship much of relevant to make that necessary in practice.
For example, NetworkManager's udev rules set some attributes. But those are mainly for NetworkManager itself (and NetworkManager knows to wait). If you have a tool that creates a software device, then there is no problem to not wait for those.
Also, "other rules that the user confiugres locally", are entirely user's choice. Of course, the tools that this user runs, need to interoperate with the user's environment.
IMHO The more compelling reason is compliance with MAC address filtering applied to VMs.
FWIW, conditionalizing the policy on ConditionVirtualization=!vm would be trivial to add…
Maybe having ConditionKernelCommandLine= could allow convenient customization for users and could be set by CoreOS (or suggested/documented).
Thomas
Hi,
On 6/27/22 10:10, Zbigniew Jędrzejewski-Szmek wrote:
On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
Hi!
I already participated in the upstream discussion, so what I write here will be a restatement to some extent, but with a look from the side of Fedora specifically.
The proposal has two variants: 1. just changing the policy to MACAddressPolicy=none or 2. limiting the change to bridge and bond devices.
Re variant 1: MACAddressPolicy=persistent applies to all devices that don't have a hardware address. The proposal as written (blanket MACAddressPolicy=none) would change behaviour for all kinds of devices, incl. e.g. software devices like veths, and cheap hardware devices without a fixed MAC. The proposal doesn't provide any justification for this (except for simplicity of implementation) and this variant seems pretty bad and I'm strongly opposed.
<a bit offtopic, sorry>
I did not know systemd/udev can save/restore MAC addresses for devices where there is no MAC address. I have looking into a related issue on my todo list. There are some cheap x86 devices with wifi which don't have a nvram (eeprom) chip instead they load nvram from /lib/firmware. On most devices there is enough otp inside the wifi chip that they do have a unique MAC inside the wifi chip's otp memory so everything works well.
But on some devices that otp is either not there or not programmed, so they take their MAC from the nvram in /lib/firmware which means that all devices from the same model end up using the same MAC (which is defined in the nvram in /lib/firmware).
Am I reading the above correctly that udev/systemd already is able to deal with this and I just need to make it so that the device has no MAC set at all and then the first time systemd sees this it will generate a random MAC and use that on future boots ?
Any docs on this / code you can point me to how systemd determines it needs to do this ?
<even further offtopic>
The same also applies to some bluetooth devices:
https://github.com/bluez/bluez/issues/107
I wonder if similar functionality for bluetooth would be welcome in systemd ?
Regards,
Hans
Re variant 2: the proposal limited to brige/bond devices seems much more reasonable. In particular, the case described below of a server (virtualized or not) in a big datacenter is the one case where the benefits of MACAddressPolicy=none are clearly visible. I still don't think it's worth changing the default, but here the cost:benefit ratio is much closer.
== Benefit to Fedora ==
Pros:
Consistent behavior with RHEL8 and RHEL9.
Avoid race of udev and the tool that creates the interface.
The race will happen if the creation is done in a specific way. But at the same time, even the Change proposal describes how to avoid the race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the situation can be summarized as "we have a bunch of 'big' tools that create devices like NetworkManager or systemd-networkd, which already know or can be easily fixed to avoid the race, and a manual tool which can be invoked in a way that avoids the race". Instead of changing the default in udev we could educate people how to invoke it better.
- Bridge and bond interfaces can get the MAC addresses from their first port.
In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will get a MAC related to one of its physical NIC devices. In the case of provisioning new systems (especially in a large datacenter) information about the server can be used to configure the network environment (DHCP, Firewall, etc) before the server is ever even powered on. This does mean that you may have to update your network environment configuration if you replace a NIC (con), but that you won't have to update your network environment configuration if you re-install your server (pro), which is what you'd have to do now with `MACAddressPolicy=persistent`.
Yep. This is *the* case.
Cons:
- Deviate from upstream systemd.
It is also important to mention that Fedora will "deviate" from itself (it's former self). We would be changing a default in place since ~2013 [1].
[1] https://github.com/systemd/systemd/commit/16b9b87aee
It is desirable that RHEL and Fedora behaves similar. A possible outcome could be the current behavior stays and RHEL 10 would change behavior. On the other hand, different distributions (or even Fedora spins) have different uses and needs. Deviating might be fine. In the same vein, there is also a desire to stay close to upstream systemd behavior. But the uses of systemd project go beyond Fedora/RHEL, so deviating here may also be fine.
So:
- Variant 1 is not good, variant 2 makes more sense.
- The motivating case for v.2 is the "big datacenter" case and race when 'ip' is invoked.
- We could improve the tools: 'ip link' could be taught to wait until udev has stopped processing the device, users can be taught to use better invocations.
- For "small" users (individual machine admins who just install some server and configure it after turning it on and/or swap network cards between machines), having stable MAC addresses that doesn't depend on the order of device discovery or removal of a single network card seems better.
- For "big" users (the datacenter case), changing the policy make sense, but at the same time, those folks can just insert a policy override, they're most likely using some ansible/puppet/cheffy thingy.
- RHEL is more of the "big" case, Fedora is more the "small" case. Sometimes RHEL and Fedora have different defaults, sometimes RHEL takes years to follow what Fedora does.
So overall, I think this proposal would make most sense when limited to specific types of hardware (bridge and bond?), and also to specific spins: it's a better fit for CoreOS than Workstation…
But I haven't seen discussion of other approaches: would making 'ip' better be an option? 'udev' already locks some devices when processing using a simple BSD lock. Should we do the same for network devices and just teach 'ip' to do the same? (This would be much simpler and maybe more appealing that teaching it to wait for udev to report that it's finished).
Zbyszek _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Mon, Jun 27, 2022 at 11:20:13AM +0200, Hans de Goede wrote:
Hi,
On 6/27/22 10:10, Zbigniew Jędrzejewski-Szmek wrote:
On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
Hi!
I already participated in the upstream discussion, so what I write here will be a restatement to some extent, but with a look from the side of Fedora specifically.
The proposal has two variants: 1. just changing the policy to MACAddressPolicy=none or 2. limiting the change to bridge and bond devices.
Re variant 1: MACAddressPolicy=persistent applies to all devices that don't have a hardware address. The proposal as written (blanket MACAddressPolicy=none) would change behaviour for all kinds of devices, incl. e.g. software devices like veths, and cheap hardware devices without a fixed MAC. The proposal doesn't provide any justification for this (except for simplicity of implementation) and this variant seems pretty bad and I'm strongly opposed.
<a bit offtopic, sorry>
I did not know systemd/udev can save/restore MAC addresses for devices where there is no MAC address. I have looking into a related issue on my todo list. There are some cheap x86 devices with wifi which don't have a nvram (eeprom) chip instead they load nvram from /lib/firmware. On most devices there is enough otp inside the wifi chip that they do have a unique MAC inside the wifi chip's otp memory so everything works well.
But on some devices that otp is either not there or not programmed, so they take their MAC from the nvram in /lib/firmware which means that all devices from the same model end up using the same MAC (which is defined in the nvram in /lib/firmware).
Am I reading the above correctly that udev/systemd already is able to deal with this and I just need to make it so that the device has no MAC set at all and then the first time systemd sees this it will generate a random MAC and use that on future boots ?
Yes. (With the following clarification: the MAC address is generated from as some hash of machine-id + device name. So "first time" and "any time" are the same — no state is ever saved, you're supposed to get the same result for a specific device on a specific system.)
Any docs on this / code you can point me to how systemd determines it needs to do this ?
udev looks at /sys/class/net/hub0/addr_assign_type. But the docs are not great :( The description is split between two man pages: - https://www.freedesktop.org/software/systemd/man/systemd.link.html#MACAddres... - https://www.freedesktop.org/software/systemd/man/systemd.net-naming-scheme.h...
<even further offtopic>
The same also applies to some bluetooth devices:
https://github.com/bluez/bluez/issues/107
I wonder if similar functionality for bluetooth would be welcome in systemd ?
I think so. From what I can see, we don't do much setup for bluetooth devices, so I'm not sure if udevd is a better place for this than e.g. bluez. But if yes, it's nice to do things uniformly, i.e. in this case treat bluebooth interfaces more like other network interfaces… The implementation should be simple, since it'd be a question of extending existing code to cover one more case.
Zbyszek
Hi,
On 6/27/22 13:18, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Jun 27, 2022 at 11:20:13AM +0200, Hans de Goede wrote:
Hi,
On 6/27/22 10:10, Zbigniew Jędrzejewski-Szmek wrote:
On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
Hi!
I already participated in the upstream discussion, so what I write here will be a restatement to some extent, but with a look from the side of Fedora specifically.
The proposal has two variants: 1. just changing the policy to MACAddressPolicy=none or 2. limiting the change to bridge and bond devices.
Re variant 1: MACAddressPolicy=persistent applies to all devices that don't have a hardware address. The proposal as written (blanket MACAddressPolicy=none) would change behaviour for all kinds of devices, incl. e.g. software devices like veths, and cheap hardware devices without a fixed MAC. The proposal doesn't provide any justification for this (except for simplicity of implementation) and this variant seems pretty bad and I'm strongly opposed.
<a bit offtopic, sorry>
I did not know systemd/udev can save/restore MAC addresses for devices where there is no MAC address. I have looking into a related issue on my todo list. There are some cheap x86 devices with wifi which don't have a nvram (eeprom) chip instead they load nvram from /lib/firmware. On most devices there is enough otp inside the wifi chip that they do have a unique MAC inside the wifi chip's otp memory so everything works well.
But on some devices that otp is either not there or not programmed, so they take their MAC from the nvram in /lib/firmware which means that all devices from the same model end up using the same MAC (which is defined in the nvram in /lib/firmware).
Am I reading the above correctly that udev/systemd already is able to deal with this and I just need to make it so that the device has no MAC set at all and then the first time systemd sees this it will generate a random MAC and use that on future boots ?
Yes. (With the following clarification: the MAC address is generated from as some hash of machine-id + device name. So "first time" and "any time" are the same — no state is ever saved, you're supposed to get the same result for a specific device on a specific system.)
Any docs on this / code you can point me to how systemd determines it needs to do this ?
udev looks at /sys/class/net/hub0/addr_assign_type. But the docs are not great :( The description is split between two man pages:
Thanks this was useful. I've submitted a kernel patch upstream now which fixes the described issue by using a random MAC + setting the addr_assign_type:
https://lore.kernel.org/linux-wireless/20220708133712.102179-2-hdegoede@redh...
Regards,
Hans
<even further offtopic>
The same also applies to some bluetooth devices:
https://github.com/bluez/bluez/issues/107
I wonder if similar functionality for bluetooth would be welcome in systemd ?
I think so. From what I can see, we don't do much setup for bluetooth devices, so I'm not sure if udevd is a better place for this than e.g. bluez. But if yes, it's nice to do things uniformly, i.e. in this case treat bluebooth interfaces more like other network interfaces… The implementation should be simple, since it'd be a question of extending existing code to cover one more case.
Zbyszek
On 6/27/22 04:10, Zbigniew Jędrzejewski-Szmek wrote:
Re variant 2: the proposal limited to brige/bond devices seems much more reasonable. In particular, the case described below of a server (virtualized or not) in a big datacenter is the one case where the benefits of MACAddressPolicy=none are clearly visible. I still don't think it's worth changing the default, but here the cost:benefit ratio is much closer.
I think that's reasonable. The proposal owners are willing to make this change to the proposal so that it only applies to bond/bridge/team devices if that is more acceptable.
On Mon, 2022-06-27 at 10:10 +0200, Zbigniew Jędrzejewski-Szmek wrote:
- Deviate from upstream systemd.
It is also important to mention that Fedora will "deviate" from itself (it's former self). We would be changing a default in place since ~2013 [1].
Hi,
Is that the case? I think in practice, it only changed in F31/2019 with systemd 242: https://github.com/systemd/systemd/blob/eaee50fee2d0baec9116ae76945f18fef4b8...
Granted, 2019 is also a long time...
Thomas
On Mon, Jun 27, 2022 at 06:08:13PM +0200, Thomas Haller wrote:
On Mon, 2022-06-27 at 10:10 +0200, Zbigniew Jędrzejewski-Szmek wrote:
- Deviate from upstream systemd.
It is also important to mention that Fedora will "deviate" from itself (it's former self). We would be changing a default in place since ~2013 [1].
Hi,
Is that the case? I think in practice, it only changed in F31/2019 with systemd 242: https://github.com/systemd/systemd/blob/eaee50fee2d0baec9116ae76945f18fef4b8...
Granted, 2019 is also a long time...
2013 — MACAddressPolicy=persistent mostly for hardware devices 2019 — MACAddressPolicy=persistent for bridge and bond and some other stuff
Zbyszek
On Mon, 2022-06-27 at 10:10 +0200, Zbigniew Jędrzejewski-Szmek wrote:
On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
Hi!
I already participated in the upstream discussion, so what I write here will be a restatement to some extent, but with a look from the side of Fedora specifically.
The proposal has two variants: 1. just changing the policy to MACAddressPolicy=none or 2. limiting the change to bridge and bond devices.
Re variant 1: MACAddressPolicy=persistent applies to all devices that don't have a hardware address. The proposal as written (blanket MACAddressPolicy=none) would change behaviour for all kinds of devices, incl. e.g. software devices like veths, and cheap hardware devices without a fixed MAC.
the proposal was only about software devices.
The difference is that software devices are explicitly added by some tool (if we ignore module parameters like "max_bonds" -- which on fedora/systemd is already 0 by default).
A hardware device gets automatically added when loading the kernel. More importantly, udev already does renaming, so any user is well advised to wait for udev to settles. At which point, they can also wait for the MAC address policy.
The proposal doesn't provide any justification for this (except for simplicity of implementation) and this variant seems pretty bad and I'm strongly opposed.
The first point: the race. And that udev touches interfaces created by somebody, without being explicitly told to do so.
Yes, udev always does something to the device, like attaching attributes such as ID_NET_NAMING_SCHEME. But aside the MAC address policy, it doesn't do anything relevant where a race would matter.
Re variant 2: the proposal limited to brige/bond devices seems much more reasonable. In particular, the case described below of a server (virtualized or not) in a big datacenter is the one case where the benefits of MACAddressPolicy=none are clearly visible. I still don't think it's worth changing the default, but here the cost:benefit ratio is much closer.
== Benefit to Fedora ==
Pros:
Consistent behavior with RHEL8 and RHEL9.
Avoid race of udev and the tool that creates the interface.
The race will happen if the creation is done in a specific way. But at the same time, even the Change proposal describes how to avoid the race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the situation can be summarized as "we have a bunch of 'big' tools that create devices like NetworkManager or systemd-networkd, which already know or can be easily fixed to avoid the race, and a manual tool which can be invoked in a way that avoids the race". Instead of changing the default in udev we could educate people how to invoke it better.
I don't think that big projects matter much. They can learn this. For example, docker had a problem with this ([1]), which presumably got fixed long ago.
[1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_request...
The problem are smaller projects and user scripts/tools.
For example, NetworkManager-ci contains hundreds of scripts for testing, and -- while we should know better -- we don't wait for udev after creating a veth interface for testing. That is no problem for us, we just set MACAddressPolicy=none. But it makes me wonder, just how many "naive" scripts are out there that don't get this right.
You say, "the race will happen if the creation is done in a specific way". The problem is that the specific way is the most straight forward one.
Also, to educate people how to use iproute2 (?) better, seems not clearly best. Is there really a problem on the user's side when they are creating an interface and don't expect udev to interfere?
- Bridge and bond interfaces can get the MAC addresses from their
first port.
In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will get a MAC related to one of its physical NIC devices. In the case of provisioning new systems (especially in a large datacenter) information about the server can be used to configure the network environment (DHCP, Firewall, etc) before the server is ever even powered on. This does mean that you may have to update your network environment configuration if you replace a NIC (con), but that you won't have to update your network environment configuration if you re-install your server (pro), which is what you'd have to do now with `MACAddressPolicy=persistent`.
Yep. This is *the* case.
Cons:
- Deviate from upstream systemd.
It is also important to mention that Fedora will "deviate" from itself (it's former self). We would be changing a default in place since ~2013 [1].
[1] https://github.com/systemd/systemd/commit/16b9b87aee
It is desirable that RHEL and Fedora behaves similar. A possible outcome could be the current behavior stays and RHEL 10 would change behavior. On the other hand, different distributions (or even Fedora spins) have different uses and needs. Deviating might be fine. In the same vein, there is also a desire to stay close to upstream systemd behavior. But the uses of systemd project go beyond Fedora/RHEL, so deviating here may also be fine.
So:
- Variant 1 is not good, variant 2 makes more sense.
- The motivating case for v.2 is the "big datacenter" case and race
when 'ip' is invoked.
- We could improve the tools: 'ip link' could be taught to wait until
udev has stopped processing the device, users can be taught to use better invocations.
see below.
- For "small" users (individual machine admins who just install some
server and configure it after turning it on and/or swap network cards between machines), having stable MAC addresses that doesn't depend on the order of device discovery or removal of a single network card seems better.
That claim is most relevant, but seems not well justified. It comes from a desire for order, where things are stable.
I am not so convinced though, given that the solution requires to involve udev. A counter point would be that many users might not care for this feature, and those who do, can enable it just fine.
- For "big" users (the datacenter case), changing the policy make
sense, but at the same time, those folks can just insert a policy override, they're most likely using some ansible/puppet/cheffy thingy.
- RHEL is more of the "big" case, Fedora is more the "small" case.
Sometimes RHEL and Fedora have different defaults, sometimes RHEL takes years to follow what Fedora does.
I personally care mostly about RHEL here and to find out what to do. So if we agree that RHEL is allowed to have a different downstream behavior than Fedora, I'd be satisfied!
So overall, I think this proposal would make most sense when limited to specific types of hardware (bridge and bond?), and also to specific spins: it's a better fit for CoreOS than Workstation…
Makes sense.
I think CoreOS would be tempted to set MACAddressPolicy=none. Thus, a major Fedora user would go against the distro default. That raises a question about why Workstation behaves different. But if CoreOS (or other spins) are allowed to do that, I think Dusty would also be satisfied.
But I haven't seen discussion of other approaches: would making 'ip' better be an option? 'udev' already locks some devices when processing using a simple BSD lock. Should we do the same for network devices and just teach 'ip' to do the same? (This would be much simpler and maybe more appealing that teaching it to wait for udev to report that it's finished).
iproute2 is the obvious candidate, but there can be any tools that use the kernel API some other way (e.g. pyroute2). It wouldn't solve the problem for them making the "solution" less universal and attractive.
I would be surprised if iproute2 would accept patches to integrate with udev. It seems like a low-level tool, and I could understand if the answer is to use the tool correctly. Unfortunately, the correct way is not immediately obvious.
Thank you for all your feedback!!
Thomas
On 6/27/22 13:03, Thomas Haller wrote:
On Mon, 2022-06-27 at 10:10 +0200, Zbigniew Jędrzejewski-Szmek wrote:
- For "big" users (the datacenter case), changing the policy make
sense, but at the same time, those folks can just insert a policy override, they're most likely using some ansible/puppet/cheffy thingy.
- RHEL is more of the "big" case, Fedora is more the "small" case.
Sometimes RHEL and Fedora have different defaults, sometimes RHEL takes years to follow what Fedora does.
I personally care mostly about RHEL here and to find out what to do. So if we agree that RHEL is allowed to have a different downstream behavior than Fedora, I'd be satisfied!
RHEL and Fedora *can* deviate, but I think it's valuable for them to have alignment whenever possible. i.e. if they deviate the benefit should outweigh the cost. Optimally we find a default that works best in most cases that we can share.
So overall, I think this proposal would make most sense when limited to specific types of hardware (bridge and bond?), and also to specific spins: it's a better fit for CoreOS than Workstation…
Makes sense.
I think CoreOS would be tempted to set MACAddressPolicy=none. Thus, a major Fedora user would go against the distro default. That raises a question about why Workstation behaves different. But if CoreOS (or other spins) are allowed to do that, I think Dusty would also be satisfied.
While I agree it's nice to have the flexibility to deviate on just certain variants, I'd much prefer to have the policy set Fedora wide if we can. Every delta is a cost and our goal in Fedora CoreOS is to try to not deviate from Fedora unless it really really benefits the users. The more we deviate over time the more maintenance burden we accumulate.
If this change got rejected at the Fedora level I don't know if Fedora CoreOS would pick it up individually. From one perspective it wouldn't make sense: deviating from the rest of Fedora, but from another perspective it would make sense: deviating from our downstream of RHCOS (based on RHEL).
Again, I'd really prefer to find a compromise here.
On 6/27/22 13:03, Thomas Haller wrote:
On Mon, 2022-06-27 at 10:10 +0200, Zbigniew Jędrzejewski-Szmek wrote:
On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
Hi!
I already participated in the upstream discussion, so what I write here will be a restatement to some extent, but with a look from the side of Fedora specifically.
The proposal has two variants: 1. just changing the policy to MACAddressPolicy=none or 2. limiting the change to bridge and bond devices.
Re variant 1: MACAddressPolicy=persistent applies to all devices that don't have a hardware address. The proposal as written (blanket MACAddressPolicy=none) would change behaviour for all kinds of devices, incl. e.g. software devices like veths, and cheap hardware devices without a fixed MAC.
the proposal was only about software devices.
The difference is that software devices are explicitly added by some tool (if we ignore module parameters like "max_bonds" -- which on fedora/systemd is already 0 by default).
A hardware device gets automatically added when loading the kernel. More importantly, udev already does renaming, so any user is well advised to wait for udev to settles. At which point, they can also wait for the MAC address policy.
The proposal doesn't provide any justification for this (except for simplicity of implementation) and this variant seems pretty bad and I'm strongly opposed.
The first point: the race. And that udev touches interfaces created by somebody, without being explicitly told to do so.
Yes, udev always does something to the device, like attaching attributes such as ID_NET_NAMING_SCHEME. But aside the MAC address policy, it doesn't do anything relevant where a race would matter.
Re variant 2: the proposal limited to brige/bond devices seems much more reasonable. In particular, the case described below of a server (virtualized or not) in a big datacenter is the one case where the benefits of MACAddressPolicy=none are clearly visible. I still don't think it's worth changing the default, but here the cost:benefit ratio is much closer.
== Benefit to Fedora ==
Pros:
Consistent behavior with RHEL8 and RHEL9.
Avoid race of udev and the tool that creates the interface.
The race will happen if the creation is done in a specific way. But at the same time, even the Change proposal describes how to avoid the race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the situation can be summarized as "we have a bunch of 'big' tools that create devices like NetworkManager or systemd-networkd, which already know or can be easily fixed to avoid the race, and a manual tool which can be invoked in a way that avoids the race". Instead of changing the default in udev we could educate people how to invoke it better.
I don't think that big projects matter much. They can learn this. For example, docker had a problem with this ([1]), which presumably got fixed long ago.
[1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_request...
The problem are smaller projects and user scripts/tools.
For example, NetworkManager-ci contains hundreds of scripts for testing, and -- while we should know better -- we don't wait for udev after creating a veth interface for testing. That is no problem for us, we just set MACAddressPolicy=none. But it makes me wonder, just how many "naive" scripts are out there that don't get this right.
At least for block devices, waiting for udev to finish can actually be a non-trivial overhead, especially when udev has been explicitly told to *not* do anything with the devices in question. It has a measurable impact on how long LVM commands take to run, and LVM is known to be a significant contributor to the amount of time Qubes OS needs to start a VM.
On Sat, Jun 25, 2022 at 08:39:22PM +0000, devel-request@lists.fedoraproject.org wrote:
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
== Owner ==
- Name: [[User:thaller|Thomas Haller]] (NetworkManager)
- Email: thaller@redhat.com
- Name: [[User:dustymabe| Dusty Mabe]] (Fedora CoreOS)
- Email: dmabe@redhat.com
== Detailed Description ==
On Fedora, udev by default changes the MAC address of a wide range of software devices. This was introduced by systemd 242 in early 2019 (Fedora 31), when `MACAddressPolicy=` was extended to affect more types of devices.
With `MACAddressPolicy=persistent` udev's aim is to provide a stable MAC address, otherwise the kernel will assign a random one. However, that can cause problems:
Firstly, software devices are always created by some tool that has plans for the device. The tool may not expect that udev is going to change the MAC address and races against that. The best solution for the tool is to set the MAC address when creating an interface. This will prevent udev from changing the MAC address according to the MACAddressPolicy. Otherwise, the tool should wait for udev to initialize the device to avoid the race. In theory, a tool is always advised to wait for udev to initialize the device. However, if it were not for MACAddressPolicy, in common scenarios udev doesn't do anything relevant for software devices to make that necessary.
Secondly, for interface types bridge and bond, an unset MAC address has a special meaning to the kernel and the MAC address of the first port is used. If udev changes the MAC address, that no longer works. Now the generated MAC address is not directly discoverable as it is based on `/etc/machine-id` ([https://www.man7.org/linux/man-pages/man5/machine-id.5.html machine-id(5)]), among other data. Even if there were a tool to easily calculate the MAC address, it could be cumbersome to use it without logging into the machine first. The MAC address can directly affect the assigned IP address, for example when using DHCP. When booting a new virtual machine, the user might know the MAC address of the (virtual) "physical" interfaces. When bonding/bridging those interfaces, the bond/bridge would get one of the well known MAC addresses. `MACAddressPolicy=persistent` interferes with that.
The goal of persistent policy is to provide a stable MAC address. Note that if the tool or user who created the interface would want a certain MAC address, they have all the means to set it already. That applies regardless whether the tool is iproute2, NetworkManager, systemd-networkd. Neither NetworkManager nor systemd-networkd rely on udev's MACAddressPolicy for setting the MAC address. This behavior is mostly useful for plain `ip link add`, but it's unclear which real world user wants this behavior.
Of course, the user is welcome to configure the MAC address in any way they want. Including, dropping a link file that sets `MACAddressPolicy=persistent`. The problem is once udev sets a MAC address, it cannot be unset. Which makes this problematic to do by default.
While Fedora inherited this behavior from upstream systemd, RHEL-9 does not follow this behavior ([https://gitlab.com/redhat/centos-stream/rpms/systemd/-/blob/c8953519504bf2e6... centos9], [https://bugzilla.redhat.com/show_bug.cgi?id=1921094 rh#1921094]). For RHEL-8, this doesn't apply because the systemd there is too old to change the MAC address of most software devices.
This could be either implemented by patching `/usr/lib/systemd/network/99-default.link` to have a different policy, or by dropping a link file with higher priority. In the latter case, that override could be shipped either by udev or even by NetworkManager.
Another option is to change the scope of this proposal to only change to `MACAddressPolicy=none` for the device types where this causes the most issues (bridge/bond/team).
+1 (for very large values of 1 :)
"MACAddressPolicy=none" was the default, reliable, expected behavior for literally *decades* until it got broken around the time of F32 (systemd 242): https://bugzilla.redhat.com/show_bug.cgi?id=1834537
Unbreaking the behavior (at least for bridge/bond/team interfaces) is welcome -- better late than never!
Thanks much, --Gabriel
== Feedback ==
This was also discussed on upstream systemd mailing list [https://lists.freedesktop.org/archives/systemd-devel/2022-May/047893.html here]. The upstream systemd maintainers' opinion is that the current udev behavior is desirable, but accepts that distributions (or network stacks such as NetworkManager) may choose to change the default depending on their needs ([https://lists.freedesktop.org/archives/systemd-devel/2022-May/047943.html reference]). The goal here is to find out what the Fedora community thinks is the most appropriate default.
The RHEL-9 bug is [https://bugzilla.redhat.com/show_bug.cgi?id=1921094 [rh#1921094]].
== Benefit to Fedora ==
Pros:
Consistent behavior with RHEL8 and RHEL9.
Avoid race of udev and the tool that creates the interface.
Bridge and bond interfaces can get the MAC addresses from their first port.
In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will get a MAC related to one of its physical NIC devices. In the case of provisioning new systems (especially in a large datacenter) information about the server can be used to configure the network environment (DHCP, Firewall, etc) before the server is ever even powered on. This does mean that you may have to update your network environment configuration if you replace a NIC (con), but that you won't have to update your network environment configuration if you re-install your server (pro), which is what you'd have to do now with `MACAddressPolicy=persistent`.
Cons:
- Deviate from upstream systemd.
It is desirable that RHEL and Fedora behaves similar. A possible outcome could be the current behavior stays and RHEL 10 would change behavior. On the other hand, different distributions (or even Fedora spins) have different uses and needs. Deviating might be fine. In the same vein, there is also a desire to stay close to upstream systemd behavior. But the uses of systemd project go beyond Fedora/RHEL, so deviating here may also be fine.
== Scope ==
- Proposal owners:
The main goal of this request is to generate productive discussion and find the desired behavior. The implementation/changes are either way very simple.
- Other developers:
Other projects that wish a certain MAC address are welcome to set it for their devices. Including using udev's MACAddressPolicy.
- Release engineering:
Not needed for this change.
- Policies and guidelines: N/A (not needed for this Change)
- Trademark approval: N/A (not needed for this Change)
- Alignment with Objectives:
== Upgrade/compatibility impact ==
After the change, the MAC address for the affected device types changes.
== How To Test ==
- Create a software device two times, for example `ip link add type
bridge`. Note that the MAC address is either stable or random, depending on the `MACAddressPolicy=`.
- Note that if the software device has the MAC address set initially,
udev does not change it (`ip link add address aa:aa:aa:aa:aa:aa type bridge`). That depends on `/sys/class/net/$dev/addr_assign_type`.
- Create a bridge/bond interface without setting the MAC address.
Note that if `MACAddressPolicy=none`, the MAC address is random at first. Note that attaching the first port will update the controller's MAC address. On the other hand, with `MACAddressPolicy=persistent`, the MAC address of the controller is fixed and not inherited from the port.
- Run
ip monitor link & while : ; do ip link del xxx ip link add name xxx type dummy \ && ip link set xxx addr aa:00:00:00:00:00 \ && ip link show xxx | grep -q aa:00:00:00:00:00 \ || break done
to reproduce the race between a simple tool and udev changing the MAC address.
== User Experience ==
Bond/bridge devices would again get assigned the MAC address of the first NIC added to the device. If we chose to not limit the scope of this change to just bonds/bridges then all software devices would get randomly assigned MAC addresses.
== Dependencies ==
None.
== Contingency Plan ==
If the change is rejected, nothing needs to be done. The change itself will be simple to implement. Contingency deadline: beta freeze Blocks release? No
== Documentation ==
TODO.
== Release Notes ==
-- Vipul Siddharth He/His/Him FPgM team member
As I said before I don't care that much what the policy is but I do care very much that Fedora keeps changing it.
Twice now I have had to go and reconfigure my networks after a Fedora upgrade has changed the MAC assignment policy.
My vote is therefore to leave it as it is so that I don't have to go and change everything again!
Basically I don't care what the MAC of a machine is but I care very much if it changes because I have to update DHCP so that it can continue to issue the correct IPs and DNS for IPv6 addresses which change when the MAC changes.
Tom
On 25/06/2022 20:06, Vipul Siddharth wrote:
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
== Owner ==
- Name: [[User:thaller|Thomas Haller]] (NetworkManager)
- Email: thaller@redhat.com
- Name: [[User:dustymabe| Dusty Mabe]] (Fedora CoreOS)
- Email: dmabe@redhat.com
== Detailed Description ==
On Fedora, udev by default changes the MAC address of a wide range of software devices. This was introduced by systemd 242 in early 2019 (Fedora 31), when `MACAddressPolicy=` was extended to affect more types of devices.
With `MACAddressPolicy=persistent` udev's aim is to provide a stable MAC address, otherwise the kernel will assign a random one. However, that can cause problems:
Firstly, software devices are always created by some tool that has plans for the device. The tool may not expect that udev is going to change the MAC address and races against that. The best solution for the tool is to set the MAC address when creating an interface. This will prevent udev from changing the MAC address according to the MACAddressPolicy. Otherwise, the tool should wait for udev to initialize the device to avoid the race. In theory, a tool is always advised to wait for udev to initialize the device. However, if it were not for MACAddressPolicy, in common scenarios udev doesn't do anything relevant for software devices to make that necessary.
Secondly, for interface types bridge and bond, an unset MAC address has a special meaning to the kernel and the MAC address of the first port is used. If udev changes the MAC address, that no longer works. Now the generated MAC address is not directly discoverable as it is based on `/etc/machine-id` ([https://www.man7.org/linux/man-pages/man5/machine-id.5.html machine-id(5)]), among other data. Even if there were a tool to easily calculate the MAC address, it could be cumbersome to use it without logging into the machine first. The MAC address can directly affect the assigned IP address, for example when using DHCP. When booting a new virtual machine, the user might know the MAC address of the (virtual) "physical" interfaces. When bonding/bridging those interfaces, the bond/bridge would get one of the well known MAC addresses. `MACAddressPolicy=persistent` interferes with that.
The goal of persistent policy is to provide a stable MAC address. Note that if the tool or user who created the interface would want a certain MAC address, they have all the means to set it already. That applies regardless whether the tool is iproute2, NetworkManager, systemd-networkd. Neither NetworkManager nor systemd-networkd rely on udev's MACAddressPolicy for setting the MAC address. This behavior is mostly useful for plain `ip link add`, but it's unclear which real world user wants this behavior.
Of course, the user is welcome to configure the MAC address in any way they want. Including, dropping a link file that sets `MACAddressPolicy=persistent`. The problem is once udev sets a MAC address, it cannot be unset. Which makes this problematic to do by default.
While Fedora inherited this behavior from upstream systemd, RHEL-9 does not follow this behavior ([https://gitlab.com/redhat/centos-stream/rpms/systemd/-/blob/c8953519504bf2e6... centos9], [https://bugzilla.redhat.com/show_bug.cgi?id=1921094 rh#1921094]). For RHEL-8, this doesn't apply because the systemd there is too old to change the MAC address of most software devices.
This could be either implemented by patching `/usr/lib/systemd/network/99-default.link` to have a different policy, or by dropping a link file with higher priority. In the latter case, that override could be shipped either by udev or even by NetworkManager.
Another option is to change the scope of this proposal to only change to `MACAddressPolicy=none` for the device types where this causes the most issues (bridge/bond/team).
== Feedback ==
This was also discussed on upstream systemd mailing list [https://lists.freedesktop.org/archives/systemd-devel/2022-May/047893.html here]. The upstream systemd maintainers' opinion is that the current udev behavior is desirable, but accepts that distributions (or network stacks such as NetworkManager) may choose to change the default depending on their needs ([https://lists.freedesktop.org/archives/systemd-devel/2022-May/047943.html reference]). The goal here is to find out what the Fedora community thinks is the most appropriate default.
The RHEL-9 bug is [https://bugzilla.redhat.com/show_bug.cgi?id=1921094 [rh#1921094]].
== Benefit to Fedora ==
Pros:
Consistent behavior with RHEL8 and RHEL9.
Avoid race of udev and the tool that creates the interface.
Bridge and bond interfaces can get the MAC addresses from their first port.
In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will get a MAC related to one of its physical NIC devices. In the case of provisioning new systems (especially in a large datacenter) information about the server can be used to configure the network environment (DHCP, Firewall, etc) before the server is ever even powered on. This does mean that you may have to update your network environment configuration if you replace a NIC (con), but that you won't have to update your network environment configuration if you re-install your server (pro), which is what you'd have to do now with `MACAddressPolicy=persistent`.
Cons:
- Deviate from upstream systemd.
It is desirable that RHEL and Fedora behaves similar. A possible outcome could be the current behavior stays and RHEL 10 would change behavior. On the other hand, different distributions (or even Fedora spins) have different uses and needs. Deviating might be fine. In the same vein, there is also a desire to stay close to upstream systemd behavior. But the uses of systemd project go beyond Fedora/RHEL, so deviating here may also be fine.
== Scope ==
- Proposal owners:
The main goal of this request is to generate productive discussion and find the desired behavior. The implementation/changes are either way very simple.
- Other developers:
Other projects that wish a certain MAC address are welcome to set it for their devices. Including using udev's MACAddressPolicy.
- Release engineering:
Not needed for this change.
- Policies and guidelines: N/A (not needed for this Change)
- Trademark approval: N/A (not needed for this Change)
- Alignment with Objectives:
== Upgrade/compatibility impact ==
After the change, the MAC address for the affected device types changes.
== How To Test ==
- Create a software device two times, for example `ip link add type
bridge`. Note that the MAC address is either stable or random, depending on the `MACAddressPolicy=`.
- Note that if the software device has the MAC address set initially,
udev does not change it (`ip link add address aa:aa:aa:aa:aa:aa type bridge`). That depends on `/sys/class/net/$dev/addr_assign_type`.
- Create a bridge/bond interface without setting the MAC address.
Note that if `MACAddressPolicy=none`, the MAC address is random at first. Note that attaching the first port will update the controller's MAC address. On the other hand, with `MACAddressPolicy=persistent`, the MAC address of the controller is fixed and not inherited from the port.
Run
ip monitor link & while : ; do ip link del xxx ip link add name xxx type dummy \ && ip link set xxx addr aa:00:00:00:00:00 \ && ip link show xxx | grep -q aa:00:00:00:00:00 \ || break done
to reproduce the race between a simple tool and udev changing the MAC address.
== User Experience ==
Bond/bridge devices would again get assigned the MAC address of the first NIC added to the device. If we chose to not limit the scope of this change to just bonds/bridges then all software devices would get randomly assigned MAC addresses.
== Dependencies ==
None.
== Contingency Plan ==
If the change is rejected, nothing needs to be done. The change itself will be simple to implement. Contingency deadline: beta freeze Blocks release? No
== Documentation ==
TODO.
== Release Notes ==
Hi,
On Mon, 2022-06-27 at 13:09 +0100, Tom Hughes via devel wrote:
Twice now I have had to go and reconfigure my networks after a Fedora upgrade has changed the MAC assignment policy.
Interesting. Are you sure it was twice? I thought it changed "only" once in F31 (2019).
Thomas
On 27/06/2022 17:05, Thomas Haller wrote:
On Mon, 2022-06-27 at 13:09 +0100, Tom Hughes via devel wrote:
Twice now I have had to go and reconfigure my networks after a Fedora upgrade has changed the MAC assignment policy.
Interesting. Are you sure it was twice? I thought it changed "only" once in F31 (2019).
No it has just changed again in F36 at least for bridges, back to what it was before the previous change I believe.
Tom
On 27/06/2022 17:09, Tom Hughes via devel wrote:
On 27/06/2022 17:05, Thomas Haller wrote:
On Mon, 2022-06-27 at 13:09 +0100, Tom Hughes via devel wrote:
Twice now I have had to go and reconfigure my networks after a Fedora upgrade has changed the MAC assignment policy.
Interesting. Are you sure it was twice? I thought it changed "only" once in F31 (2019).
No it has just changed again in F36 at least for bridges, back to what it was before the previous change I believe.
I've had a look through the git log for my DNS and taking one machine as an example:
initially - bridge had MAC of 00:b0:c2:02:4c:f3 from member Aug 2015 (F22) - bridge changed to 1a:81:14:46:3c:c7 Jan 2020 (F31) - bridge changed to fe:e3:75:bd:6a:8c May 2022 (F36) - bridge changed back to 1a:81:14:46:3c:c7
Only the first one corresponds to a member, so I think F22 is where persistent addresses were first introduced. I'm not sure what happened in F31 than then got reverted in F36.
Tom
On Mon, 2022-06-27 at 17:26 +0100, Tom Hughes wrote:
On 27/06/2022 17:09, Tom Hughes via devel wrote:
On 27/06/2022 17:05, Thomas Haller wrote:
On Mon, 2022-06-27 at 13:09 +0100, Tom Hughes via devel wrote:
Twice now I have had to go and reconfigure my networks after a Fedora upgrade has changed the MAC assignment policy.
Interesting. Are you sure it was twice? I thought it changed "only" once in F31 (2019).
No it has just changed again in F36 at least for bridges, back to what it was before the previous change I believe.
I've had a look through the git log for my DNS and taking one machine as an example:
initially - bridge had MAC of 00:b0:c2:02:4c:f3 from member Aug 2015 (F22) - bridge changed to 1a:81:14:46:3c:c7 Jan 2020 (F31) - bridge changed to fe:e3:75:bd:6a:8c May 2022 (F36) - bridge changed back to 1a:81:14:46:3c:c7
Only the first one corresponds to a member, so I think F22 is where persistent addresses were first introduced. I'm not sure what happened in F31 than then got reverted in F36.
Tom
Hi Tom,
Thanks for checking.
I don't know why that would have happened. It also depends which tools you are using for creating the bridge.
I agree, this is something to avoid! And maybe a the best reason to not change anything on Fedora 37.
Thomas
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
IMHO, the default should be "none" for all devices. I do not see why systemd spoofs MAC addresses by default.
Kevin Kofler
On Sat, Jun 25, 2022 at 3:06 PM Vipul Siddharth siddharthvipul1@gmail.com wrote:
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
== Owner ==
- Name: [[User:thaller|Thomas Haller]] (NetworkManager)
- Email: thaller@redhat.com
- Name: [[User:dustymabe| Dusty Mabe]] (Fedora CoreOS)
- Email: dmabe@redhat.com
== Detailed Description ==
On Fedora, udev by default changes the MAC address of a wide range of software devices. This was introduced by systemd 242 in early 2019 (Fedora 31), when `MACAddressPolicy=` was extended to affect more types of devices.
With `MACAddressPolicy=persistent` udev's aim is to provide a stable MAC address, otherwise the kernel will assign a random one. However, that can cause problems:
Firstly, software devices are always created by some tool that has plans for the device. The tool may not expect that udev is going to change the MAC address and races against that. The best solution for the tool is to set the MAC address when creating an interface. This will prevent udev from changing the MAC address according to the MACAddressPolicy. Otherwise, the tool should wait for udev to initialize the device to avoid the race. In theory, a tool is always advised to wait for udev to initialize the device. However, if it were not for MACAddressPolicy, in common scenarios udev doesn't do anything relevant for software devices to make that necessary.
Secondly, for interface types bridge and bond, an unset MAC address has a special meaning to the kernel and the MAC address of the first port is used. If udev changes the MAC address, that no longer works. Now the generated MAC address is not directly discoverable as it is based on `/etc/machine-id` ([https://www.man7.org/linux/man-pages/man5/machine-id.5.html machine-id(5)]), among other data. Even if there were a tool to easily calculate the MAC address, it could be cumbersome to use it without logging into the machine first. The MAC address can directly affect the assigned IP address, for example when using DHCP. When booting a new virtual machine, the user might know the MAC address of the (virtual) "physical" interfaces. When bonding/bridging those interfaces, the bond/bridge would get one of the well known MAC addresses. `MACAddressPolicy=persistent` interferes with that.
The goal of persistent policy is to provide a stable MAC address. Note that if the tool or user who created the interface would want a certain MAC address, they have all the means to set it already. That applies regardless whether the tool is iproute2, NetworkManager, systemd-networkd. Neither NetworkManager nor systemd-networkd rely on udev's MACAddressPolicy for setting the MAC address. This behavior is mostly useful for plain `ip link add`, but it's unclear which real world user wants this behavior.
Of course, the user is welcome to configure the MAC address in any way they want. Including, dropping a link file that sets `MACAddressPolicy=persistent`. The problem is once udev sets a MAC address, it cannot be unset. Which makes this problematic to do by default.
While Fedora inherited this behavior from upstream systemd, RHEL-9 does not follow this behavior ([https://gitlab.com/redhat/centos-stream/rpms/systemd/-/blob/c8953519504bf2e6... centos9], [https://bugzilla.redhat.com/show_bug.cgi?id=1921094 rh#1921094]). For RHEL-8, this doesn't apply because the systemd there is too old to change the MAC address of most software devices.
This could be either implemented by patching `/usr/lib/systemd/network/99-default.link` to have a different policy, or by dropping a link file with higher priority. In the latter case, that override could be shipped either by udev or even by NetworkManager.
Another option is to change the scope of this proposal to only change to `MACAddressPolicy=none` for the device types where this causes the most issues (bridge/bond/team).
== Feedback ==
This was also discussed on upstream systemd mailing list [https://lists.freedesktop.org/archives/systemd-devel/2022-May/047893.html here]. The upstream systemd maintainers' opinion is that the current udev behavior is desirable, but accepts that distributions (or network stacks such as NetworkManager) may choose to change the default depending on their needs ([https://lists.freedesktop.org/archives/systemd-devel/2022-May/047943.html reference]). The goal here is to find out what the Fedora community thinks is the most appropriate default.
The RHEL-9 bug is [https://bugzilla.redhat.com/show_bug.cgi?id=1921094 [rh#1921094]].
I'd be interested in seeing the justification for the revert in RHEL 9. From my perspective, it doesn't make sense to disable this because it makes bonds/teams/etc. device dependent. This can be particularly bad if you've got a bond and then you swap out the hardware over time. The MAC address would change on the bond, which could break any layer 2 rules in place on the network.
The RHEL 9 bug seems to be private, though.
Once upon a time, Neal Gompa ngompa13@gmail.com said:
I'd be interested in seeing the justification for the revert in RHEL 9. From my perspective, it doesn't make sense to disable this because it makes bonds/teams/etc. device dependent. This can be particularly bad if you've got a bond and then you swap out the hardware over time. The MAC address would change on the bond, which could break any layer 2 rules in place on the network.
The flip side of that is that converting a stand-alone interface to a LAG changes MAC address on Linux, while it doesn't on most other things like routers and switches. Replacing a NIC is a more uncommon event IMHO, which already would be disruptive to things that care about the MAC in a non-LAG setup.
I don't care strongly either way myself; I've started copying the MAC manually when I set up a LAG, but it would be nice to drop that step.
On 6/27/22 22:35, Neal Gompa wrote:
On Sat, Jun 25, 2022 at 3:06 PM Vipul Siddharth siddharthvipul1@gmail.com wrote:
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
== Owner ==
- Name: [[User:thaller|Thomas Haller]] (NetworkManager)
- Email: thaller@redhat.com
- Name: [[User:dustymabe| Dusty Mabe]] (Fedora CoreOS)
- Email: dmabe@redhat.com
== Detailed Description ==
On Fedora, udev by default changes the MAC address of a wide range of software devices. This was introduced by systemd 242 in early 2019 (Fedora 31), when `MACAddressPolicy=` was extended to affect more types of devices.
With `MACAddressPolicy=persistent` udev's aim is to provide a stable MAC address, otherwise the kernel will assign a random one. However, that can cause problems:
Firstly, software devices are always created by some tool that has plans for the device. The tool may not expect that udev is going to change the MAC address and races against that. The best solution for the tool is to set the MAC address when creating an interface. This will prevent udev from changing the MAC address according to the MACAddressPolicy. Otherwise, the tool should wait for udev to initialize the device to avoid the race. In theory, a tool is always advised to wait for udev to initialize the device. However, if it were not for MACAddressPolicy, in common scenarios udev doesn't do anything relevant for software devices to make that necessary.
Secondly, for interface types bridge and bond, an unset MAC address has a special meaning to the kernel and the MAC address of the first port is used. If udev changes the MAC address, that no longer works. Now the generated MAC address is not directly discoverable as it is based on `/etc/machine-id` ([https://www.man7.org/linux/man-pages/man5/machine-id.5.html machine-id(5)]), among other data. Even if there were a tool to easily calculate the MAC address, it could be cumbersome to use it without logging into the machine first. The MAC address can directly affect the assigned IP address, for example when using DHCP. When booting a new virtual machine, the user might know the MAC address of the (virtual) "physical" interfaces. When bonding/bridging those interfaces, the bond/bridge would get one of the well known MAC addresses. `MACAddressPolicy=persistent` interferes with that.
The goal of persistent policy is to provide a stable MAC address. Note that if the tool or user who created the interface would want a certain MAC address, they have all the means to set it already. That applies regardless whether the tool is iproute2, NetworkManager, systemd-networkd. Neither NetworkManager nor systemd-networkd rely on udev's MACAddressPolicy for setting the MAC address. This behavior is mostly useful for plain `ip link add`, but it's unclear which real world user wants this behavior.
Of course, the user is welcome to configure the MAC address in any way they want. Including, dropping a link file that sets `MACAddressPolicy=persistent`. The problem is once udev sets a MAC address, it cannot be unset. Which makes this problematic to do by default.
While Fedora inherited this behavior from upstream systemd, RHEL-9 does not follow this behavior ([https://gitlab.com/redhat/centos-stream/rpms/systemd/-/blob/c8953519504bf2e6... centos9], [https://bugzilla.redhat.com/show_bug.cgi?id=1921094 rh#1921094]). For RHEL-8, this doesn't apply because the systemd there is too old to change the MAC address of most software devices.
This could be either implemented by patching `/usr/lib/systemd/network/99-default.link` to have a different policy, or by dropping a link file with higher priority. In the latter case, that override could be shipped either by udev or even by NetworkManager.
Another option is to change the scope of this proposal to only change to `MACAddressPolicy=none` for the device types where this causes the most issues (bridge/bond/team).
== Feedback ==
This was also discussed on upstream systemd mailing list [https://lists.freedesktop.org/archives/systemd-devel/2022-May/047893.html here]. The upstream systemd maintainers' opinion is that the current udev behavior is desirable, but accepts that distributions (or network stacks such as NetworkManager) may choose to change the default depending on their needs ([https://lists.freedesktop.org/archives/systemd-devel/2022-May/047943.html reference]). The goal here is to find out what the Fedora community thinks is the most appropriate default.
The RHEL-9 bug is [https://bugzilla.redhat.com/show_bug.cgi?id=1921094 [rh#1921094]].
I'd be interested in seeing the justification for the revert in RHEL 9. From my perspective, it doesn't make sense to disable this because it makes bonds/teams/etc. device dependent. This can be particularly bad if you've got a bond and then you swap out the hardware over time. The MAC address would change on the bond, which could break any layer 2 rules in place on the network.
Hmm. To me this feel more in line with a user's expectation. Let's draw a parallel here to a single NIC (no bond) situation. Here are the two cases:
- single NIC - reinstalling your OS you don't need to update any external network env/config - the NIC fails - replace the NIC - update your external network env/configuration with the new MAC
- dual NIC (bond) - with MACAddressPolicy=persistent - reinstall OS you need to update external network env/config - a single NIC fails - replace the NIC - No need to update your external network env/config - no new MAC - with MACAddressPolicy=none - reinstall OS you DON'T need to update external network env/config - a single NIC fails - replace the NIC - update your external network env/configuration with the new MAC
To me the proposal to use MACAddressPolicy=none for bond/bridge/team gets us way more inline with what users expect, which is that if they have to replace a NIC they have to update something. I don't think users expect that if they re-install their OS they'll need to update their network env/config.
I also suspect that an OS re-install is much much more common than a NIC failure replacement.
The RHEL 9 bug seems to be private, though.
On 6/25/22 15:06, Vipul Siddharth wrote:
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
Based on the feedback here on the list we have modified the scope of this proposal to now be limited to changing the MACAddressPolicy=none for bond/bridge/team devices only.
New summary:
``` The systemd-udev package installs "/usr/lib/systemd/network/99-default.link", which sets Link.MACAddressPolicy=persistent for all software NIC devices. This proposal is to add to the policy so that we use Link.MACAddressPolicy=none for bond/bridge/team devices. ```
https://fedoraproject.org/wiki/Changes/MAC_Address_Policy_none
On Di, 05.07.22 22:35, Dusty Mabe (dusty@dustymabe.com) wrote:
On 6/25/22 15:06, Vipul Siddharth wrote:
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
Based on the feedback here on the list we have modified the scope of this proposal to now be limited to changing the MACAddressPolicy=none for bond/bridge/team devices only.
New summary:
The systemd-udev package installs "/usr/lib/systemd/network/99-default.link", which sets Link.MACAddressPolicy=persistent for all software NIC devices. This proposal is to add to the policy so that we use Link.MACAddressPolicy=none for bond/bridge/team devices.https://fedoraproject.org/wiki/Changes/MAC_Address_Policy_none
So, with changing this back you'll also break all those setups where it is assumed the bridge MAC just works and is stable and independent from the devices added to it and the order in which they are added in.
What makes you so sure, that changing this *again* is so much better than just leaving it the way it is now? This change isn't precisely new, and changing stuff forth and back comes at quite a price.
Generally, if we change behaviour like this, then the pros must seriously outweight the cons. Now you might argue that when the original change was made that wasn't the case, that's a valid opinion, though one I disagree with. But that has no effect on today, on the question whether changing it again is worth it. I am pretty sure that there are *also* numerous scripts that benefit from the predictability and stability you get with the status quo, and which you'll break if you revert to the old state – again.
I am not convinced we should flip flop on this all the time. Yes, it's unfortunate that people tripped over this, but you are not really making it better if you then break it *again*, given the benefit is unclear, and the major software creating bridges/bonds/… doesn't care anyway (i.e. NM, networkd, …).
I for one do not see where you'd get the crystal ball to look into to know that by changing this *again* you'll make bazillions of people happy, and only very few people sad, because you break their stuff. It might very well be that you'll make more people sad because you break their stuff again, than were happy before.
Also, let's not forget that allowing users to set the policy is a good thing, we should let them. Given that the original change was already made a lot of software that cannot work with such changes has already been updated to override the default policy. That's a good thing, since the overall system becomes more robust and people can more safely change the policies locally, with less breakage to expect.
if you now revert to the status quo ante, then you basically also say: fuck it, we don't care that software is fixed to work with changed policies, let's keep things brittle that you cannot change policies anymore effectively because we are sticking our head in the sand and don't care that they are fixed.
So, I am not convinced.
I can accept that this wasn't handled particularly nicely originally. My proposal for addressing this is via documentation, i.e update the udev and iproute docs to explicitly point to the issue and how to deal with it, with an example drop-in to make it easy to deal with it.
Lennart
-- Lennart Poettering, Berlin
On 7/6/22 04:37, Lennart Poettering wrote:
On Di, 05.07.22 22:35, Dusty Mabe (dusty@dustymabe.com) wrote:
On 6/25/22 15:06, Vipul Siddharth wrote:
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary ==
The `systemd-udev` package installs `"/usr/lib/systemd/network/99-default.link"`, which sets `Link.MACAddressPolicy=persistent`. This proposal is to change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address. This is particularly important for bridge and bond devices.
This change can either only apply to bridge/bond devices, or to various software devices. That is to be discussed.
Based on the feedback here on the list we have modified the scope of this proposal to now be limited to changing the MACAddressPolicy=none for bond/bridge/team devices only.
New summary:
The systemd-udev package installs "/usr/lib/systemd/network/99-default.link", which sets Link.MACAddressPolicy=persistent for all software NIC devices. This proposal is to add to the policy so that we use Link.MACAddressPolicy=none for bond/bridge/team devices.https://fedoraproject.org/wiki/Changes/MAC_Address_Policy_none
So, with changing this back you'll also break all those setups where it is assumed the bridge MAC just works and is stable and independent from the devices added to it and the order in which they are added in.
One thing we do need to think about here is the "upgrade" case. We could consider leaving existing (upgrading) systems alone.
What makes you so sure, that changing this *again* is so much better than just leaving it the way it is now? This change isn't precisely new, and changing stuff forth and back comes at quite a price.
Indeed it does come at a price, which is why we're discussing the merits here and also why we scaled back the proposal to just cover the use case where it matters the most. For example, this kernel documentation is a good place where a user is now confused because the kernel behaves one way for bonds, but systemd delivers configuration to make it behave differently: https://wiki.linuxfoundation.org/networking/bonding#where_does_a_bonding_dev...
If you look at the longer term landscape (i.e. years from now) the change is worth it if it's the right change to make. I think the fact that RHEL9 didn't pick up the original change is an indication of it's value in enterprise environments where bond/bridge/team devices are more common.
Generally, if we change behaviour like this, then the pros must seriously outweight the cons. Now you might argue that when the original change was made that wasn't the case, that's a valid opinion, though one I disagree with. But that has no effect on today, on the question whether changing it again is worth it. I am pretty sure that there are *also* numerous scripts that benefit from the predictability and stability you get with the status quo, and which you'll break if you revert to the old state – again.
There is a lot that happens upstream systemd that we should more carefully consider in Fedora. For example this probably should have been proposed to Fedora as a change then (by systemd maintainer in Fedora I suspect) and the merits examined at that point in time.
I am not convinced we should flip flop on this all the time. Yes, it's unfortunate that people tripped over this, but you are not really making it better if you then break it *again*, given the benefit is unclear, and the major software creating bridges/bonds/… doesn't care anyway (i.e. NM, networkd, …).
This definitely affects NetworkManager and people do care.. See https://github.com/coreos/fedora-coreos-tracker/issues/919 https://github.com/systemd/systemd/issues/15208
I for one do not see where you'd get the crystal ball to look into to know that by changing this *again* you'll make bazillions of people happy, and only very few people sad, because you break their stuff. It might very well be that you'll make more people sad because you break their stuff again, than were happy before.
The discussion here is one way to gather feedback. There is also the collective expertise of the members of FESCO, who make decisions like this all the time.
Also, let's not forget that allowing users to set the policy is a good thing, we should let them. Given that the original change was already made a lot of software that cannot work with such changes has already been updated to override the default policy. That's a good thing, since the overall system becomes more robust and people can more safely change the policies locally, with less breakage to expect.
if you now revert to the status quo ante, then you basically also say: fuck it, we don't care that software is fixed to work with changed policies, let's keep things brittle that you cannot change policies anymore effectively because we are sticking our head in the sand and don't care that they are fixed.
I don't think that accurately reflects what we are saying here. I'm not sure why making a change here makes things more brittle. Maybe it depends on the implementation?
So, I am not convinced.
I can accept that this wasn't handled particularly nicely originally. My proposal for addressing this is via documentation, i.e update the udev and iproute docs to explicitly point to the issue and how to deal with it, with an example drop-in to make it easy to deal with it.
devel@lists.stg.fedoraproject.org
-
Chris Adams -
Daniel P. Berrangé -
Demi Marie Obenour -
Dusty Mabe -
Gabriel L. Somlo -
Hans de Goede -
Kevin Kofler -
Lennart Poettering -
Neal Gompa -
Thomas Haller -
Tom Hughes -
Vipul Siddharth -
Zbigniew Jędrzejewski-Szmek