nmstate-devel August 2022

nmstate-devel@lists.fedorahosted.org

2 participants
3 discussions

Re: OVS missing some interfaces on node reboot or NetworkManager retstart
by Fernando Fernandez Mancera 29 Aug '22

29 Aug '22

Hi Nobin, just wanted to let you know that I backported the fix to nm-1-38, nm-1-36 and nm-1-34 branches. You can check it here: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/tree/nm-1-34 Let me know if you need anything else. Thanks, Fernando. On Mon, Aug 29, 2022 at 11:08 AM Nobin Mathew <nobin.mathew(a)ericsson.com> wrote: > > Thanks Fernando for the quick response... > > It will be great if you can provide a backported patch for 1.34, otherwise please share the patch which fixed this problem, I will try to backport. > > -Nobin > > -----Original Message----- > From: Fernando Fernandez Mancera <ferferna(a)redhat.com> > Sent: Monday, August 29, 2022 1:06 PM > To: Nobin Mathew <nobin.mathew(a)ericsson.com> > Cc: nmstate-devel(a)lists.fedorahosted.org > Subject: Re: [nmstate-devel] OVS missing some interfaces on node reboot or NetworkManager retstart > > Hi! > > On Mon, Aug 29, 2022 at 7:36 AM Nobin Mathew <nobin.mathew(a)ericsson.com> wrote: > > > > Hi, > > > > We r running > > > > ###### > > > > /home/ # NetworkManager -V > > > > 1.34.0 > > > > /home/ # ovs-vsctl -V > > > > ovs-vsctl (Open vSwitch) 2.14.2 > > > > DB Schema 8.2.0 > > > > > > > > /home/ # uname -a > > > > Linux 5.3.18-150300.59.87-default SMP Thu Jul 21 14:31:28 UTC 2022 > > (cc90276) x86_64 x86_64 x86_64 GNU/Linux > > > > > > > > this is SLES15 sp3 machine. > > > > > > > > pool1-bm25c5:~> nmstatectl --version > > > > 1.2.0 > > > > pool1-bm25c5:~> > > > > > > > > ###### > > > > when we reboot node or restart NetworkManager(systemctl restart NetworkManager) we are loosing some of the ovs interfaces (randomly, not on all restarts). > > > > I think this issue is related to this bug: > https://bugzilla.redhat.com/show_bug.cgi?id=2080236 > > It is a NetworkManager issue and it was fixed in NetworkManager-1.39.6, could you check it out? Would you need an upstream backport to 1.34? > > Thanks! > Fernando. > > > logs are here: > > > > Missing interfaces: > > > > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 > > 5555731-93530804397fe1ff&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= > > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78 > > d466%23file-missing-interfaces > > > > NetworkManager trace log for restart duration > > > > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 > > 5555731-cfdd31dcdea77648&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= > > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78 > > d466%23file-networkmanager-logs-with-trace > > > > /etc/NetworkManager/system-connections > > > > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 > > 5555731-3474c8a3dfe835ee&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= > > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78 > > d466%23file-connection-conf > > > > ovsdb-client-dump for non working case(or when interfaces are missing) > > > > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 > > 5555731-289bfcbe422b0355&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= > > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78 > > d466%23file-non-working-case-ovsdb-client-dump > > > > ovsdb-client-dump for working case(or when all interfaces are present) > > > > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 > > 5555731-b1efb31f6496469e&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= > > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78 > > d466%23file-on-working-ovsdb-client-dump > > > > ovs-vsctl-show when interfaces are missing > > > > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 > > 5555731-75a8fd4d20fe10f6&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= > > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78 > > d466%23file-ovs-vsctl-show > > > > ovs-vswitchd-log for the restart duration > > > > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 > > 5555731-8c324322ae3a8e8f&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= > > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78 > > d466%23file-ovs-vswitchd-log > > > > ovsdb-tool-show-log for the restart duration > > > > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 > > 5555731-c5b663cb0344f371&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= > > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78 > > d466%23file-sudo-ovsdb-tool-mmmmmm-show-log > > > > In this particular instance of run, we lost two interfaces. > > > > we have a script which sets "autoconnect-slaves" to 0, which was introduced for an older version of NM. grep -lF -e type=ovs-bridge -e type=ovs-interface -e type=ovs-port /etc/NetworkManager/system-connections/*|xargs -I % nmcli connection modify filename % connection.autoconnect-slaves 0 this scripts also runs currently, that why all "autoconnect-slaves" set to zero. > > > > when we are losing interfaces that port additions are not coming to ovsdb: > > > > 2022-08-26T22:38:26.176Z|00179|bridge|INFO|bridge data_bridge: added > > interface traffic5_iface on port 1 > > > > 2022-08-26T22:38:26.348Z|00182|bridge|INFO|bridge data_bridge: added > > interface traffic1_iface on port 2 > > > > 2022-08-26T22:38:26.376Z|00183|bridge|INFO|bridge data_bridge: added > > interface bm25c_ext_iface on port 3 > > > > 2022-08-26T22:38:26.376Z|00184|bridge|INFO|bridge data_bridge: added > > interface bond0 on port 4 > > > > Only 4 interfaces are getting added back, two are missing, "ovsdb-tool -mmmmmm show-log " shows the same. > > > > Any pointers where it is going wrong? > > > > -Nobin > > > > > > > > _______________________________________________ > > nmstate-devel mailing list -- nmstate-devel(a)lists.fedorahosted.org > > To unsubscribe send an email to > > nmstate-devel-leave(a)lists.fedorahosted.org > > Fedora Code of Conduct: > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: > > https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 > > 5555731-09b7aeb775896a89&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= > > https%3A%2F%2Flists.fedorahosted.org%2Farchives%2Flist%2Fnmstate-devel > > %40lists.fedorahosted.org Do not reply to spam, report it: > > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 > > 5555731-1df4bd1e5f7cb906&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u= > > https%3A%2F%2Fpagure.io%2Ffedora-infrastructure%2Fnew_issue

2 1

Re: OVS missing some interfaces on node reboot or NetworkManager retstart
by Fernando Fernandez Mancera 29 Aug '22

29 Aug '22

Hi! On Mon, Aug 29, 2022 at 7:36 AM Nobin Mathew <nobin.mathew(a)ericsson.com> wrote: > > Hi, > > We r running > > ###### > > /home/ # NetworkManager -V > > 1.34.0 > > /home/ # ovs-vsctl -V > > ovs-vsctl (Open vSwitch) 2.14.2 > > DB Schema 8.2.0 > > > > /home/ # uname -a > > Linux 5.3.18-150300.59.87-default SMP Thu Jul 21 14:31:28 UTC 2022 (cc90276) x86_64 x86_64 x86_64 GNU/Linux > > > > this is SLES15 sp3 machine. > > > > pool1-bm25c5:~> nmstatectl --version > > 1.2.0 > > pool1-bm25c5:~> > > > > ###### > > when we reboot node or restart NetworkManager(systemctl restart NetworkManager) we are loosing some of the ovs interfaces (randomly, not on all restarts). > I think this issue is related to this bug: https://bugzilla.redhat.com/show_bug.cgi?id=2080236 It is a NetworkManager issue and it was fixed in NetworkManager-1.39.6, could you check it out? Would you need an upstream backport to 1.34? Thanks! Fernando. > logs are here: > > Missing interfaces: > > https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-missi… > > NetworkManager trace log for restart duration > > https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-netwo… > > /etc/NetworkManager/system-connections > > https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-conne… > > ovsdb-client-dump for non working case(or when interfaces are missing) > > https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-non-w… > > ovsdb-client-dump for working case(or when all interfaces are present) > > https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-on-wo… > > ovs-vsctl-show when interfaces are missing > > https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-ovs-v… > > ovs-vswitchd-log for the restart duration > > https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-ovs-v… > > ovsdb-tool-show-log for the restart duration > > https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-sudo-… > > In this particular instance of run, we lost two interfaces. > > we have a script which sets "autoconnect-slaves" to 0, which was introduced for an older version of NM. grep -lF -e type=ovs-bridge -e type=ovs-interface -e type=ovs-port /etc/NetworkManager/system-connections/*|xargs -I % nmcli connection modify filename % connection.autoconnect-slaves 0 this scripts also runs currently, that why all "autoconnect-slaves" set to zero. > > when we are losing interfaces that port additions are not coming to ovsdb: > > 2022-08-26T22:38:26.176Z|00179|bridge|INFO|bridge data_bridge: added interface traffic5_iface on port 1 > > 2022-08-26T22:38:26.348Z|00182|bridge|INFO|bridge data_bridge: added interface traffic1_iface on port 2 > > 2022-08-26T22:38:26.376Z|00183|bridge|INFO|bridge data_bridge: added interface bm25c_ext_iface on port 3 > > 2022-08-26T22:38:26.376Z|00184|bridge|INFO|bridge data_bridge: added interface bond0 on port 4 > > Only 4 interfaces are getting added back, two are missing, "ovsdb-tool -mmmmmm show-log " shows the same. > > Any pointers where it is going wrong? > > -Nobin > > > > _______________________________________________ > nmstate-devel mailing list -- nmstate-devel(a)lists.fedorahosted.org > To unsubscribe send an email to nmstate-devel-leave(a)lists.fedorahosted.org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/nmstate-devel@lists.fedorahost… > Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

2 2

OVS missing some interfaces on node reboot or NetworkManager retstart
by Nobin Mathew 29 Aug '22

29 Aug '22

Hi, We r running ###### /home/ # NetworkManager -V 1.34.0 /home/ # ovs-vsctl -V ovs-vsctl (Open vSwitch) 2.14.2 DB Schema 8.2.0 /home/ # uname -a Linux 5.3.18-150300.59.87-default SMP Thu Jul 21 14:31:28 UTC 2022 (cc90276) x86_64 x86_64 x86_64 GNU/Linux this is SLES15 sp3 machine. pool1-bm25c5:~> nmstatectl --version 1.2.0 pool1-bm25c5:~> ###### when we reboot node or restart NetworkManager(systemctl restart NetworkManager) we are loosing some of the ovs interfaces (randomly, not on all restarts). logs are here: Missing interfaces: https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-missi… NetworkManager trace log for restart duration https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-netwo… /etc/NetworkManager/system-connections https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-conne… ovsdb-client-dump for non working case(or when interfaces are missing) https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-non-w… ovsdb-client-dump for working case(or when all interfaces are present) https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-on-wo… ovs-vsctl-show when interfaces are missing https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-ovs-v… ovs-vswitchd-log for the restart duration https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-ovs-v… ovsdb-tool-show-log for the restart duration https://gist.github.com/nmathew/d7d472b938f389c4efbadd01ef78d466#file-sudo-… In this particular instance of run, we lost two interfaces. we have a script which sets "autoconnect-slaves" to 0, which was introduced for an older version of NM. grep -lF -e type=ovs-bridge -e type=ovs-interface -e type=ovs-port /etc/NetworkManager/system-connections/*|xargs -I % nmcli connection modify filename % connection.autoconnect-slaves 0 this scripts also runs currently, that why all "autoconnect-slaves" set to zero. when we are losing interfaces that port additions are not coming to ovsdb: 2022-08-26T22:38:26.176Z|00179|bridge|INFO|bridge data_bridge: added interface traffic5_iface on port 1 2022-08-26T22:38:26.348Z|00182|bridge|INFO|bridge data_bridge: added interface traffic1_iface on port 2 2022-08-26T22:38:26.376Z|00183|bridge|INFO|bridge data_bridge: added interface bm25c_ext_iface on port 3 2022-08-26T22:38:26.376Z|00184|bridge|INFO|bridge data_bridge: added interface bond0 on port 4 Only 4 interfaces are getting added back, two are missing, "ovsdb-tool -mmmmmm show-log " shows the same. Any pointers where it is going wrong? -Nobin

1 0

2024

2023

2022

2021

2020

2019

nmstate-devel August 2022