When doing CPU or memory hotplug, multiple udev events will be triggered together. This will bring unnecessary workload for the host and make udev take much longer to settle.
Previous commit f2883275 added a workaround to solve the problem of systemd failing kdump servive due to burst of restart requests, but the unnecessary workload is not reduced. And currently, this is causing timeout failure on Hyper-V environment.
This patch fixes the problem by merging concurrent kdump restart requests. Tested with Hyper-V VM, no new issues found.
Signed-off-by: Kairui Song kasong@redhat.com --- Update from V1: Fix packaging problem, make kdump-udev-throttler executable and include it in main package.
98-kexec.rules | 8 ++++---- kdump-udev-throttler | 25 +++++++++++++++++++++++++ kexec-tools.spec | 3 +++ 3 files changed, 32 insertions(+), 4 deletions(-) create mode 100755 kdump-udev-throttler
diff --git a/98-kexec.rules b/98-kexec.rules index e32ee13..eee1df1 100644 --- a/98-kexec.rules +++ b/98-kexec.rules @@ -1,4 +1,4 @@ -SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service" +SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="memory", ACTION=="online", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="kdump-udev-throttler" diff --git a/kdump-udev-throttler b/kdump-udev-throttler new file mode 100755 index 0000000..272d226 --- /dev/null +++ b/kdump-udev-throttler @@ -0,0 +1,25 @@ +#!/bin/bash + +throttle_lock="/var/lock/kdump-udev-throttle" +interval=2 + +exec 9>$throttle_lock +flock -n 9 + +if [ $? -ne 0 ]; then + echo "Another app is currently holding the throttle lock, extend throttle timing and exiting" + exit 0 +fi + +timeout_time() { + echo $(($(stat -c "%Y" $throttle_lock) + $interval)) +} + +# Wait until no one try to acquire the throttle_lock in past $interval seconds +while [[ $(timeout_time) -gt $(date +%s) ]]; do + sleep 1 +done + +systemctl try-restart kdump + +exit 0 diff --git a/kexec-tools.spec b/kexec-tools.spec index 6330534..91d322f 100644 --- a/kexec-tools.spec +++ b/kexec-tools.spec @@ -29,6 +29,7 @@ Source24: kdump.sysconfig.ppc64le Source25: kdumpctl.8 Source26: live-image-kdump-howto.txt Source27: early-kdump-howto.txt +Source28: kdump-udev-throttler
####################################### # These are sources for mkdumpramfs @@ -171,6 +172,7 @@ install -m 755 %{SOURCE23} $RPM_BUILD_ROOT%{_prefix}/lib/kdump/kdump-lib-initram # For s390x the ELF header is created in the kdump kernel and therefore kexec # udev rules are not required install -m 644 %{SOURCE14} $RPM_BUILD_ROOT%{_udevrulesdir}/98-kexec.rules +install -m 755 %{SOURCE28} $RPM_BUILD_ROOT%{_udevrulesdir}/../kdump-udev-throttler %endif install -m 644 %{SOURCE15} $RPM_BUILD_ROOT%{_mandir}/man5/kdump.conf.5 install -m 644 %{SOURCE16} $RPM_BUILD_ROOT%{_unitdir}/kdump.service @@ -298,6 +300,7 @@ done %config(noreplace,missingok) %{_sysconfdir}/kdump.conf %ifnarch s390x %config %{_udevrulesdir} +%{_udevrulesdir}/../kdump-udev-throttler %endif %{dracutlibdir}/modules.d/* %dir %{_localstatedir}/crash
Hi Kairui, On 10/19/18 at 04:24pm, Kairui Song wrote:
When doing CPU or memory hotplug, multiple udev events will be triggered together. This will bring unnecessary workload for the host and make udev take much longer to settle.
Previous commit f2883275 added a workaround to solve the problem of systemd failing kdump servive due to burst of restart requests, but the unnecessary workload is not reduced. And currently, this is causing timeout failure on Hyper-V environment.
This patch fixes the problem by merging concurrent kdump restart requests. Tested with Hyper-V VM, no new issues found.
Signed-off-by: Kairui Song kasong@redhat.com
Update from V1: Fix packaging problem, make kdump-udev-throttler executable and include it in main package.
98-kexec.rules | 8 ++++---- kdump-udev-throttler | 25 +++++++++++++++++++++++++ kexec-tools.spec | 3 +++ 3 files changed, 32 insertions(+), 4 deletions(-) create mode 100755 kdump-udev-throttler
diff --git a/98-kexec.rules b/98-kexec.rules index e32ee13..eee1df1 100644 --- a/98-kexec.rules +++ b/98-kexec.rules @@ -1,4 +1,4 @@ -SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service" +SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="memory", ACTION=="online", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="kdump-udev-throttler" diff --git a/kdump-udev-throttler b/kdump-udev-throttler new file mode 100755 index 0000000..272d226 --- /dev/null +++ b/kdump-udev-throttler @@ -0,0 +1,25 @@ +#!/bin/bash
+throttle_lock="/var/lock/kdump-udev-throttle" +interval=2
+exec 9>$throttle_lock +flock -n 9
+if [ $? -ne 0 ]; then
echo "Another app is currently holding the throttle lock, extend throttle timing and exiting"exit 0+fi
Kairui, here we need restart the kdump service because of something changed in the system. If we exit here then the kdump initrd will be not updated for the change.
+timeout_time() {
echo $(($(stat -c "%Y" $throttle_lock) + $interval))+}
+# Wait until no one try to acquire the throttle_lock in past $interval seconds +while [[ $(timeout_time) -gt $(date +%s) ]]; do
sleep 1+done
+systemctl try-restart kdump
+exit 0 diff --git a/kexec-tools.spec b/kexec-tools.spec index 6330534..91d322f 100644 --- a/kexec-tools.spec +++ b/kexec-tools.spec @@ -29,6 +29,7 @@ Source24: kdump.sysconfig.ppc64le Source25: kdumpctl.8 Source26: live-image-kdump-howto.txt Source27: early-kdump-howto.txt +Source28: kdump-udev-throttler
####################################### # These are sources for mkdumpramfs @@ -171,6 +172,7 @@ install -m 755 %{SOURCE23} $RPM_BUILD_ROOT%{_prefix}/lib/kdump/kdump-lib-initram # For s390x the ELF header is created in the kdump kernel and therefore kexec # udev rules are not required install -m 644 %{SOURCE14} $RPM_BUILD_ROOT%{_udevrulesdir}/98-kexec.rules +install -m 755 %{SOURCE28} $RPM_BUILD_ROOT%{_udevrulesdir}/../kdump-udev-throttler %endif install -m 644 %{SOURCE15} $RPM_BUILD_ROOT%{_mandir}/man5/kdump.conf.5 install -m 644 %{SOURCE16} $RPM_BUILD_ROOT%{_unitdir}/kdump.service @@ -298,6 +300,7 @@ done %config(noreplace,missingok) %{_sysconfdir}/kdump.conf %ifnarch s390x %config %{_udevrulesdir} +%{_udevrulesdir}/../kdump-udev-throttler %endif %{dracutlibdir}/modules.d/* %dir %{_localstatedir}/crash -- 2.17.1
Thanks Dave
Hi Dave,
kdump-udev-throttler uses its own lock: /var/lock/kdump-udev-throttle, if another process is owning this lock, it means another service restart request is pending and the service will be restarted later (in less than 2 seconds). It's safe to exit here to avoid duplicated service restart request. This lock won't conflict from kdumpctl's lock.
But I just noticed this does have a risk of missing some udev event, the systemctl call later may block, and make the process holding two locks at the same time. I didn't meet the problem because if in past 2 seconds there are no udev events comming up, the system would most likely be settled already. I'll update the patch, thanks for the catch!
On Fri, Oct 19, 2018 at 5:38 PM Dave Young dyoung@redhat.com wrote:
Hi Kairui, On 10/19/18 at 04:24pm, Kairui Song wrote:
When doing CPU or memory hotplug, multiple udev events will be triggered together. This will bring unnecessary workload for the host and make udev take much longer to settle.
Previous commit f2883275 added a workaround to solve the problem of systemd failing kdump servive due to burst of restart requests, but the unnecessary workload is not reduced. And currently, this is causing timeout failure on Hyper-V environment.
This patch fixes the problem by merging concurrent kdump restart requests. Tested with Hyper-V VM, no new issues found.
Signed-off-by: Kairui Song kasong@redhat.com
Update from V1: Fix packaging problem, make kdump-udev-throttler executable and include it in main package.
98-kexec.rules | 8 ++++---- kdump-udev-throttler | 25 +++++++++++++++++++++++++ kexec-tools.spec | 3 +++ 3 files changed, 32 insertions(+), 4 deletions(-) create mode 100755 kdump-udev-throttler
diff --git a/98-kexec.rules b/98-kexec.rules index e32ee13..eee1df1 100644 --- a/98-kexec.rules +++ b/98-kexec.rules @@ -1,4 +1,4 @@ -SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service" +SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="memory", ACTION=="online", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="kdump-udev-throttler" diff --git a/kdump-udev-throttler b/kdump-udev-throttler new file mode 100755 index 0000000..272d226 --- /dev/null +++ b/kdump-udev-throttler @@ -0,0 +1,25 @@ +#!/bin/bash
+throttle_lock="/var/lock/kdump-udev-throttle" +interval=2
+exec 9>$throttle_lock +flock -n 9
+if [ $? -ne 0 ]; then
echo "Another app is currently holding the throttle lock, extend throttle timing and exiting"exit 0+fi
Kairui, here we need restart the kdump service because of something changed in the system. If we exit here then the kdump initrd will be not updated for the change.
+timeout_time() {
echo $(($(stat -c "%Y" $throttle_lock) + $interval))+}
+# Wait until no one try to acquire the throttle_lock in past $interval seconds +while [[ $(timeout_time) -gt $(date +%s) ]]; do
sleep 1+done
+systemctl try-restart kdump
+exit 0 diff --git a/kexec-tools.spec b/kexec-tools.spec index 6330534..91d322f 100644 --- a/kexec-tools.spec +++ b/kexec-tools.spec @@ -29,6 +29,7 @@ Source24: kdump.sysconfig.ppc64le Source25: kdumpctl.8 Source26: live-image-kdump-howto.txt Source27: early-kdump-howto.txt +Source28: kdump-udev-throttler
####################################### # These are sources for mkdumpramfs @@ -171,6 +172,7 @@ install -m 755 %{SOURCE23} $RPM_BUILD_ROOT%{_prefix}/lib/kdump/kdump-lib-initram # For s390x the ELF header is created in the kdump kernel and therefore kexec # udev rules are not required install -m 644 %{SOURCE14} $RPM_BUILD_ROOT%{_udevrulesdir}/98-kexec.rules +install -m 755 %{SOURCE28} $RPM_BUILD_ROOT%{_udevrulesdir}/../kdump-udev-throttler %endif install -m 644 %{SOURCE15} $RPM_BUILD_ROOT%{_mandir}/man5/kdump.conf.5 install -m 644 %{SOURCE16} $RPM_BUILD_ROOT%{_unitdir}/kdump.service @@ -298,6 +300,7 @@ done %config(noreplace,missingok) %{_sysconfdir}/kdump.conf %ifnarch s390x %config %{_udevrulesdir} +%{_udevrulesdir}/../kdump-udev-throttler %endif %{dracutlibdir}/modules.d/* %dir %{_localstatedir}/crash -- 2.17.1
Thanks Dave
-
Dave Young -
Kairui Song