When doing CPU or memory hotplug, multiple udev events will be triggered together. This will bring unnecessary workload for the host and make udev take much longer to settle.
Previous commit f2883275 added a workaround to solve the problem of systemd failing kdump servive due to burst of restart requests, but the unnecessary workload is not reduced. And currently, this is causing timeout failure on Hyper-V environment.
This patch fixes the problem by merging concurrent kdump restart requests. Tested with Hyper-V VM, no new issues found.
Signed-off-by: Kairui Song kasong@redhat.com
--- Update from V2: - A potential risk of missing some udev events by release the lock before calling systemctl - Add some inline comments for kdump-udev-throttler
Update from V1: Fix packaging problem, make kdump-udev-throttler executable and include it in main package.
98-kexec.rules | 8 +++---- kdump-udev-throttler | 50 ++++++++++++++++++++++++++++++++++++++++++++ kexec-tools.spec | 3 +++ 3 files changed, 57 insertions(+), 4 deletions(-) create mode 100755 kdump-udev-throttler
diff --git a/98-kexec.rules b/98-kexec.rules index e32ee13..eee1df1 100644 --- a/98-kexec.rules +++ b/98-kexec.rules @@ -1,4 +1,4 @@ -SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service" +SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="memory", ACTION=="online", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="kdump-udev-throttler" diff --git a/kdump-udev-throttler b/kdump-udev-throttler new file mode 100755 index 0000000..61a3580 --- /dev/null +++ b/kdump-udev-throttler @@ -0,0 +1,50 @@ +#!/bin/bash +# This util helps to reduce the workload of kdump service restarting +# on udev event. When hotplugging memory / CPU, multiple udev +# events may be triggered concurrently, and obviously, we don't want +# to restart kdump service for each event. + +# This script will be called by udev, and make sure kdump service is +# restart after all events we are watching are settled. + +# On each call, this script will update the timestamp of +# $throttle_lock and try to lock it. The first instance acquired the +# file lock will keep waiting for events to settle by checking the +# timestamp of $throttle_lock, other instances will just exit after +# updating the timestamp. In this way, we can make sure kdump service +# is restarted for exactly once on udev events. + + +throttle_lock="/var/lock/kdump-udev-throttle" +interval=2 + +exec 9>$throttle_lock +if [ $? -ne 0 ]; then + echo "kdump-udev-throttler: Failed to create the lock file! Fallback to non-throttled kdump service restart" + systemctl try-restart kdump + exit 1 +fi + +flock -n 9 +if [ $? -ne 0 ]; then + echo "kdump-udev-throttler: Another app is currently holding the throttle lock, extend throttle timing and exiting" + exit 0 +fi + +timeout_time() { + echo $(($(stat -c "%Y" $throttle_lock) + $interval)) +} + +# Wait until no one try to acquire the throttle_lock in past +# $interval seconds +while [[ $(timeout_time) -gt $(date +%s) ]]; do + sleep 1 +done + +# Release the lock, systemctl may block and make the process +# holding two locks at the same time and we might miss some events +exec 9>&- + +systemctl try-restart kdump + +exit 0 diff --git a/kexec-tools.spec b/kexec-tools.spec index 6330534..91d322f 100644 --- a/kexec-tools.spec +++ b/kexec-tools.spec @@ -29,6 +29,7 @@ Source24: kdump.sysconfig.ppc64le Source25: kdumpctl.8 Source26: live-image-kdump-howto.txt Source27: early-kdump-howto.txt +Source28: kdump-udev-throttler
####################################### # These are sources for mkdumpramfs @@ -171,6 +172,7 @@ install -m 755 %{SOURCE23} $RPM_BUILD_ROOT%{_prefix}/lib/kdump/kdump-lib-initram # For s390x the ELF header is created in the kdump kernel and therefore kexec # udev rules are not required install -m 644 %{SOURCE14} $RPM_BUILD_ROOT%{_udevrulesdir}/98-kexec.rules +install -m 755 %{SOURCE28} $RPM_BUILD_ROOT%{_udevrulesdir}/../kdump-udev-throttler %endif install -m 644 %{SOURCE15} $RPM_BUILD_ROOT%{_mandir}/man5/kdump.conf.5 install -m 644 %{SOURCE16} $RPM_BUILD_ROOT%{_unitdir}/kdump.service @@ -298,6 +300,7 @@ done %config(noreplace,missingok) %{_sysconfdir}/kdump.conf %ifnarch s390x %config %{_udevrulesdir} +%{_udevrulesdir}/../kdump-udev-throttler %endif %{dracutlibdir}/modules.d/* %dir %{_localstatedir}/crash
self-nack, this patch still have issues: - having udev.children-max=1, it will block for a very long time - kdump service may get restart unexpectedly on system bootup On Fri, Oct 19, 2018 at 6:24 PM Kairui Song kasong@redhat.com wrote:
When doing CPU or memory hotplug, multiple udev events will be triggered together. This will bring unnecessary workload for the host and make udev take much longer to settle.
Previous commit f2883275 added a workaround to solve the problem of systemd failing kdump servive due to burst of restart requests, but the unnecessary workload is not reduced. And currently, this is causing timeout failure on Hyper-V environment.
This patch fixes the problem by merging concurrent kdump restart requests. Tested with Hyper-V VM, no new issues found.
Signed-off-by: Kairui Song kasong@redhat.com
Update from V2:
- A potential risk of missing some udev events by release the lock before calling systemctl
- Add some inline comments for kdump-udev-throttler
Update from V1: Fix packaging problem, make kdump-udev-throttler executable and include it in main package.
98-kexec.rules | 8 +++---- kdump-udev-throttler | 50 ++++++++++++++++++++++++++++++++++++++++++++ kexec-tools.spec | 3 +++ 3 files changed, 57 insertions(+), 4 deletions(-) create mode 100755 kdump-udev-throttler
diff --git a/98-kexec.rules b/98-kexec.rules index e32ee13..eee1df1 100644 --- a/98-kexec.rules +++ b/98-kexec.rules @@ -1,4 +1,4 @@ -SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service" +SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="memory", ACTION=="online", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="kdump-udev-throttler" diff --git a/kdump-udev-throttler b/kdump-udev-throttler new file mode 100755 index 0000000..61a3580 --- /dev/null +++ b/kdump-udev-throttler @@ -0,0 +1,50 @@ +#!/bin/bash +# This util helps to reduce the workload of kdump service restarting +# on udev event. When hotplugging memory / CPU, multiple udev +# events may be triggered concurrently, and obviously, we don't want +# to restart kdump service for each event.
+# This script will be called by udev, and make sure kdump service is +# restart after all events we are watching are settled.
+# On each call, this script will update the timestamp of +# $throttle_lock and try to lock it. The first instance acquired the +# file lock will keep waiting for events to settle by checking the +# timestamp of $throttle_lock, other instances will just exit after +# updating the timestamp. In this way, we can make sure kdump service +# is restarted for exactly once on udev events.
+throttle_lock="/var/lock/kdump-udev-throttle" +interval=2
+exec 9>$throttle_lock +if [ $? -ne 0 ]; then
echo "kdump-udev-throttler: Failed to create the lock file! Fallback to non-throttled kdump service restart"systemctl try-restart kdumpexit 1+fi
+flock -n 9 +if [ $? -ne 0 ]; then
echo "kdump-udev-throttler: Another app is currently holding the throttle lock, extend throttle timing and exiting"exit 0+fi
+timeout_time() {
echo $(($(stat -c "%Y" $throttle_lock) + $interval))+}
+# Wait until no one try to acquire the throttle_lock in past +# $interval seconds +while [[ $(timeout_time) -gt $(date +%s) ]]; do
sleep 1+done
+# Release the lock, systemctl may block and make the process +# holding two locks at the same time and we might miss some events +exec 9>&-
+systemctl try-restart kdump
+exit 0 diff --git a/kexec-tools.spec b/kexec-tools.spec index 6330534..91d322f 100644 --- a/kexec-tools.spec +++ b/kexec-tools.spec @@ -29,6 +29,7 @@ Source24: kdump.sysconfig.ppc64le Source25: kdumpctl.8 Source26: live-image-kdump-howto.txt Source27: early-kdump-howto.txt +Source28: kdump-udev-throttler
####################################### # These are sources for mkdumpramfs @@ -171,6 +172,7 @@ install -m 755 %{SOURCE23} $RPM_BUILD_ROOT%{_prefix}/lib/kdump/kdump-lib-initram # For s390x the ELF header is created in the kdump kernel and therefore kexec # udev rules are not required install -m 644 %{SOURCE14} $RPM_BUILD_ROOT%{_udevrulesdir}/98-kexec.rules +install -m 755 %{SOURCE28} $RPM_BUILD_ROOT%{_udevrulesdir}/../kdump-udev-throttler %endif install -m 644 %{SOURCE15} $RPM_BUILD_ROOT%{_mandir}/man5/kdump.conf.5 install -m 644 %{SOURCE16} $RPM_BUILD_ROOT%{_unitdir}/kdump.service @@ -298,6 +300,7 @@ done %config(noreplace,missingok) %{_sysconfdir}/kdump.conf %ifnarch s390x %config %{_udevrulesdir} +%{_udevrulesdir}/../kdump-udev-throttler %endif %{dracutlibdir}/modules.d/* %dir %{_localstatedir}/crash -- 2.17.1
-
Kairui Song