kexec October 2018

kexec@lists.fedoraproject.org

5 participants
11 discussions

[PATCH v3 0/3] Fix serveral problems with kdump reloading on udev event
by Kairui Song 31 Oct '18

31 Oct '18

Currently kdump will restart for multiple times and may trigger a initramfs rebuild on memory / CPU hotplug if any configuration is modified. Besides, currently simply restart the service via systemctl directly will take a period of time as well, as kdumpctl will do a lot of config checking, which is not appreciated by udev, udev want everything to get done in a very short time. And kdumpctl may get kill for several times because systemd will kill the previous service starting process for later restart request. This may introduce extra risk too. Simply add reload support and use systemdctl reload will not work well either because systemd will ignore later reload request, so kdump may get reload earlier than the hotplug is finished. This patch series should solve the problem by applying following fixes: - Adding reload support as a fast path to reload kdump resources, bypassing most checking. So the reload progress will be faster and never trigger a initrmafs rebuild. - Let the reload request run in async mode, so udev will never get blocked. - Throttle udev events to avoid unnecessary kdump reloading and make sure kdump will reload after the udev settled. Update from V2: - Use "-ne" instead of "!=" for comparing return values of functions in kdumpctl. "!=" is used more for pattern matching. - Removed duplicated "is-system-running" check in kdump-udev-throttler - Add more detailed commit message about reason to use --no-block in udev rules Update from V1: - Add cover letter - Add --no-block option to systemd-run, starting from systemd-v220, systemd-run will run in synchronous mode by default, need to give --no-block explicitly to run in async mode. Kairui Song (3): kdumpctl: Add reload support Rewrite kdump's udev rules Throttle kdump reload request triggered by udev event 98-kexec.rules | 16 +++++++++--- kdump-udev-throttler | 47 +++++++++++++++++++++++++++++++++++ kdump.service | 1 + kdumpctl | 59 +++++++++++++++++++++++++++++++++++++------- kexec-tools.spec | 3 +++ 5 files changed, 113 insertions(+), 13 deletions(-) create mode 100755 kdump-udev-throttler -- 2.17.1

2 4

[PATCH v2 0/3] Fix serveral problems with kdump reloading on
by Kairui Song 29 Oct '18

29 Oct '18

Currently kdump will restart for multiple times and may trigger a initramfs rebuild on memory / CPU hotplug if any configuration is modified. Besides, currently simply restart the service via systemctl directly will take a period of time as well, as kdumpctl will do a lot of config checking, which is not appreciated by udev, udev want everything to get done in a very short time. And kdumpctl may get kill for several times because systemd will kill the previous service starting process for later restart request. This may introduce extra risk too. Simply add reload support and use systemdctl reload will not work well either because systemd will ignore later reload request, so kdump may get reload earlier than the hotplug is finished. This patch series should solve the problem by applying following fixes: - Adding reload support as a fast path to reload kdump resources, bypassing most checking. So the reload progress will be faster and never trigger a initrmafs rebuild. - Let the reload request run in async mode, so udev will never get blocked. - Throttle udev events to avoid unnecessary kdump reloading and make sure kdump will reload after the udev settled. Update from V1: - Add cover letter - Add --no-block option to systemd-run, starting from systemd-v220, systemd-run will run in synchronous mode by default, need to give --no-block explicitly to run in async mode. Kairui Song (3): kdumpctl: Add reload support Rewrite kdump's udev rules Throttle kdump reload request triggered by udev event 98-kexec.rules | 16 +++++++++--- kdump-udev-throttler | 54 ++++++++++++++++++++++++++++++++++++++++ kdump.service | 1 + kdumpctl | 59 +++++++++++++++++++++++++++++++++++++------- kexec-tools.spec | 3 +++ 5 files changed, 120 insertions(+), 13 deletions(-) create mode 100755 kdump-udev-throttler -- 2.17.1

3 11

[PATCH 1/3] kdumpctl: Add reload support
by Kairui Song 24 Oct '18

24 Oct '18

Add reload support to kdumpctl, reload will simply unload current loaded kexec crash kernel and initramfs, and load it again. Changes in /etc/sysconfig/kdump will take effect with kdumpctl reload, but reloading will not check the content of /etc/kdump.conf and won't rebuild anything. reload is fast, the only time-consuming part of kdumpctl reload is loading kernel and initramfs with kexec which is always necessary. Signed-off-by: Kairui Song <kasong(a)redhat.com> --- kdump.service | 1 + kdumpctl | 59 +++++++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 51 insertions(+), 9 deletions(-) diff --git a/kdump.service b/kdump.service index 5144597..f888dd6 100644 --- a/kdump.service +++ b/kdump.service @@ -7,6 +7,7 @@ DefaultDependencies=no Type=oneshot ExecStart=/usr/bin/kdumpctl start ExecStop=/usr/bin/kdumpctl stop +ExecReload=/usr/bin/kdumpctl reload RemainAfterExit=yes StartLimitInterval=0 diff --git a/kdumpctl b/kdumpctl index ece406f..5e139e9 100755 --- a/kdumpctl +++ b/kdumpctl @@ -283,6 +283,16 @@ get_pcs_cluster_modified_files() setup_initrd() { + KDUMP_BOOTDIR=$(check_boot_dir "${KDUMP_BOOTDIR}") + + if [ -z "$KDUMP_KERNELVER" ]; then + kdump_kver=`uname -r` + else + kdump_kver=$KDUMP_KERNELVER + fi + + kdump_kernel="${KDUMP_BOOTDIR}/${KDUMP_IMG}-${kdump_kver}${KDUMP_IMG_EXT}" + DEFAULT_INITRD="${KDUMP_BOOTDIR}/initramfs-`uname -r`.img" DEFAULT_INITRD_BAK="${KDUMP_BOOTDIR}/.initramfs-`uname -r`.img.default" if [ $DEFAULT_DUMP_MODE == "fadump" ]; then @@ -533,16 +543,8 @@ check_rebuild() local _force_no_rebuild force_no_rebuild="0" local ret system_modified="0" - KDUMP_BOOTDIR=$(check_boot_dir "${KDUMP_BOOTDIR}") - - if [ -z "$KDUMP_KERNELVER" ]; then - kdump_kver=`uname -r` - else - kdump_kver=$KDUMP_KERNELVER - fi - - kdump_kernel="${KDUMP_BOOTDIR}/${KDUMP_IMG}-${kdump_kver}${KDUMP_IMG_EXT}" setup_initrd + if [ $? -ne 0 ]; then return 1 fi @@ -1004,6 +1006,42 @@ start() echo "Starting kdump: [OK]" } +reload() +{ + check_current_status + if [ $? == 1 ]; then + echo "Kdump is not running: [WARNING]" + return 0 + fi + + if [ $DEFAULT_DUMP_MODE == "fadump" ]; then + stop_fadump + else + stop_kdump + fi + + if [ $? != 0 ]; then + echo "Stopping kdump: [FAILED]" + return 1 + fi + + echo "Stopping kdump: [OK]" + + setup_initrd + if [ $? != 0 ]; then + echo "Starting kdump: [FAILED]" + return 1 + fi + + start_dump + if [ $? != 0 ]; then + echo "Starting kdump: [FAILED]" + return 1 + fi + + echo "Starting kdump: [OK]" +} + stop_fadump() { echo 0 > $FADUMP_REGISTER_SYS_NODE @@ -1087,6 +1125,9 @@ main () esac exit $EXIT_CODE ;; + reload) + reload + ;; restart) stop start -- 2.17.1

1 2

[PATCH v3] Merge concurrent kdump restart requests triggered by udev events
by Kairui Song 23 Oct '18

23 Oct '18

When doing CPU or memory hotplug, multiple udev events will be triggered together. This will bring unnecessary workload for the host and make udev take much longer to settle. Previous commit f2883275 added a workaround to solve the problem of systemd failing kdump servive due to burst of restart requests, but the unnecessary workload is not reduced. And currently, this is causing timeout failure on Hyper-V environment. This patch fixes the problem by merging concurrent kdump restart requests. Tested with Hyper-V VM, no new issues found. Signed-off-by: Kairui Song <kasong(a)redhat.com> --- Update from V2: - A potential risk of missing some udev events by release the lock before calling systemctl - Add some inline comments for kdump-udev-throttler Update from V1: Fix packaging problem, make kdump-udev-throttler executable and include it in main package. 98-kexec.rules | 8 +++---- kdump-udev-throttler | 50 ++++++++++++++++++++++++++++++++++++++++++++ kexec-tools.spec | 3 +++ 3 files changed, 57 insertions(+), 4 deletions(-) create mode 100755 kdump-udev-throttler diff --git a/98-kexec.rules b/98-kexec.rules index e32ee13..eee1df1 100644 --- a/98-kexec.rules +++ b/98-kexec.rules @@ -1,4 +1,4 @@ -SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service" +SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="memory", ACTION=="online", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="kdump-udev-throttler" diff --git a/kdump-udev-throttler b/kdump-udev-throttler new file mode 100755 index 0000000..61a3580 --- /dev/null +++ b/kdump-udev-throttler @@ -0,0 +1,50 @@ +#!/bin/bash +# This util helps to reduce the workload of kdump service restarting +# on udev event. When hotplugging memory / CPU, multiple udev +# events may be triggered concurrently, and obviously, we don't want +# to restart kdump service for each event. + +# This script will be called by udev, and make sure kdump service is +# restart after all events we are watching are settled. + +# On each call, this script will update the timestamp of +# $throttle_lock and try to lock it. The first instance acquired the +# file lock will keep waiting for events to settle by checking the +# timestamp of $throttle_lock, other instances will just exit after +# updating the timestamp. In this way, we can make sure kdump service +# is restarted for exactly once on udev events. + + +throttle_lock="/var/lock/kdump-udev-throttle" +interval=2 + +exec 9>$throttle_lock +if [ $? -ne 0 ]; then + echo "kdump-udev-throttler: Failed to create the lock file! Fallback to non-throttled kdump service restart" + systemctl try-restart kdump + exit 1 +fi + +flock -n 9 +if [ $? -ne 0 ]; then + echo "kdump-udev-throttler: Another app is currently holding the throttle lock, extend throttle timing and exiting" + exit 0 +fi + +timeout_time() { + echo $(($(stat -c "%Y" $throttle_lock) + $interval)) +} + +# Wait until no one try to acquire the throttle_lock in past +# $interval seconds +while [[ $(timeout_time) -gt $(date +%s) ]]; do + sleep 1 +done + +# Release the lock, systemctl may block and make the process +# holding two locks at the same time and we might miss some events +exec 9>&- + +systemctl try-restart kdump + +exit 0 diff --git a/kexec-tools.spec b/kexec-tools.spec index 6330534..91d322f 100644 --- a/kexec-tools.spec +++ b/kexec-tools.spec @@ -29,6 +29,7 @@ Source24: kdump.sysconfig.ppc64le Source25: kdumpctl.8 Source26: live-image-kdump-howto.txt Source27: early-kdump-howto.txt +Source28: kdump-udev-throttler ####################################### # These are sources for mkdumpramfs @@ -171,6 +172,7 @@ install -m 755 %{SOURCE23} $RPM_BUILD_ROOT%{_prefix}/lib/kdump/kdump-lib-initram # For s390x the ELF header is created in the kdump kernel and therefore kexec # udev rules are not required install -m 644 %{SOURCE14} $RPM_BUILD_ROOT%{_udevrulesdir}/98-kexec.rules +install -m 755 %{SOURCE28} $RPM_BUILD_ROOT%{_udevrulesdir}/../kdump-udev-throttler %endif install -m 644 %{SOURCE15} $RPM_BUILD_ROOT%{_mandir}/man5/kdump.conf.5 install -m 644 %{SOURCE16} $RPM_BUILD_ROOT%{_unitdir}/kdump.service @@ -298,6 +300,7 @@ done %config(noreplace,missingok) %{_sysconfdir}/kdump.conf %ifnarch s390x %config %{_udevrulesdir} +%{_udevrulesdir}/../kdump-udev-throttler %endif %{dracutlibdir}/modules.d/* %dir %{_localstatedir}/crash -- 2.17.1

1 1

[PATCH v2] Merge concurrent kdump restart requests triggered by udev events
by Kairui Song 19 Oct '18

19 Oct '18

When doing CPU or memory hotplug, multiple udev events will be triggered together. This will bring unnecessary workload for the host and make udev take much longer to settle. Previous commit f2883275 added a workaround to solve the problem of systemd failing kdump servive due to burst of restart requests, but the unnecessary workload is not reduced. And currently, this is causing timeout failure on Hyper-V environment. This patch fixes the problem by merging concurrent kdump restart requests. Tested with Hyper-V VM, no new issues found. Signed-off-by: Kairui Song <kasong(a)redhat.com> --- Update from V1: Fix packaging problem, make kdump-udev-throttler executable and include it in main package. 98-kexec.rules | 8 ++++---- kdump-udev-throttler | 25 +++++++++++++++++++++++++ kexec-tools.spec | 3 +++ 3 files changed, 32 insertions(+), 4 deletions(-) create mode 100755 kdump-udev-throttler diff --git a/98-kexec.rules b/98-kexec.rules index e32ee13..eee1df1 100644 --- a/98-kexec.rules +++ b/98-kexec.rules @@ -1,4 +1,4 @@ -SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service" +SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="memory", ACTION=="online", PROGRAM="kdump-udev-throttler" +SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="kdump-udev-throttler" diff --git a/kdump-udev-throttler b/kdump-udev-throttler new file mode 100755 index 0000000..272d226 --- /dev/null +++ b/kdump-udev-throttler @@ -0,0 +1,25 @@ +#!/bin/bash + +throttle_lock="/var/lock/kdump-udev-throttle" +interval=2 + +exec 9>$throttle_lock +flock -n 9 + +if [ $? -ne 0 ]; then + echo "Another app is currently holding the throttle lock, extend throttle timing and exiting" + exit 0 +fi + +timeout_time() { + echo $(($(stat -c "%Y" $throttle_lock) + $interval)) +} + +# Wait until no one try to acquire the throttle_lock in past $interval seconds +while [[ $(timeout_time) -gt $(date +%s) ]]; do + sleep 1 +done + +systemctl try-restart kdump + +exit 0 diff --git a/kexec-tools.spec b/kexec-tools.spec index 6330534..91d322f 100644 --- a/kexec-tools.spec +++ b/kexec-tools.spec @@ -29,6 +29,7 @@ Source24: kdump.sysconfig.ppc64le Source25: kdumpctl.8 Source26: live-image-kdump-howto.txt Source27: early-kdump-howto.txt +Source28: kdump-udev-throttler ####################################### # These are sources for mkdumpramfs @@ -171,6 +172,7 @@ install -m 755 %{SOURCE23} $RPM_BUILD_ROOT%{_prefix}/lib/kdump/kdump-lib-initram # For s390x the ELF header is created in the kdump kernel and therefore kexec # udev rules are not required install -m 644 %{SOURCE14} $RPM_BUILD_ROOT%{_udevrulesdir}/98-kexec.rules +install -m 755 %{SOURCE28} $RPM_BUILD_ROOT%{_udevrulesdir}/../kdump-udev-throttler %endif install -m 644 %{SOURCE15} $RPM_BUILD_ROOT%{_mandir}/man5/kdump.conf.5 install -m 644 %{SOURCE16} $RPM_BUILD_ROOT%{_unitdir}/kdump.service @@ -298,6 +300,7 @@ done %config(noreplace,missingok) %{_sysconfdir}/kdump.conf %ifnarch s390x %config %{_udevrulesdir} +%{_udevrulesdir}/../kdump-udev-throttler %endif %{dracutlibdir}/modules.d/* %dir %{_localstatedir}/crash -- 2.17.1

2 2

[PATCH] Merge concurrent kdump restart requests triggered by udev events
by Kairui Song 19 Oct '18

19 Oct '18

1 0

[PATCH] kdumpctl: busybox hostname does not have -I
by Rantala, Tommi T. (Nokia - FI/Espoo) 18 Oct '18

18 Oct '18

Busybox hostname does not (currently) have -I, so also try with -i. Signed-off-by: Tommi Rantala <tommi.t.rantala(a)nokia.com> --- kdumpctl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kdumpctl b/kdumpctl index ece406f..9be2b9f 100755 --- a/kdumpctl +++ b/kdumpctl @@ -881,7 +881,7 @@ selinux_relabel() check_fence_kdump_config() { local hostname=`hostname` - local ipaddrs=`hostname -I` + local ipaddrs=`hostname -I 2>/dev/null || hostname -i` local nodes=$(get_option_value "fence_kdump_nodes") for node in $nodes; do -- 2.17.1

1 0

[PATCH v2] kdumpctl: Print warning in case the raw device is formatted and contains filesystem.
by Kenneth Dsouza 15 Oct '18

15 Oct '18

From: Kenneth D'souza <kdsouza(a)redhat.com> Changes since v1: Added warning instead of error. Currently the kdumpctl script doesn't check if the raw device is formatted which might destroy existing data at the time of dump capture. This patch addresses this issue, by ensuring kdumpctl prints a warning in case it finds the raw device to be formatted. Signed-off-by: Kenneth D'souza <kdsouza(a)redhat.com> --- kdumpctl | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/kdumpctl b/kdumpctl index 6a01c13..201586c 100755 --- a/kdumpctl +++ b/kdumpctl @@ -779,6 +779,11 @@ save_raw() echo "raw partition $raw_target not found" return 1 } + check_fs=$(lsblk --nodeps -npo FSTYPE $raw_target) + if [[ $(echo $check_fs | wc -w) -ne 0 ]]; then + echo "Warning: Detected '$check_fs' signature on $raw_target, data loss is expected." + return 0 + fi kdump_dir=`grep ^path $KDUMP_CONFIG_FILE | cut -d' ' -f2-` if [ -z "${kdump_dir}" ]; then coredir="/var/crash/`date +"%Y-%m-%d-%H:%M"`" -- 2.14.3

2 2

[PATCH v3] kdump-lib-initramfs.sh: Add check to remount to rw mode only if dump target is ro.
by Kenneth Dsouza 12 Oct '18

12 Oct '18

From: Kenneth D'souza <kdsouza(a)redhat.com> Currently the script does not check if the dump target is read-only and would always mount to read-write mode. This caused an issue with nfs mount as the fstab options would be reconsidered while remounting to read-write mode. The remount would fail with the below error as all options cannot be changed runtime. mount.nfs: mount(2): Invalid argument mount.nfs: an incorrect mount option was specified Which in result would not save the vmcore on the dump target. This patch addresses this issue by checking the dump target status for read-only. If yes, remount to read-write mode without reconsidering the fstab options. Signed-off-by: Kenneth D'souza <kdsouza(a)redhat.com> --- kdump-lib-initramfs.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 2c18c87..0c1e233 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -87,6 +87,7 @@ dump_fs() local _dev=$(findmnt -k -f -n -r -o SOURCE $1) local _mp=$(findmnt -k -f -n -r -o TARGET $1) + local _op=$(findmnt -k -f -n -r -o OPTIONS $1) echo "kdump: dump target is $_dev" @@ -100,7 +101,12 @@ dump_fs() echo "kdump: saving to $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/" - mount -o remount,rw $_mp || return 1 + # Only remount to read-write mode if the dump target is mounted read-only. + if [[ "$_op" = "ro"* ]]; then + echo "kdump: Mounting Dump target $_dev in rw mode." + mount -o remount,rw $dev $_mp || return 1 + fi + mkdir -p $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR || return 1 save_vmcore_dmesg_fs ${DMESG_COLLECTOR} "$_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/" -- 2.14.3

2 1

[PATCH v2] kdump-lib-initramfs.sh: Add check to remount to rw mode only if dump target is ro.
by Kenneth Dsouza 11 Oct '18

11 Oct '18

From: Kenneth D'souza <kdsouza(a)redhat.com> Currently the script does not check if the dump target is read-only and would always mount to read-write mode. This caused an issue with nfs mount as the fstab options would be reconsidered while remounting to read-write mode. The remount would fail with the below error as all options cannot be changed runtime. mount.nfs: mount(2): Invalid argument mount.nfs: an incorrect mount option was specified Which in result would not save the vmcore on the dump target. This patch addresses this issue by checking the dump target status for read-only. If yes, remount to read-write mode without reconsidering the fstab options. Signed-off-by: Kenneth D'souza <kdsouza(a)redhat.com> --- kdump-lib-initramfs.sh | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 2c18c87..bc74871 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -87,6 +87,7 @@ dump_fs() local _dev=$(findmnt -k -f -n -r -o SOURCE $1) local _mp=$(findmnt -k -f -n -r -o TARGET $1) + local _op=$(findmnt -k -f -n -r -o OPTIONS $1) echo "kdump: dump target is $_dev" @@ -100,7 +101,13 @@ dump_fs() echo "kdump: saving to $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/" - mount -o remount,rw $_mp || return 1 + # Only remount to read-write mode if the dump target is mounted read-only. + check_status=$(echo $_op | grep -w rw) + if [ -z "$check_status" ]; then + echo "kdump: Mounting Dump target $_dev in rw mode." + mount -o remount,rw $dev $_mp || return 1 + fi + mkdir -p $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR || return 1 save_vmcore_dmesg_fs ${DMESG_COLLECTOR} "$_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/" -- 2.14.3

1 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

kexec October 2018