kexec July 2017

kexec@lists.fedoraproject.org

9 participants
18 discussions

[PATCH v2] mkdumprd: use 300s as the default systemd unit timeout for kdump mount
by Xunlei Pang 01 Aug '17

01 Aug '17

Currently, systemd uses 90s as the default mount unit timeout, in some cases, although it works well in 1st kernel, it's not enough under kdump and results in mount timeout, further results in kdump dumping failure. We've met several such issues, we decided to enlarge this default value a little for kdump. We finalize "x-systemd.device-timeout=300" as the default timeout to the mount options if there is no "x-systemd.device-timeout=X" specified. It can be overridden by /etc/fstab mount options,etc, so that users can specify other timeout values if they want to. Note: this is different from rd.timeout which was introduced by dracut initqueue. Signed-off-by: Xunlei Pang <xlpang(a)redhat.com> --- v1->v2: -Use 300s other than 600s as the default value. mkdumprd | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mkdumprd b/mkdumprd index d3ecbd6..29e51a9 100644 --- a/mkdumprd +++ b/mkdumprd @@ -104,6 +104,12 @@ to_mount() { _options=$(echo $_options | sed 's/noauto//') _options=${_options/#ro/rw} #mount fs target as rw in 2nd kernel + # kdump turns out to require longer default systemd mount timeout + # than 1st kernel(90s by default), we choose 300s for kdump. + if ! strstr $_options "x-systemd.device-timeout"; then + _options="$_options,x-systemd.device-timeout=300" + fi + _mntopts="$_target $_fstype $_options" #for non-nfs _dev converting to use udev persistent name if [ -b "$_source" ]; then -- 1.8.3.1

4 6

[PATCH] mkdumprd: use 600s as the default systemd unit timeout for kdump mount
by Xunlei Pang 30 Jul '17

30 Jul '17

Currently, systemd uses 90s as the default mount unit timeout, in some cases, it's not enough and results in mount timeout further results in kdump dumping failure, but the device can actually be ready after a while. We've met several such issues. So, we add a "x-systemd.device-timeout=600"(600s should be long enough) as the default timeout to the mount options if there is no "x-systemd.device-timeout=X" specified. It can be overridden by /etc/fstab mount options, so that users can specify other timeout values if they want to. Note: this is different from rd.timeout which was introduced by dracut initqueue. Signed-off-by: Xunlei Pang <xlpang(a)redhat.com> --- mkdumprd | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mkdumprd b/mkdumprd index d3ecbd6..30f8ba6 100644 --- a/mkdumprd +++ b/mkdumprd @@ -104,6 +104,11 @@ to_mount() { _options=$(echo $_options | sed 's/noauto//') _options=${_options/#ro/rw} #mount fs target as rw in 2nd kernel + # use 600s as default systemd mount timeout if none + if ! strstr $_options "x-systemd.device-timeout"; then + _options="$_options,x-systemd.device-timeout=600" + fi + _mntopts="$_target $_fstype $_options" #for non-nfs _dev converting to use udev persistent name if [ -b "$_source" ]; then -- 1.8.3.1

4 17

[PATCH V2] Improve 'cpu add' udev rules
by Pratyush Anand 30 Jul '17

30 Jul '17

Currently kdump service is restarted even when any new file is added in cpu subsystem. So, it can be restarted multiple times in the cases like loading of acpi_cpufreq module or online/offline of any cpu. However, we should see kdump service restart only once in case a new CPU is added or removed. cpu crash notes buffer is created when a new CPU is added. It's location does not change when a CPU is onlined/offlined or acpi_cpufreq driver is loaded. Therefore, no need to restart kdump service in such cases. Thus, we need to introduce an extra filter for the kernel name of the directory created by cpu_add which is KERNEL=="cpu[0-9]*". This will ensure that kdump service is not restarted when any new file is added in /removed from cpu subsystem. Signed-off-by: Pratyush Anand <panand(a)redhat.com> --- - Although, I have not reproduced this issue on fedora,rather it was in RHEL6, but it looks like that we should better have this modified rules, which is more justified. Changes since v1: - v1 used to filter with subsystem as 'acpi',which might not work on playform like powerpc where they have non-acpi method of cpu addition. 98-kexec.rules | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/98-kexec.rules b/98-kexec.rules index e32ee13cb2e8..5dcf9fae419a 100644 --- a/98-kexec.rules +++ b/98-kexec.rules @@ -1,4 +1,4 @@ -SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump.service" +SUBSYSTEM=="cpu", ACTION=="add", KERNEL=="cpu[0-9]*", PROGRAM="/bin/systemctl try-restart kdump.service" +SUBSYSTEM=="cpu", ACTION=="remove", KERNEL=="cpu[0-9]*", PROGRAM="/bin/systemctl try-restart kdump.service" SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service" SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service" -- 2.9.4

2 1

[PATCH] kdump-lib.sh: support dump ssh via local ipv6 address
by Ziyue Yang 26 Jul '17

26 Jul '17

When dumping to ssh via local ipv6 address, the ssh parameter in kdump.conf is supposed to have the form like ssh user@fe80::cc1:8bff:fe90:b95f%eth0 where "%eth0" is an existing network interface supporting ipv6. However, the get_remote_host function in kdump-lib.sh currently doesn't remove the network interface in the ipv6 address, causing the ip command in kdump_install_net function to fail, leading to a "Bad kdump location" message. This patch supports dumping ssh via local ipv6 address by enabling get_remote_host function to remove network interface in ipv6 addresses if there are any. Signed-off-by: Ziyue Yang <ziyang(a)redhat.com> --- kdump-lib.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kdump-lib.sh b/kdump-lib.sh index 3f0af91..cca68e5 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -323,6 +323,10 @@ get_remote_host() _config_val=${_config_val%:/*} _config_val=${_config_val#[} _config_val=${_config_val%]} + # factor out network inteface segment in local ipv6 address + if is_ipv6_address $_config_val; then + _config_val=${_config_val%%%*} + fi echo $_config_val } -- 2.9.3

3 6

[PATCH] module-setup: suppress the early iscsi error messages
by Xunlei Pang 25 Jul '17

25 Jul '17

Currently, we throw the error message at the very beginning, as a result on a pure-hardware(all-offload) iscsi machine with many iscsi partitions, we suffered from too much noise as follows: iscsiadm: No records found Unable to find iscsi record for /sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.1/host0/session1 iscsiadm: No records found Unable to find iscsi record for /sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.3/host1/session2 iscsiadm: No records found Unable to find iscsi record for /sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.1/host0/session1 iscsiadm: No records found Unable to find iscsi record for /sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.3/host1/session2 iscsiadm: No records found Unable to find iscsi record for /sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.1/host0/session1 iscsiadm: No records found Unable to find iscsi record for /sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.3/host1/session2 iscsiadm: No records found Unable to find iscsi record for /sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.1/host0/session1 iscsiadm: No records found Unable to find iscsi record for /sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.3/host1/session2 iscsiadm: No records found Unable to find iscsi record for /sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.1/host0/session1 iscsiadm: No records found Unable to find iscsi record for /sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.3/host1/session2 kexec: loaded kdump kernel Starting kdump: [OK] There's no need to know the very early error messages, we can remove the error output which is actually normal for the pure hardware iscsi. As for unexpected errors, we kept the error outputs in the succeeding kdump_iscsi_get_rec_val() calls by not appending "2>/dev/null". Signed-off-by: Xunlei Pang <xlpang(a)redhat.com> --- dracut-module-setup.sh | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index 59b4f04..43e3f2a 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -545,11 +545,9 @@ kdump_setup_iscsi_device() { dinfo "Found iscsi component $1" - # Check once before getting explicit values, so we can output a decent - # error message. - - if ! /sbin/iscsiadm -m session -r ${path} >/dev/null ; then - derror "Unable to find iscsi record for $path" + # Check once before getting explicit values, so we can bail out early, + # e.g. in case of pure-hardware(all-offload) iscsi. + if ! /sbin/iscsiadm -m session -r ${path} &>/dev/null ; then return 1 fi -- 1.8.3.1

2 1

[PATCH] Improve 'cpu add' udev rules
by Pratyush Anand 18 Jul '17

18 Jul '17

Currently kdump service is restarted even when any new file is added in cpu subsystem. So, it can be restarted multiple times in the cases like loading of acpi_cpufreq module or online/offline of any cpu. However, we should see kdump service restart only once in case a new CPU is added or removed. cpu crash notes buffer is created when a new CPU is added. It's location does not change when a CPU is onlined/offlined or acpi_cpufreq driver is loaded. Therefore, no need to restart kdump service in such cases. IMO, there is only one processor driver (drivers/acpi/processor_core.c) which can enumerate a hot added cpu. An `udevadm info` enquiry to acpi device created by this driver gives: # udevadm info -a -p /sys/bus/acpi/devices/LNXCPU\:00/ looking at device '/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0010:00/LNXCPU:00': KERNEL=="LNXCPU:00" SUBSYSTEM=="acpi" Therefore, this patch modifies the rule to filter such devices. Signed-off-by: Pratyush Anand <panand(a)redhat.com> --- 98-kexec.rules | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/98-kexec.rules b/98-kexec.rules index e32ee13cb2e8..d43760e38f8f 100644 --- a/98-kexec.rules +++ b/98-kexec.rules @@ -1,4 +1,4 @@ -SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump.service" -SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump.service" +SUBSYSTEM=="acpi", ACTION=="add", KERNEL=="LNXCPU*", PROGRAM="/etc/init.d/kdump condrestart" +SUBSYSTEM=="acpi", ACTION=="remove", KERNEL=="LNXCPU*", PROGRAM="/etc/init.d/kdump condrestart" SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service" SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service" -- 2.9.4

2 3

[PATCH] mkdumprd: remove useless "x-initrd.mount"
by Xunlei Pang 18 Jul '17

18 Jul '17

After the following systemd commit, "x-initrd.mount" option became useless actually, we can safely remove it now. commit ce3f6d82b003f365f718f24e48f55b8a0372b924 Author: nmartensen <nis.martensen(a)web.de> Date: Fri Jan 15 07:55:25 2016 +0100 fstab-generator: remove bogus condition The sysroot mount is already taken care of by the add_sysroot_mount function. With this condition left in, we can we can get something like this: initrd-root-fs.target.requires `-- usr.mount -> /run/systemd/generator/usr.mount in the main system (i.e., not in the initramfs). In the initramfs, the previous condition already kicks in. diff --git a/src/fstab-generator/fstab-generator.c <snip> if (initrd) post = SPECIAL_INITRD_FS_TARGET; - else if (mount_in_initrd(me)) - post = SPECIAL_INITRD_ROOT_FS_TARGET; else if (mount_is_network(me)) post = SPECIAL_REMOTE_FS_TARGET; else "mount_in_initrd(me)" is true with "x-initrd.mount" option, the behaviour of systemd fstab generator changed after the above-mentioned patch, it always generates local mount units required by local-fs.target regardless of "x-initrd.mount". After failure, it enters dracut emergency, further triggers kdump emergency service, thus there is no problem. Signed-off-by: Xunlei Pang <xlpang(a)redhat.com> --- mkdumprd | 15 +-------------- 1 file changed, 1 insertion(+), 14 deletions(-) diff --git a/mkdumprd b/mkdumprd index 5a25853..468b564 100644 --- a/mkdumprd +++ b/mkdumprd @@ -103,20 +103,7 @@ to_mount() { # kernel, filter it out here. _options=$(echo $_options | sed 's/noauto//') _options=${_options/#ro/rw} #mount fs target as rw in 2nd kernel - # "x-initrd.mount" mount failure will trigger isolate emergency service - # W/o this, systemd won't isolate, thus we won't get to emergency. - # This is not applicable to remote fs mount, because if we use - # "x-initrd.mount", remote mount will become required by - # "initrd-root-fs.target", instead of "remote-fs.target". That's how it is - # handled within systemd internal. We need remote mount to be required - # "remote-fs.target", because we need to bring up network before any remote - # mount and "remote-fs.target" can be a checkpoint of that. - # If remote mount fails, dracut-initqueue will still start and once - # dracut-initqueue finishes, kdump service will start. Because remote mount - # failed, kdump service will fail and it will lead to kdump error handler. - if ! is_nfs_dump_target; then - _options="$_options,x-initrd.mount" - fi + _mntopts="$_target $_fstype $_options" #for non-nfs _dev converting to use udev persistent name if [ -b "$_source" ]; then -- 1.8.3.1

3 6

[PATCH] kdumpctl: use "apicid" other than "initial apicid"
by Xunlei Pang 13 Jul '17

13 Jul '17

We met a problem on AMD machines, when using "nr_cpus=4" for kdump, and crash happens on cpus other than cpu0, kdump kernel will fail to boot and eventually reset. After some debugging, we found that it stuck at the kernel path do_boot_cpu()-> ... ->wakeup_secondary_cpu_via_init(): apic_icr_write(APIC_INT_LEVELTRIG|APIC_INT_ASSERT|APIC_DM_INIT, phys_apicid); that is, it stuck at sending INIT from AP to BP and reset, which is actually what "disable_cpu_apicid=X" tries to solve. Printing the value of @phys_apicid showed that it was the value of "apicid" other that of "initial apicid" showed by /proc/cpuinfo. As described in x86 specification: "In MP systems, the local APIC ID is also used as a processor ID by the BIOS and the operating system. Some processors permit software to modify the APIC ID. However, the ability of software to modify the APIC ID is processor model specific. Because of this, operating system software should avoid writing to the local APIC ID register. The value returned by bits 31-24 of the EBX register (when the CPUID instruction is executed with a source operand value of 1 in the EAX register) is always the Initial APIC ID (determined by the platform initialization). This is true even if software has changed the value in the Local APIC ID register." From kernel commit 151e0c7de("x86, apic, kexec: Add disable_cpu_apicid kernel parameter"), we can see in generic_processor_info(), it uses a)read_apic_id() and b)@apicid to compare with @disabled_cpu_apicid. a)@apicid which is actually @phys_apicid above-mentioned is from the following calltrace(on the problematic AMD machine): generic_processor_info+0x37/0x300 acpi_register_lapic+0x30/0x90 acpi_parse_lapic+0x40/0x50 acpi_table_parse_entries_array+0x171/0x1de acpi_boot_init+0xed/0x50f The value of @apicid(from acpi MADT) is equal to the value of "apicid" showed by /proc/cpuinfo as proved by our debug printk. b)read_apic_id() gets the value from LAPIC ID register which is "apicid" as well. While the value of "initial apicid" is from cpuid instruction. One example of "apicid" and "initial apicid" of cpu0 from /proc/cpuinfo on AMD machine: apicid : 32 initial apicid : 0 Therefore, we should assign /proc/cpuifo "apicid" to "disable_cpu_apicid=X". We've never met such issue before, because we usually tested "nr_cpus=1", and mostly on Intel machines, and "apicid" and "initial apicid" have the same value in most cases on Intel machines. Signed-off-by: Xunlei Pang <xlpang(a)redhat.com> --- kdumpctl | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/kdumpctl b/kdumpctl index 4d6b3e8..46b65d2 100755 --- a/kdumpctl +++ b/kdumpctl @@ -77,15 +77,15 @@ remove_cmdline_param() } # -# This function returns the "initial apicid" of the -# boot cpu (cpu 0) if present. +# This function returns the "apicid" of the boot +# cpu (cpu 0) if present. # -get_bootcpu_initial_apicid() +get_bootcpu_apicid() { awk ' \ BEGIN { CPU = "-1"; } \ $1=="processor" && $2==":" { CPU = $NF; } \ - CPU=="0" && /initial apicid/ { print $NF; } \ + CPU=="0" && /^apicid/ { print $NF; } \ ' \ /proc/cpuinfo } @@ -206,7 +206,7 @@ prepare_cmdline() cmdline="${cmdline} ${KDUMP_COMMANDLINE_APPEND}" - id=`get_bootcpu_initial_apicid` + id=`get_bootcpu_apicid` if [ ! -z ${id} ] ; then cmdline=`append_cmdline "${cmdline}" disable_cpu_apicid ${id}` fi -- 1.8.3.1

5 10

[PATCH v3] kdumpctl: fix infinite loop caused by running under bash
by Ziyue Yang 12 Jul '17

12 Jul '17

Description of problem (https://bugzilla.redhat.com/show_bug.cgi?id=1465735): Run `kdumpctl status` as normal user, get below error messages: Another app is currently holding the kdump lock; waiting for it to exit... flock: 9: Bad file descriptor Another app is currently holding the kdump lock; waiting for it to exit... flock: 9: Bad file descriptor ... The bug is caused by behavior difference between bash and sh (bash in posix). In the function single_instance_lock in kdumpctl script, there is exec 9>/var/lock/kdump which will fail in user mode. However, this fail will cause script exiting under bash but not exiting under sh, causing infinite loop because the flock will always fail. According to the 16th item in ftp://ftp.gnu.org/old-gnu/Manuals/bash-2.02/html_node/bashref_66.html If a POSIX.2 special builtin returns an error status, a non- interactive shell exits. And according to https://www.gnu.org/software/bash/manual/html_node/Special-Builtins.html exec is one of the POSIX.2 special builtin's. This patch fixes the bug by checking exec return value. Signed-off-by: Ziyue Yang <ziyang(a)redhat.com> --- kdumpctl | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kdumpctl b/kdumpctl index e440bbb..1b50966 100755 --- a/kdumpctl +++ b/kdumpctl @@ -37,6 +37,10 @@ single_instance_lock() local rc timeout=5 exec 9>/var/lock/kdump + if [ $? -ne 0 ]; then + echo "Create file lock failed" + exit 1 + fi flock -n 9 rc=$? -- 2.9.3

3 2

[PATCH v4 0/8] collect kdump targets and improve kdump per these information
by Xunlei Pang 12 Jul '17

12 Jul '17

We collect all the kdump targets i.e. devices recognized under kdump, then improve kdump according to the type of the target. Currently we have two: dump target and root target in case of "dump_to_rootfs". If we know there is no crypt target, we can remove dracut "crypt" module, we only add rd.lvm.lv=X regarding the lvm target to kdump. If we know there is only network dumping(ssh/nfs), we can omit some non-network modules like dm/lvm/multipath/crypt/iscsi/fcoe/etc. If we know ..., we can omit ... This series does two major things: 1)Patch 1~5 solve https://bugzilla.redhat.com/1451717 2)Patch 6~8 reduce the initramfs size by omitting needless dracut modules. Xunlei Pang (8): kdump-lib.sh: fix improper get_block_dump_target() kdump-lib.sh: introduce get_kdump_targets() mkdumprd: change for_each_block_target() to use get_kdump_targets() kdumpctl: use generated rd.lvm.lv=X mkdumprd: omit crypt when there is no crypt kdump target mkdumprd: omit dracut modules in case of no dm target mkdumprd: omit dracut modules in case of network dumping module-setup: fix 99kdumpbase network dependency dracut-module-setup.sh | 2 +- kdump-lib.sh | 111 +++++++++++++++++++++++++++++++++++++++++++------ kdumpctl | 69 ++++++++++++++++++------------ mkdumprd | 103 +++++++++++++++++++++++++-------------------- 4 files changed, 200 insertions(+), 85 deletions(-) -- 1.8.3.1

4 13

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

kexec July 2017