kexec March 2022

kexec@lists.fedoraproject.org

6 participants
14 discussions

[PATCH 00/11] crashkernel=auto: support fadump, reserve extra SWIOTLB memory and remove crashkernel.default
by Coiby Xu 16 Jun '22

16 Jun '22

The crashkernel=auto implementation in kernel space has been rejected upstream [1]. The current user space implementation [2] [3] ships a crashkernel.default but hasn't supported the swiotlb memory requirement, custom crashkernel value from user and fadump. The crashkernel.default implementation seems to be overly complex, - the default crashkernel value rarely changes. This is no need to ship the same crashkernel.default default for every kernel package of a architecture; - when deciding the value of crashkernel for a new kernel, the crashkernel.default of existing kernel is took into consideration We can simply let the kexec-tools maintain the default crashkernel values and provide an API for kdump-anacon-addon to query it. And for a newly installed kernel, we can simply call "kdumpctl reset-crashkernel KERNELPATH" to set its crashkernel value. For the unfulfilled requirements, - crashkernel is introduced to /etc/kdump.conf for the user can set custom crashkernel value to tell kexec-tools to manage crashkernel value automatically. - "kdumpctl reset-crashkernel" has been written for the above purpose. - "kdumpctl fadump on/off" is added for supporting fadump. [1] https://lore.kernel.org/linux-mm/20210507010432.IN24PudKT%25akpm@linux-foun… [2] https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171 [3] https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org… Coiby Xu (11): update default crashkernel value factor out kdump_get_arch_recommend_crashkernel provide get_default_crashkernel for kdump_anaconda_addon introduce crashkernel option to kdump.conf add a helper function to write a config value to kdump.conf add a helper function to read kernel cmdline parameter from grubby --info rewrite reset_crashkernel to provide more features for the user and to be called by kernel installation hook allow to add extra memory to crashkernel string Reserve extra memory when SME or SEV is active provide kdumpctl fadump on/off use "kdumpctl reset-crashkernel KERNELIMAGE" in kernel installation hook 92-crashkernel.install | 135 +---------------------------- kdump-lib-initramfs.sh | 9 ++ kdump-lib.sh | 95 +++++++++++++++------ kdump.conf | 6 ++ kdump.conf.5 | 7 ++ kdumpctl | 188 ++++++++++++++++++++++++++++++++++++----- kdumpctl.8 | 16 ++-- 7 files changed, 271 insertions(+), 185 deletions(-) -- 2.31.1

6 45

[PATCH] mkdumprd: allow spaces after 'path' config phrase when network dump
by Kazuhito Hagio 26 May '22

26 May '22

Without this patch, when there are two or more spaces after 'path' configuration phrase with ssh or nfs setting, SAVE_PATH is set to '/var/crash' in mkdumprd, and in most cases kdump service fails to start. ssh kdump(a)192.168.122.1 path /kdump ^^ This behavior would be too sensitive and different from the other configurations. With this patch, mkdumprd allows such spaces. Signed-off-by: Kazuhito Hagio <k-hagio(a)ab.jp.nec.com> --- mkdumprd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdumprd b/mkdumprd index a6f7fe8..aa0abfd 100644 --- a/mkdumprd +++ b/mkdumprd @@ -13,7 +13,7 @@ export IN_KDUMP=1 conf_file="/etc/kdump.conf" SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" -SAVE_PATH=$(grep ^path $conf_file| cut -d' ' -f2) +SAVE_PATH=$(awk '/^path/ {print $2}' $conf_file) [ -z "$SAVE_PATH" ] && SAVE_PATH=$DEFAULT_PATH # strip the duplicated "/" SAVE_PATH=$(echo $SAVE_PATH | tr -s /) -- 2.18.0

4 3

[PATCH 1/4] update kexec-kdump-howto
by Coiby Xu 18 Apr '22

18 Apr '22

1. yum is deprecated so use dnf instead 2. use the "kdumpctl reset-crashkernel" API Signed-off-by: Coiby Xu <coxu(a)redhat.com> --- kexec-kdump-howto.txt | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/kexec-kdump-howto.txt b/kexec-kdump-howto.txt index 1aeffc7..0121d0e 100644 --- a/kexec-kdump-howto.txt +++ b/kexec-kdump-howto.txt @@ -44,7 +44,7 @@ ia64 and ppc64. If you're reading this document, you should already have kexec-tools installed. If not, you install it via the following command: - # yum install kexec-tools + # dnf install kexec-tools Now load a kernel with kexec: @@ -66,23 +66,24 @@ How to configure kdump Again, we assume if you're reading this document, you should already have kexec-tools installed. If not, you install it via the following command: - # yum install kexec-tools + # dnf install kexec-tools To be able to do much of anything interesting in the way of debug analysis, you'll also need to install the kernel-debuginfo package, of the same arch as your running kernel, and the crash utility: - # yum --enablerepo=\*debuginfo install kernel-debuginfo.$(uname -m) crash + # dnf --enablerepo=\*debuginfo install kernel-debuginfo.$(uname -m) crash -Next up, we need to modify some boot parameters to reserve a chunk of memory for -the capture kernel. With the help of grubby, it's very easy to append -"crashkernel=128M" to the end of your kernel boot parameters. Note that the X -values are such that X = the amount of memory to reserve for the capture kernel. -And based on arch and system configuration, one might require more than 128M to -be reserved for kdump. One need to experiment and test kdump, if 128M is not -sufficient, try reserving more memory. +Next up, we need to reserve a chunk of memory for the capture kernel. To use +the default crashkernel value, you can kdumpctl: - # grubby --args="crashkernel=128M" --update-kernel=/boot/vmlinuz-`uname -r` + # kdumpctl reset-crashkernel --kernel=/boot/vmlinuz-`uname -r` + +If based on arch and system configuration, the default crashkernel isn't +sufficent, you can specify a larger value e.g crashkernel=256M after +experimenting and testing kdump with the help of grubby: + + # grubby --args="crashkernel=256M" --update-kernel=/boot/vmlinuz-`uname -r` Note that there is an alternative form in which to specify a crashkernel memory reservation, in the event that more control is needed over the size and -- 2.34.1

2 15

[PATCH 0/8] unit tests with ShellSpec
by Coiby Xu 14 Apr '22

14 Apr '22

v1: - issues found by Tao - clean up files creatd by mktemp - use DUMMY_PARAM when calling a mocked - clean up unsued parameters - mock uname for kdump mode tests to fake current running kernel - {add,remove}_kernel_arg_in_grub_etc_default replaced by _update_* Potential benefits brought by unit tests ======================================= - catch as many bugs as possible and catch them as early as possible - improve the code quality and make the code robust - Keep API stable - prevent regressions from change during the development and in the future (it's quite annoying to solve one problem only to find another cropping up) - make it easy for code review because of concrete test cases Why ShellSpec ============= ShellSpec [1] is chosen for the following benefits, - rich features: to name a few - mocking - Parameterized tests - code coverage - good documentation and examples For the full comparison between shellspec and other frameworks, please refer to [2]. How to run the tests ==================== After installing shellspec and optionally kcov $ curl -fsSL https://git.io/shellspec | sh $ sudo dnf install kcov 1. go to the root of this repo 2. create a file with _spec as suffix in the spec folder, e.g. spec/kdumpctl_spec.sh 3. run shellspec shellspec --kcov --kcov-options "--include-pattern=kdumpctl,kdump-lib.sh,kdump-lib-initramfs.sh,kdump-logger.sh" Running: /bin/sh [bash 5.1.0(1)-release] ................................................................ Finished in 11.15 seconds (user 6.97 seconds, sys 9.54 seconds) 64 examples, 0 failures Code covered: 16.08%, Executed lines: 224, Instrumented lines: 1393 [1] https://github.com/shellspec/shellspec [2] https://github.com/dodie/testing-in-bash#detailed-comparision Coiby Xu (8): unit tests: prepare for kdumpctl and kdump-lib.sh to be unit-tested unit tests: add tests for get_grub_kernel_boot_parameter unit tests: add tests for get_dump_mode_by_fadump_val unit tests: add tests for kdumpctl read_proc_environ_var and _is_osbuild unit tests: add tests for _{update,read}_kernel_arg_in_grub_etc_default in kdumpctl unit tests: add tests for "kdumpctl reset-crashkernel" unit tests: add tests for kdump_get_conf_val in kdump-lib-initramfs.sh unit tests: add check_config with with the default kdump.conf .shellspec | 0 kdump-lib.sh | 7 +- kdumpctl | 24 +- spec/kdump-lib-initramfs_spec.sh | 41 ++++ spec/kdumpctl_general_spec.sh | 183 ++++++++++++++ spec/kdumpctl_reset_crashkernel_spec.sh | 223 ++++++++++++++++++ spec/support/bin/@grubby | 3 + ...846f63134c7295458cf36300ba5b-0-rescue.conf | 8 + ...58cf36300ba5b-5.14.14-200.fc34.x86_64.conf | 8 + ...458cf36300ba5b-5.15.6-100.fc34.x86_64.conf | 8 + spec/support/grub_env | 3 + 11 files changed, 499 insertions(+), 9 deletions(-) create mode 100644 .shellspec create mode 100644 spec/kdump-lib-initramfs_spec.sh create mode 100644 spec/kdumpctl_general_spec.sh create mode 100644 spec/kdumpctl_reset_crashkernel_spec.sh create mode 100755 spec/support/bin/@grubby create mode 100644 spec/support/boot_load_entries/e986846f63134c7295458cf36300ba5b-0-rescue.conf create mode 100644 spec/support/boot_load_entries/e986846f63134c7295458cf36300ba5b-5.14.14-200.fc34.x86_64.conf create mode 100644 spec/support/boot_load_entries/e986846f63134c7295458cf36300ba5b-5.15.6-100.fc34.x86_64.conf create mode 100644 spec/support/grub_env -- 2.34.1

2 13

[PATCH] x86_64: Reserve extra memory for crashkernel if AMD sme/sev enabled
by Tao Liu 07 Apr '22

07 Apr '22

When AMD sme/sev enabled, it will require extra memory reserved for swiotlb. Currently the swiotlb kernel parameter is removed by /etc/sysconfig/kdump for 2nd kernel, so a 64MB memory for swiotlb will be reserved by default [1], and the 2nd kernel will fail of OOM with the default crashkernel value. swiotlb memory is mandatory for AMD sme/sev. For rhel8, kernel commit 0adb0f4 ("[x86] Reserve at most 64M of SWIOTLB memory for crashkernel") implemented the reservation in the kernel side, as part of the kernel crashkernel=auto implementation. For rhel9 however, crashkernel=auto has been mirrored to userspace kexec-tools, so this patch will enable the swiotlb reservation for AMD sme/sev in kexec-tools as well. [1]: https://elixir.bootlin.com/linux/v5.14/source/arch/x86/mm/mem_encrypt.c#L200 Signed-off-by: Tao Liu <ltao(a)redhat.com> --- kdump-lib-initramfs.sh | 10 ++++++++++ kdump-lib.sh | 11 +++++++++-- kdumpctl | 7 ++++++- 3 files changed, 25 insertions(+), 3 deletions(-) diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 9be0fe9..699babd 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -8,6 +8,11 @@ KDUMP_CONFIG_FILE="/etc/kdump.conf" FENCE_KDUMP_CONFIG_FILE="/etc/sysconfig/fence_kdump" FENCE_KDUMP_SEND="/usr/libexec/fence_kdump_send" +# Reserve 64MB for swiotlb when AMD sme/sev enabled. +AMD_SME_SEV_SWIOTLB=64 +# Copied from include/linux/swiotlb.h +IO_TLB_SHIFT=11 + # Read kdump config in well formated style kdump_read_conf() { @@ -130,3 +135,8 @@ is_fs_dump_target() { [ -n "$(kdump_get_conf_val "ext[234]\|xfs\|btrfs\|minix")" ] } + +is_amd_sme_sev_enabled() +{ + dmesg | grep -sq "AMD Memory Encryption Features active" +} diff --git a/kdump-lib.sh b/kdump-lib.sh index 4ed5035..8ea6bfe 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -831,7 +831,7 @@ get_recommend_size() # $1 dump mode, if not specified, dump_mode will be judged by is_fadump_capable kdump_get_arch_recommend_crashkernel() { - local _arch _ck_cmdline _dump_mode + local _arch _ck_cmdline _dump_mode _mem_enc if [[ -z "$1" ]]; then if is_fadump_capable; then @@ -845,7 +845,14 @@ kdump_get_arch_recommend_crashkernel() _arch=$(uname -m) - if [[ $_arch == "x86_64" ]] || [[ $_arch == "s390x" ]]; then + if [[ $_arch == "x86_64" ]]; then + if is_amd_sme_sev_enabled; then + _mem_enc=$AMD_SME_SEV_SWIOTLB + else + _mem_enc=0 + fi + _ck_cmdline="1G-4G:$((192+$_mem_enc))M,4G-64G:$((256+$_mem_enc))M,64G-:$((512+$_mem_enc))M" + elif [[ $_arch == "s390x" ]]; then _ck_cmdline="1G-4G:192M,4G-64G:256M,64G-:512M" elif [[ $_arch == "aarch64" ]]; then _ck_cmdline="2G-:448M" diff --git a/kdumpctl b/kdumpctl index 1869753..50aafda 100755 --- a/kdumpctl +++ b/kdumpctl @@ -21,6 +21,7 @@ FADUMP_REGISTER_SYS_NODE="/sys/kernel/fadump_registered" #kdump shall be the default dump mode DEFAULT_DUMP_MODE="kdump" image_time=0 +size_mb=$((1024 * 1024)) standard_kexec_args="-d -p" @@ -622,6 +623,11 @@ load_kdump() { local ret + # Force 2nd kernel use swiotlb for AMD sme/sve to 64MB at most. + if is_amd_sme_sev_enabled; then + KDUMP_COMMANDLINE_APPEND+=" swiotlb=$((($AMD_SME_SEV_SWIOTLB * $size_mb) >> $IO_TLB_SHIFT)) " + fi + KEXEC_ARGS=$(prepare_kexec_args "${KEXEC_ARGS}") KDUMP_COMMANDLINE=$(prepare_cmdline "${KDUMP_COMMANDLINE}" "${KDUMP_COMMANDLINE_REMOVE}" "${KDUMP_COMMANDLINE_APPEND}") @@ -1189,7 +1195,6 @@ do_estimate() local -A large_mods local baseline local kernel_size mod_size initrd_size baseline_size runtime_size reserved_size estimated_size recommended_size - local size_mb=$((1024 * 1024)) setup_initrd if [[ ! -f $TARGET_INITRD ]]; then -- 2.33.1

3 10

[PATCH v2 00/14] kdumpctl: various fixes and cleanups
by Philipp Rudo 02 Apr '22

02 Apr '22

Hi, while looking into transforming mkdumprd and mkfadumprd into library functions I noticed various nits in kdumpctl. This series addresses them. The series is made up of two parts: Patches 1-8 are small independent cleanups and fixes. Patches 9-15 tries to reduce the number of file accesses to /etc/kdump.conf. It achieves this by only parsing kdump.conf once in check_config and storing the parsed values in an array. Later accesses can then simply use the value stored in the array instead of calling kdump_get_conf_val. Thanks Philipp v2: * Drop patch 2 as caused problems when value were edited in /etc/sysconfig/kdump * Fix false '!' in patch 13 Philipp Rudo (14): kdump-capture.service: switch to journal for stdout kdump-lib: fix typo in variable name kdumpctl: remove unnecessary uses of $? kdump-lib-initramfs: merge definitions for default ssh key kdumpctl: fix comment in check_and_wait_network_ready kdumpctl: forbid aliases from ssh config kdumpctl: simplify propagate_ssh_key kdumpctl: merge check_ssh_config into check_config kdumpctl: reduce file operations on kdump.conf kdumpctl: drop SAVE_PATH variable kdumpctl: drop SSH_KEY_LOCATION variable kdumpctl: drop DUMP_TARGET variable kdumpctl: remove kdump_get_conf_val in save_raw kdumpctl: simplify local_fs_dump_target dracut-kdump-capture.service | 4 +- dracut-kdump.sh | 2 +- kdump-lib-initramfs.sh | 1 + kdump-lib.sh | 14 +-- kdumpctl | 227 +++++++++++++++-------------------- mkdumprd | 2 +- 6 files changed, 112 insertions(+), 138 deletions(-) -- 2.35.1

2 15

[PATCH] kdumpctl/estimate: Fix unnecessary warning
by Philipp Rudo 02 Apr '22

02 Apr '22

do_estimate prints the warning that the reserved crashkernel is lower than the recommended one even then when both values are identical. This might cause confusion. So omit printing the warning when both values are equal. Signed-off-by: Philipp Rudo <prudo(a)redhat.com> --- kdumpctl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kdumpctl b/kdumpctl index 1869753..b7922a6 100755 --- a/kdumpctl +++ b/kdumpctl @@ -1270,7 +1270,7 @@ do_estimate() done fi - if [[ $reserved_size -le $recommended_size ]]; then + if [[ $reserved_size -lt $recommended_size ]]; then echo "WARNING: Current crashkernel size is lower than recommended size $((recommended_size / size_mb))M." fi } -- 2.35.1

2 1

Bug in lib/kdump/kdump-lib.sh ... on LVM crypto volumes
by John Donnelly 26 Mar '22

26 Mar '22

Hi bhe(a)redhat.com Suggested I forward this fix to you. It exists in RH8 and RH9 Please confirm > > Hi > > There is a bug in /lib/kdump/kdump-lib.sh when there are LVM crypto volumes > on the system : > > > Provisioning during the installation. In this case lvm pool is created: > # lvdisplay > --- Logical volume --- > LV Name pool00 > VG Name ol > LV UUID LARWgk-Fgdq-naoV-R2eN-ooWQ-mcIF-N34LbR > LV Write Access read/write (activated read only) > LV Creation host, time localhost, 2021-11-16 14:06:22 -0500 > LV Pool metadata pool00_tmeta > LV Pool data pool00_tdata > LV Status available > # open 0 > LV Size <20.15 GiB > Allocated pool data 27.26% > Allocated metadata 21.96% > Current LE 5158 > Segments 1 > Allocation inherit > Read ahead sectors auto > - currently set to 8192 > Block device 252:5 > > > > # kdumpctl rebuild > kdump: Rebuilding /boot/initramfs-86_64kdump.img > > > /lib/kdump/kdump-lib.sh: eval: line 939: syntax error near unexpected token > `;' > /lib/kdump/kdump-lib.sh: eval: line 939: `; echo $TYPE' > /lib/kdump/kdump-lib.sh: eval: line 939: syntax error near unexpected token > `;' > /lib/kdump/kdump-lib.sh: eval: line 939: `; echo $TYPE' > /lib/kdump/kdump-lib.sh: eval: line 939: syntax error near unexpected token > `;' > /lib/kdump/kdump-lib.sh: eval: line 939: `; echo $TYPE' > > > > > # blkid -u filesystem,crypto -o export -- /dev/block/253:0 > DEVNAME=/dev/block/253:0 > UUID=a5ff3f28-3a29-4353-9a12-2c39e2b6d399 > TYPE=ext4 > > kexec-tools]# eval "$(blkid -u filesystem,crypto -o export > -- /dev/block/253:0); echo \$TYPE" > ext4 > > [ kexec-tools]# blkid -u filesystem,crypto -o export -- > /dev/block/11:0 > [root@ca-ostest469 kexec-tools]# eval "$(blkid -u filesystem,crypto -o > export > -- /dev/block/11:0); echo \$TYPE" > > bash: syntax error near unexpected token `;' > > so the error is seen if blkid is not able to extract information from block > device > > > The fix : > > > > Applying this patch will avoid printing the error message. > > --- kdump-lib.sh.org 2022-02-03 17:23:48.654000000 -0600 > +++ kdump-lib.sh 2022-02-03 17:25:53.962000000 -0600 > @@ -936,6 +936,13 @@ > { > [[ -b /dev/block/$1 ]] || return 1 > > + # If the given device is LVM private, skip. > + blkid -u filesystem,crypto -o export -- /dev/block/$1 > /dev/null > + if [ $? -ne 0 ] > + then > + return 1 > + fi > + > local _type=$(eval "$(blkid -u filesystem,crypto -o export -- > /dev/block/$1); echo \$TYPE") > [[ $_type == "crypto_LUKS" ]] && echo $1 > > > > > It is a non-fatal error. kdumpclt rebuild still works. > > Is there a upstream github for this ? Or a contact you can share ? This is the mailing list where kexec-tools package is maintianed. You can subscribe, send your problem description and finding, our colleages may check it. kexec(a)lists.fedoraproject.org

1 0

[PATCH 00/15] kdumpctl: various fixes and cleanups
by Philipp Rudo 25 Mar '22

25 Mar '22

Hi, while looking into transforming mkdumprd and mkfadumprd into library functions I noticed various nits in kdumpctl. This series addresses them. The series is made up of two parts: Patches 1-8 are small independent cleanups and fixes. Patches 9-15 tries to reduce the number of file accesses to /etc/kdump.conf. It achieves this by only parsing kdump.conf once in check_config and storing the parsed values in an array. Later accesses can then simply use the value stored in the array instead of calling kdump_get_conf_val. Thanks Philipp Philipp Rudo (15): kdump-capture.service: switch to journal for stdout kdumpctl: source dependencies before defining globals kdump-lib: fix typo in variable name kdumpctl: remove unnecessary uses of $? kdump-lib-initramfs: merge definitions for default ssh key kdumpctl: fix comment in check_and_wait_network_ready kdumpctl: forbid aliases from ssh config kdumpctl: simplify propagate_ssh_key kdumpctl: merge check_ssh_config into check_config kdumpctl: reduce file operations on kdump.conf kdumpctl: drop SAVE_PATH variable kdumpctl: drop SSH_KEY_LOCATION variable kdumpctl: drop DUMP_TARGET variable kdumpctl: remove kdump_get_conf_val in save_raw kdumpctl: simplify local_fs_dump_target dracut-kdump-capture.service | 4 +- dracut-kdump.sh | 2 +- kdump-lib-initramfs.sh | 1 + kdump-lib.sh | 14 +- kdumpctl | 258 ++++++++++++++++------------------- mkdumprd | 2 +- 6 files changed, 128 insertions(+), 153 deletions(-) -- 2.34.1

4 49

Re: Bug in lib/kdump/kdump-lib.sh ... on LVM crypto volumes
by John Donnelly 23 Mar '22

23 Mar '22

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

kexec March 2022