kexec August 2019

kexec@lists.fedoraproject.org

8 participants
18 discussions

[PATCHv3] kdumpctl: wait a while for network ready if dump target is ssh
by Pingfan Liu 12 Aug '19

12 Aug '19

If dump target is ipv6 address, a host should have ipv6 address ready before starting kdump service. Otherwise, kdump service fails to start due to the failure "ssh dump_server_ip mkdir -p $SAVE_PATH". And user can see message like: "Could not create root@2620:52:0:10da:46a8:42ff:fe23:3272/var/crash" I observe a long period (about 30s) on some machine before they got ipv6 address dynamiclly, which is never seen on ipv4 host. Hence kdump service has a dependency on ipv6 address. But there is no good way to resolve it. One way is asking user to run the cmd "nmcli connection modify eth0 ipv6.may-fail false". But this will block systemd until ipv6 address is ready. Despite doing so, kdump can try its best (wait 1 minutes after it starts up) before failure. How to implement the wait is arguable. It will involve too many technique details if explicitly waiting on ipv6 address, instead, just lean on 'ssh' return value to see the availability of network. Signed-off-by: Pingfan Liu <piliu(a)redhat.com> --- kdumpctl | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/kdumpctl b/kdumpctl index a1a6ee2..86511b8 100755 --- a/kdumpctl +++ b/kdumpctl @@ -730,12 +730,40 @@ check_ssh_config() return 0 } +# ipv6 host address may takes a long time to be ready. +# Instead of checking against ipv6 address, we just check the network reachable +# by the return val of 'ssh' +check_and_wait_network_ready() +{ + local start_time=$(date +%s) + local cur + local diff + + + while true; do + ssh -q -i $SSH_KEY_LOCATION -o BatchMode=yes $DUMP_TARGET mkdir -p $SAVE_PATH + # ssh exits with the exit status of the remote command or with 255 if an error occurred + if [ $? -eq 0 ]; then + return 0 + elif [ $? -ne 255 ]; then + return 1 + fi + cur=$(date +%s) + diff=$( $cur - $start_time ) + # 60s time out + if [ $diff -gt 60 ]; then + break; + fi + sleep 1 + done + + return 1 +} + check_ssh_target() { - local _ret - ssh -q -i $SSH_KEY_LOCATION -o BatchMode=yes $DUMP_TARGET mkdir -p $SAVE_PATH - _ret=$? - if [ $_ret -ne 0 ]; then + check_and_wait_network_ready + if [ $? -ne 0 ]; then echo "Could not create $DUMP_TARGET:$SAVE_PATH, you probably need to run \"kdumpctl propagate\"" >&2 return 1 fi -- 2.7.5

2 1

[PATCH] kdumpctl: wait a while for ipv6 addr ready if dump target is ipv6
by Pingfan Liu 06 Aug '19

06 Aug '19

If dump target is ipv6 address, a host should have ipv6 address ready before starting kdump service. Otherwise, kdump service fails to start due to the failure "ssh dump_server_ip mkdir -p $SAVE_PATH". And user can see message like: "Could not create root@2620:52:0:10da:46a8:42ff:fe23:3272/var/crash" A host can get ipv6 address by three ways: Neighbor Discovery Protocol/ DHCPv6/ static. I observe a long period (about 30s) on some machine before they got ipv6 address dynamiclly, which is never seen on ipv4 host. Hence kdump service has a dependency on ipv6 address. But there is no good way to resolve it. One way is asking user to run the cmd "nmcli connection modify eth0 ipv6.may-fail false". But this will block systemd until ipv6 address is ready. Despite doing so, kdump can try its best (wait 1 minutes after it starts up) before failure. Signed-off-by: Pingfan Liu <piliu(a)redhat.com> --- kdumpctl | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/kdumpctl b/kdumpctl index a1a6ee2..02302a1 100755 --- a/kdumpctl +++ b/kdumpctl @@ -730,9 +730,38 @@ check_ssh_config() return 0 } +# Alternative way, using "nmcli connection modify eth0 ipv6.may-fail false" +# to force systemd to wait until ipv6 address ready. +# But it is better to let kdump wait a minute on ipv6 address +check_and_wait_ipv6_ready() +{ + local ipaddr=$1 + local ipv6_up="false" + local left=60 + + ipaddr=$(echo $ipaddr | awk -F "@" '{ print $2 }') + if is_ipv6_address $ipaddr; then + while [ $left -gt 0 ]; do + ip -f inet6 addr | grep global + if [ ${PIPESTATUS[1]} -eq 0 ]; then + ipv6_up="true" + break; + fi + left=$(expr $left - 1) + sleep 1 + done + + if [ "$ipv6_up" == "false" ]; then + echo "ipv6 address is not ready within more than 1 minute" + fi + fi +} + check_ssh_target() { local _ret + + check_and_wait_ipv6_ready $DUMP_TARGET ssh -q -i $SSH_KEY_LOCATION -o BatchMode=yes $DUMP_TARGET mkdir -p $SAVE_PATH _ret=$? if [ $_ret -ne 0 ]; then -- 2.7.5

1 0

[PATCH] Add the 'UserKnownHostsFile' option to authenticate the kdump server
by Lianbo Jiang 05 Aug '19

05 Aug '19

When using the ssh kdump, the ssh client will use the '-o StrictHostKeyChecking=yes' option to connect the kdump server. To ensure the dump file is going to be sent to a true kdump server as opposed to any fake one, the ssh client needs to authenticate the server as well. A known_hosts file with server hostkey can be provisioned on the client side in advance to facilitate the authentication(which is added by ssh-client module in mkdumprd). Add a configuration in '/etc/kdump.conf' to specify the location of the known_hosts file to be used in the 'ssh -o UserKnownHostsFile' option. Suggested-by: Jun Wang <junw99(a)yahoo.com> Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com> --- dracut-kdump.sh | 4 ++-- kdump-lib-initramfs.sh | 6 ++++++ kdump.conf | 4 ++++ kdumpctl | 12 ++++++++++-- mkdumprd | 11 +++++++++-- 5 files changed, 31 insertions(+), 6 deletions(-) diff --git a/dracut-kdump.sh b/dracut-kdump.sh index ce56459ed088..0eafe6458530 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -71,7 +71,7 @@ dump_raw() dump_ssh() { - local _opt="-i $1 -o BatchMode=yes -o StrictHostKeyChecking=yes" + local _opt="-i $1 -o BatchMode=yes -o StrictHostKeyChecking=yes -i $3 -o UserKnownHostsFile" local _dir="$KDUMP_PATH/$HOST_IP-$DATEDIR" local _host=$2 @@ -156,7 +156,7 @@ read_kdump_conf() add_dump_code "dump_raw $config_val" ;; ssh) - add_dump_code "dump_ssh $SSH_KEY_LOCATION $config_val" + add_dump_code "dump_ssh $SSH_KEY_LOCATION $config_val $SSH_KNOWN_HOSTS" ;; esac done <<< "$(read_strip_comments $KDUMP_CONF)" diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 608dc6efc07e..7d595d5b7d06 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -11,6 +11,7 @@ DATEDIR=`date +%Y-%m-%d-%T` HOST_IP='127.0.0.1' DUMP_INSTRUCTION="" SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" +SSH_KNOWN_HOSTS="/root/.ssh/known_hosts" KDUMP_SCRIPT_DIR="/kdumpscripts" DD_BLKSIZE=512 FINAL_ACTION="systemctl reboot -f" @@ -38,6 +39,11 @@ get_kdump_confs() SSH_KEY_LOCATION=$config_val fi ;; + known_hosts) + if [ -f "$config_val" ];then + SSH_KNOWN_HOSTS="$config_val" + fi + ;; kdump_pre) KDUMP_PRE="$config_val" ;; diff --git a/kdump.conf b/kdump.conf index 1f0fc2ddc40b..2b10d57ac561 100644 --- a/kdump.conf +++ b/kdump.conf @@ -152,6 +152,9 @@ # to send fence_kdump notifications to. # (this option is mandatory to enable fence_kdump). # +# known_hosts <path> +# - The "path" represents the path of know_hosts, the default value +# is /root/.ssh/known_hosts. #raw /dev/vg/lv_kdump #ext4 /dev/vg/lv_kdump @@ -173,3 +176,4 @@ core_collector makedumpfile -l --message-level 1 -d 31 #dracut_args --omit-drivers "cfg80211 snd" --add-drivers "ext2 ext3" #fence_kdump_args -p 7410 -f auto -c 0 -i 10 #fence_kdump_nodes node1 node2 +#known_hosts /root/.ssh/known_hosts diff --git a/kdumpctl b/kdumpctl index a1a6ee24b768..7ba7e8cf2685 100755 --- a/kdumpctl +++ b/kdumpctl @@ -9,6 +9,7 @@ MKDUMPRD="/sbin/mkdumprd -f" DRACUT_MODULES_FILE="/usr/lib/dracut/modules.txt" SAVE_PATH=/var/crash SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" +SSH_KNOWN_HOSTS="/root/.ssh/known_hosts" INITRD_CHECKSUM_LOCATION="/boot/.fadump_initrd_checksum" DUMP_TARGET="" DEFAULT_INITRD="" @@ -243,7 +244,7 @@ check_config() case "$config_opt" in \#* | "") ;; - raw|ext2|ext3|ext4|minix|btrfs|xfs|nfs|ssh|sshkey|path|core_collector|kdump_post|kdump_pre|extra_bins|extra_modules|failure_action|default|final_action|force_rebuild|force_no_rebuild|dracut_args|fence_kdump_args|fence_kdump_nodes) + raw|ext2|ext3|ext4|minix|btrfs|xfs|nfs|ssh|sshkey|known_hosts|path|core_collector|kdump_post|kdump_pre|extra_bins|extra_modules|failure_action|default|final_action|force_rebuild|force_no_rebuild|dracut_args|fence_kdump_args|fence_kdump_nodes) # remove inline comments after the end of a directive. [ -z "$config_val" ] && { echo "Invalid kdump config value for option $config_opt." @@ -711,6 +712,13 @@ check_ssh_config() echo "WARNING: '$config_val' doesn't exist, using default value '$SSH_KEY_LOCATION'" fi ;; + known_hosts) + if [ -f "$config_val" ];then + SSH_KNOWN_HOSTS=$(/usr/bin/readlink -m $config_val) + else + echo "WARNING: '$config_val' doesn't exist, using default value '$SSH_KNOWN_HOSTS'" + fi + ;; path) SAVE_PATH=$config_val ;; @@ -733,7 +741,7 @@ check_ssh_config() check_ssh_target() { local _ret - ssh -q -i $SSH_KEY_LOCATION -o BatchMode=yes $DUMP_TARGET mkdir -p $SAVE_PATH + ssh -q -i $SSH_KEY_LOCATION -i $SSH_KNOWN_HOSTS -o BatchMode=yes $DUMP_TARGET mkdir -p $SAVE_PATH _ret=$? if [ $_ret -ne 0 ]; then echo "Could not create $DUMP_TARGET:$SAVE_PATH, you probably need to run \"kdumpctl propagate\"" >&2 diff --git a/mkdumprd b/mkdumprd index cf3533fe2be9..c4e2f8ba4c31 100644 --- a/mkdumprd +++ b/mkdumprd @@ -13,6 +13,7 @@ export IN_KDUMP=1 conf_file="/etc/kdump.conf" SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" +SSH_KNOWN_HOSTS="/root/.ssh/known_hosts" SAVE_PATH=$(awk '/^path/ {print $2}' $conf_file) [ -z "$SAVE_PATH" ] && SAVE_PATH=$DEFAULT_PATH # strip the duplicated "/" @@ -144,7 +145,7 @@ is_readonly_mount() { #called from while loop and shouldn't read from stdin, so we're using "ssh -n" get_ssh_size() { local _opt _out _size - _opt="-i $SSH_KEY_LOCATION -o BatchMode=yes -o StrictHostKeyChecking=yes" + _opt="-i $SSH_KEY_LOCATION -o BatchMode=yes -o StrictHostKeyChecking=yes -i $SSH_KNOWN_HOSTS -o UserKnownHostsFile" _out=$(ssh -q -n $_opt $1 "df -P $SAVE_PATH") [ $? -ne 0 ] && { perror_exit "checking remote ssh server available size failed." @@ -162,7 +163,7 @@ get_ssh_size() { mkdir_save_path_ssh() { local _opt _dir - _opt="-i $SSH_KEY_LOCATION -o BatchMode=yes -o StrictHostKeyChecking=yes" + _opt="-i $SSH_KEY_LOCATION -o BatchMode=yes -o StrictHostKeyChecking=yes -i $SSH_KNOWN_HOSTS -o UserKnownHostsFile" ssh -qn $_opt $1 mkdir -p $SAVE_PATH 2>&1 > /dev/null _ret=$? if [ $_ret -ne 0 ]; then @@ -385,6 +386,12 @@ if [ -f "$keyfile" ]; then SSH_KEY_LOCATION=$(/usr/bin/readlink -m $keyfile) fi +# if specified, get the known_hosts +known_hosts=$(awk '/^known_hosts/ {print $2}' $conf_file) +if [ -f "$known_hosts" ];then + SSH_KNOWN_HOSTS=$(/usr/bin/readlink -m $known_hosts) +fi + if [ "$(uname -m)" = "s390x" ]; then add_dracut_module "znet" fi -- 2.17.1

4 13

[PATCH] Limit the size of vmcore-dmesg.txt to no more than 100MB
by Lianbo Jiang 05 Aug '19

05 Aug '19

From: Jun Wang <junw99(a)yahoo.com> With some corrupted vmcore files, the vmcore-dmesg.txt file may grow forever till the kdump disk becomes full, and also probably causes the disk error messages as follow: ... sd 0:0:0:0: [sda] tag#6 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK sd 0:0:0:0: [sda] tag#6 CDB: Read(10) 28 00 08 06 4c 98 00 00 08 00 blk_update_request: I/O error, dev sda, sector 134630552 sd 0:0:0:0: [sda] tag#7 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK sd 0:0:0:0: [sda] tag#7 CDB: Read(10) 28 00 08 06 4c 98 00 00 08 00 blk_update_request: I/O error, dev sda, sector 134630552 ... Lets limit the size of vmcore-dmesg.txt to avoid such problems. Signed-off-by: Jun Wang <junw99(a)yahoo.com> Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com> --- dracut-kdump.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dracut-kdump.sh b/dracut-kdump.sh index 2ae1c7c5d70d..ce56459ed088 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -102,8 +102,8 @@ save_vmcore_dmesg_ssh() { local _opts="$3" local _location=$4 - echo "kdump: saving vmcore-dmesg.txt" - $_dmesg_collector /proc/vmcore | ssh $_opts $_location "dd of=$_path/vmcore-dmesg-incomplete.txt" + echo "kdump: saving vmcore-dmesg.txt, up to 100MB" + $_dmesg_collector /proc/vmcore | ssh $_opts $_location "dd of=$_path/vmcore-dmesg-incomplete.txt bs=512 count=204800" _exitcode=$? if [ $_exitcode -eq 0 ]; then -- 2.17.1

4 3

[PATCH 0/2] Fix failure of makedumpfile on Linux 5.3-rc1
by Kazuhito Hagio 05 Aug '19

05 Aug '19

From: Kazuhito Hagio <k-hagio(a)ab.jp.nec.com> # resend this set bacause the previous one is being held.. These are both backports from upstream, and tested OK with kernel 5.3.0-0.rc1.git3.1.fc31.x86_64. Patch 2/2 fixes a failure of makedumpfile on Linux 5.3-rc1 that prints error like the following: readmem: Can't convert a virtual address(fffffc97d1000000) to physical address. readmem: type_addr: 0, addr:fffffc97d1000000, size:32768 __exclude_unnecessary_pages: Can't read the buffer of struct page. create_2nd_bitmap: Can't exclude unnecessary pages. Patch 1/2 is not necessary for fixing the issue, but without the patch, if a similar issue occurs in the future, makedumpfile can take a long time to finish generating a broken vmcore meaninglessly, and 2nd kernel cannot reboot soon. Kazuhito Hagio (2): makedumpfile: Do not proceed when get_num_dumpable_cyclic() fails makedumpfile: Increase SECTION_MAP_LAST_BIT to 4 ...d-when-get_num_dumpable_cyclic-fails.patch | 44 +++++++++++++++++++ ...e-Increase-SECTION_MAP_LAST_BIT-to-4.patch | 38 ++++++++++++++++ kexec-tools.spec | 4 ++ 3 files changed, 86 insertions(+) create mode 100644 kexec-tools-2.0.20-makedumpfile-Do-not-proceed-when-get_num_dumpable_cyclic-fails.patch create mode 100644 kexec-tools-2.0.20-makedumpfile-Increase-SECTION_MAP_LAST_BIT-to-4.patch -- 2.20.1

3 4

[PATCH] Don't forward and drop journalctl logs for fadump
by Kairui Song 05 Aug '19

05 Aug '19

fadump will alter the normal boot initramfs and we don't want a normal boot to foward and drop the journalctl logs. Signed-off-by: Kairui Song <kasong(a)redhat.com> --- dracut-module-setup.sh | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index 9ef60210..3fa696da 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -839,9 +839,12 @@ install() { fi # Forward logs to console directly, this avoids unneccessary memory - # consumption and make console output more useful - mkdir -p ${initdir}/etc/systemd/journald.conf.d - echo "[Journal]" > ${initdir}/etc/systemd/journald.conf.d/kdump.conf - echo "Storage=none" >> ${initdir}/etc/systemd/journald.conf.d/kdump.conf - echo "ForwardToConsole=yes" >> ${initdir}/etc/systemd/journald.conf.d/kdump.conf + # consumption and make console output more useful. + # Only do so for non fadump image. + if ! is_fadump_capable; then + mkdir -p ${initdir}/etc/systemd/journald.conf.d + echo "[Journal]" > ${initdir}/etc/systemd/journald.conf.d/kdump.conf + echo "Storage=none" >> ${initdir}/etc/systemd/journald.conf.d/kdump.conf + echo "ForwardToConsole=yes" >> ${initdir}/etc/systemd/journald.conf.d/kdump.conf + fi } -- 2.21.0

2 1

[PATCH 0/2] Fix failure of makedumpfile on Linux 5.3-rc1
by Kazuhito Hagio 02 Aug '19

02 Aug '19

These are both backports from upstream, and tested OK with kernel 5.3.0-0.rc1.git3.1.fc31.x86_64. Patch 2/2 fixes a failure of makedumpfile on Linux 5.3-rc1 that prints error like the following: readmem: Can't convert a virtual address(fffffc97d1000000) to physical address. readmem: type_addr: 0, addr:fffffc97d1000000, size:32768 __exclude_unnecessary_pages: Can't read the buffer of struct page. create_2nd_bitmap: Can't exclude unnecessary pages. Patch 1/2 is not necessary for fixing the issue, but without the patch, if a similar issue occurs in the future, makedumpfile can take a long time to finish generating a broken vmcore meaninglessly, and 2nd kernel cannot reboot soon. Kazuhito Hagio (2): makedumpfile: Do not proceed when get_num_dumpable_cyclic() fails makedumpfile: Increase SECTION_MAP_LAST_BIT to 4 ...d-when-get_num_dumpable_cyclic-fails.patch | 44 +++++++++++++++++++ ...e-Increase-SECTION_MAP_LAST_BIT-to-4.patch | 38 ++++++++++++++++ kexec-tools.spec | 4 ++ 3 files changed, 86 insertions(+) create mode 100644 kexec-tools-2.0.20-makedumpfile-Do-not-proceed-when-get_num_dumpable_cyclic-fails.patch create mode 100644 kexec-tools-2.0.20-makedumpfile-Increase-SECTION_MAP_LAST_BIT-to-4.patch -- 2.20.1

1 2

[PATCHv2] dracut-module-setup.sh: skip alias of localhost in get_pcs_fence_kdump_nodes()
by Pingfan Liu 01 Aug '19

01 Aug '19

The current code only exclude the hostname, while localhost can have alias in /etc/hosts. All of the alias should be excluded from the fence dump node to avoid deadlock issue. Signed-off-by: Pingfan Liu <piliu(a)redhat.com> --- dracut-module-setup.sh | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index 27b9f02..ca3b105 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -689,6 +689,21 @@ kdump_check_iscsi_targets () { } } +is_localhost() { + local hostnames=$(hostname -A) + local shortnames=$(hostname -A -s) + local nodename=$1 + + hostnames="$hostnames $shortnames" + + for name in ${hostnames}; do + if [ "$name" == "$nodename" ]; then + return 0 + fi + done + return 1 +} + # retrieves fence_kdump nodes from Pacemaker cluster configuration get_pcs_fence_kdump_nodes() { local nodes @@ -703,7 +718,7 @@ get_pcs_fence_kdump_nodes() { eval $node nodename=$uname # Skip its own node name - if [ "$nodename" = `hostname` -o "$nodename" = `hostname -s` ]; then + if is_localhost $nodename; then continue fi nodes="$nodes $nodename" -- 2.20.1

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

kexec August 2019