By default, early kdump reboots the system after capturing the vmcore. If the problematic system is continuously crashing due to some issue during early boot stage, the system may fall into infinite loop restart like this:
boot -----> crash -----> early kdump (dump vmcore) ^ | '.........(reboot).............'
But now, the system crash at early stage is only captured by early kdump, and the rest is captured by normal kdump. That to say, when normal kdump service starts, it will load it again and override early kdump. It is helpful to control the logic of early kdump and normal kdump separately in final action(it is called by kdump-capture.service). For example, early kdump always passes the 'rd.earlykdump' to the second kernel when early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' to the second kernel at any time. So they can be distinguished in the second kernel.
Signed-off-by: Lianbo Jiang lijiang@redhat.com --- dracut-early-kdump.sh | 2 +- dracut-kdump-error-handler.sh | 7 ++++++- dracut-kdump.sh | 6 +++++- kdump-lib-initramfs.sh | 16 +++++++++++++--- kdump.sysconfig | 2 +- kdump.sysconfig.aarch64 | 2 +- kdump.sysconfig.i386 | 2 +- kdump.sysconfig.ppc64 | 2 +- kdump.sysconfig.ppc64le | 2 +- kdump.sysconfig.s390x | 2 +- kdump.sysconfig.x86_64 | 2 +- kdumpctl | 15 +++++++++++++-- 12 files changed, 45 insertions(+), 15 deletions(-)
diff --git a/dracut-early-kdump.sh b/dracut-early-kdump.sh index 34a9909..a799dbf 100755 --- a/dracut-early-kdump.sh +++ b/dracut-early-kdump.sh @@ -58,7 +58,7 @@ early_kdump_load() fi
$KEXEC ${EARLY_KEXEC_ARGS} $standard_kexec_args \ - --command-line="$EARLY_KDUMP_CMDLINE" \ + --command-line="$EARLY_KDUMP_CMDLINE rd.earlykdump" \ --initrd=$EARLY_KDUMP_INITRD $EARLY_KDUMP_KERNEL if [ $? == 0 ]; then echo "kexec: loaded early-kdump kernel" diff --git a/dracut-kdump-error-handler.sh b/dracut-kdump-error-handler.sh index 2f0f1d1..4f0e58c 100755 --- a/dracut-kdump-error-handler.sh +++ b/dracut-kdump-error-handler.sh @@ -1,5 +1,6 @@ #!/bin/sh
+. /lib/dracut-lib.sh . /lib/kdump-lib-initramfs.sh
set -o pipefail @@ -7,4 +8,8 @@ export PATH=$PATH:$KDUMP_SCRIPT_DIR
get_kdump_confs do_default_action -do_final_action +if getargbool 0 rd.earlykdump; then + do_earlykdump_final_action +else + do_final_action +fi diff --git a/dracut-kdump.sh b/dracut-kdump.sh index b75c2a5..119b006 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -201,4 +201,8 @@ if [ $DUMP_RETVAL -ne 0 ]; then exit 1 fi
-do_final_action +if getargbool 0 rd.earlykdump; then + do_earlykdump_final_action +else + do_final_action +fi diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 7ba99b6..f95cbcf 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -6,7 +6,7 @@ KDUMP_PATH="/var/crash" CORE_COLLECTOR="" DEFAULT_CORE_COLLECTOR="makedumpfile -l --message-level 1 -d 31" DMESG_COLLECTOR="/sbin/vmcore-dmesg" -DEFAULT_ACTION="systemctl reboot -f" +DEFAULT_ACTION="" DATEDIR=`date +%Y-%m-%d-%T` HOST_IP='127.0.0.1' DUMP_INSTRUCTION="" @@ -14,6 +14,7 @@ SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" KDUMP_SCRIPT_DIR="/kdumpscripts" DD_BLKSIZE=512 FINAL_ACTION="systemctl reboot -f" +EARLYKDUMP_FINAL_ACTION="systemctl poweroff" KDUMP_CONF="/etc/kdump.conf" KDUMP_PRE="" KDUMP_POST="" @@ -155,11 +156,20 @@ kdump_emergency_shell()
do_default_action() { - echo "Kdump: Executing default action $DEFAULT_ACTION" - eval $DEFAULT_ACTION + if [ $DEFAULT_ACTION == "" ]; then + echo "Kdump: default action string is null." + else + echo "Kdump: Executing default action $DEFAULT_ACTION" + eval $DEFAULT_ACTION + fi }
do_final_action() { eval $FINAL_ACTION } + +do_earlykdump_final_action() +{ + eval $EARLYKDUMP_FINAL_ACTION +} diff --git a/kdump.sysconfig b/kdump.sysconfig index ffe1df8..b011c1c 100644 --- a/kdump.sysconfig +++ b/kdump.sysconfig @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.aarch64 b/kdump.sysconfig.aarch64 index 0a6b14c..b8b8865 100644 --- a/kdump.sysconfig.aarch64 +++ b/kdump.sysconfig.aarch64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.i386 b/kdump.sysconfig.i386 index 18c407e..b9a2835 100644 --- a/kdump.sysconfig.i386 +++ b/kdump.sysconfig.i386 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64 b/kdump.sysconfig.ppc64 index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64 +++ b/kdump.sysconfig.ppc64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64le b/kdump.sysconfig.ppc64le index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64le +++ b/kdump.sysconfig.ppc64le @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.s390x b/kdump.sysconfig.s390x index b3aec3c..4c5fc00 100644 --- a/kdump.sysconfig.s390x +++ b/kdump.sysconfig.s390x @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.x86_64 b/kdump.sysconfig.x86_64 index f269d02..3cccbfc 100644 --- a/kdump.sysconfig.x86_64 +++ b/kdump.sysconfig.x86_64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdumpctl b/kdumpctl index fe6af22..a8fd53d 100755 --- a/kdumpctl +++ b/kdumpctl @@ -26,7 +26,7 @@ image_time=0 standard_kexec_args="-p"
# Some default values in case /etc/sysconfig/kdump doesn't include -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug rd.earlykdump"
if [ -f /etc/sysconfig/kdump ]; then . /etc/sysconfig/kdump @@ -942,6 +942,11 @@ check_default_config() fi }
+check_rd_earlykdump() +{ + egrep "rd.earlykdump" /proc/cmdline +} + start() { check_dump_feasibility @@ -969,7 +974,13 @@ start() check_current_status if [ $? == 0 ]; then echo "Kdump already running: [WARNING]" - return 0 + check_rd_earlykdump + #if earlykdump loaded, it will stop and start. + if [ $? -eq 0 ]; then + stop + else + return 0 + fi fi
if check_ssh_config; then
On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
By default, early kdump reboots the system after capturing the vmcore. If the problematic system is continuously crashing due to some issue during early boot stage, the system may fall into infinite loop restart like this:
boot -----> crash -----> early kdump (dump vmcore) ^ | '.........(reboot).............'
Didn't tell why it will fall into infinite loop in early kdump.
But now, the system crash at early stage is only captured by early kdump,
This 'now' is meaning the time for the current code, or for the code after this patch applied?
and the rest is captured by normal kdump. That to say, when normal kdump service starts, it will load it again and override early kdump. It is helpful to control the logic of early kdump and normal kdump separately in final action(it is called by kdump-capture.service). For example, early kdump always passes the 'rd.earlykdump' to the second kernel when early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' to the second kernel at any time. So they can be distinguished in the second kernel.
Signed-off-by: Lianbo Jiang lijiang@redhat.com
dracut-early-kdump.sh | 2 +- dracut-kdump-error-handler.sh | 7 ++++++- dracut-kdump.sh | 6 +++++- kdump-lib-initramfs.sh | 16 +++++++++++++--- kdump.sysconfig | 2 +- kdump.sysconfig.aarch64 | 2 +- kdump.sysconfig.i386 | 2 +- kdump.sysconfig.ppc64 | 2 +- kdump.sysconfig.ppc64le | 2 +- kdump.sysconfig.s390x | 2 +- kdump.sysconfig.x86_64 | 2 +- kdumpctl | 15 +++++++++++++-- 12 files changed, 45 insertions(+), 15 deletions(-)
diff --git a/dracut-early-kdump.sh b/dracut-early-kdump.sh index 34a9909..a799dbf 100755 --- a/dracut-early-kdump.sh +++ b/dracut-early-kdump.sh @@ -58,7 +58,7 @@ early_kdump_load() fi
$KEXEC ${EARLY_KEXEC_ARGS} $standard_kexec_args \
--command-line="$EARLY_KDUMP_CMDLINE" \
if [ $? == 0 ]; then echo "kexec: loaded early-kdump kernel"--command-line="$EARLY_KDUMP_CMDLINE rd.earlykdump" \ --initrd=$EARLY_KDUMP_INITRD $EARLY_KDUMP_KERNEL
diff --git a/dracut-kdump-error-handler.sh b/dracut-kdump-error-handler.sh index 2f0f1d1..4f0e58c 100755 --- a/dracut-kdump-error-handler.sh +++ b/dracut-kdump-error-handler.sh @@ -1,5 +1,6 @@ #!/bin/sh
+. /lib/dracut-lib.sh . /lib/kdump-lib-initramfs.sh
set -o pipefail @@ -7,4 +8,8 @@ export PATH=$PATH:$KDUMP_SCRIPT_DIR
get_kdump_confs do_default_action -do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi
Can we do it like this? if getargbool 0 rd.earlykdump; then FINAL_ACTION="systemctl poweroff" fi do_final_action
diff --git a/dracut-kdump.sh b/dracut-kdump.sh index b75c2a5..119b006 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -201,4 +201,8 @@ if [ $DUMP_RETVAL -ne 0 ]; then exit 1 fi
-do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 7ba99b6..f95cbcf 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -6,7 +6,7 @@ KDUMP_PATH="/var/crash" CORE_COLLECTOR="" DEFAULT_CORE_COLLECTOR="makedumpfile -l --message-level 1 -d 31" DMESG_COLLECTOR="/sbin/vmcore-dmesg" -DEFAULT_ACTION="systemctl reboot -f" +DEFAULT_ACTION="" DATEDIR=`date +%Y-%m-%d-%T` HOST_IP='127.0.0.1' DUMP_INSTRUCTION="" @@ -14,6 +14,7 @@ SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" KDUMP_SCRIPT_DIR="/kdumpscripts" DD_BLKSIZE=512 FINAL_ACTION="systemctl reboot -f" +EARLYKDUMP_FINAL_ACTION="systemctl poweroff" KDUMP_CONF="/etc/kdump.conf" KDUMP_PRE="" KDUMP_POST="" @@ -155,11 +156,20 @@ kdump_emergency_shell()
do_default_action() {
- echo "Kdump: Executing default action $DEFAULT_ACTION"
- eval $DEFAULT_ACTION
- if [ $DEFAULT_ACTION == "" ]; then
echo "Kdump: default action string is null."
- else
echo "Kdump: Executing default action $DEFAULT_ACTION"
eval $DEFAULT_ACTION
- fi
}
do_final_action() { eval $FINAL_ACTION }
+do_earlykdump_final_action() +{
- eval $EARLYKDUMP_FINAL_ACTION
+} diff --git a/kdump.sysconfig b/kdump.sysconfig index ffe1df8..b011c1c 100644 --- a/kdump.sysconfig +++ b/kdump.sysconfig @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
I don't understand. earlykdump get normal kernel's cmdline, then add rd.earlykdump, while normal kdump will reload and get normal kernel's cmdline again, why do you need to remove it here? Normal kdump will inherit early kdump's cmdline?
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.aarch64 b/kdump.sysconfig.aarch64 index 0a6b14c..b8b8865 100644 --- a/kdump.sysconfig.aarch64 +++ b/kdump.sysconfig.aarch64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.i386 b/kdump.sysconfig.i386 index 18c407e..b9a2835 100644 --- a/kdump.sysconfig.i386 +++ b/kdump.sysconfig.i386 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64 b/kdump.sysconfig.ppc64 index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64 +++ b/kdump.sysconfig.ppc64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64le b/kdump.sysconfig.ppc64le index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64le +++ b/kdump.sysconfig.ppc64le @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.s390x b/kdump.sysconfig.s390x index b3aec3c..4c5fc00 100644 --- a/kdump.sysconfig.s390x +++ b/kdump.sysconfig.s390x @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.x86_64 b/kdump.sysconfig.x86_64 index f269d02..3cccbfc 100644 --- a/kdump.sysconfig.x86_64 +++ b/kdump.sysconfig.x86_64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdumpctl b/kdumpctl index fe6af22..a8fd53d 100755 --- a/kdumpctl +++ b/kdumpctl @@ -26,7 +26,7 @@ image_time=0 standard_kexec_args="-p"
# Some default values in case /etc/sysconfig/kdump doesn't include -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug rd.earlykdump"
if [ -f /etc/sysconfig/kdump ]; then . /etc/sysconfig/kdump @@ -942,6 +942,11 @@ check_default_config() fi }
+check_rd_earlykdump() +{
- egrep "rd.earlykdump" /proc/cmdline
+}
start() { check_dump_feasibility @@ -969,7 +974,13 @@ start() check_current_status if [ $? == 0 ]; then echo "Kdump already running: [WARNING]"
return 0
check_rd_earlykdump
#if earlykdump loaded, it will stop and start.
if [ $? -eq 0 ]; then
stop
else
return 0
fi
fi
if check_ssh_config; then
-- 2.17.1
在 2018年12月20日 13:22, Baoquan He 写道:
On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
By default, early kdump reboots the system after capturing the vmcore. If the problematic system is continuously crashing due to some issue during early boot stage, the system may fall into infinite loop restart like this:
boot -----> crash -----> early kdump (dump vmcore) ^ | '.........(reboot).............'
Didn't tell why it will fall into infinite loop in early kdump.
Thanks for your comment.
Originally, early kdump and normal kdump are not distinguished. Early kdump only loads kernel and initramfs as early as possible, it uses the same logic as normal kdump captures vmcore and executes default/final action.
For normal kdump, the default/final actions are the reboot after capturing the vmcore, so early kdump is also the same action. When the problematic system is crashing at early boot stage, it will switch the kdump-capture service and dump vmcore, and then reboot system, boot, crash, dump vmcore... again and again.
But now, the system crash at early stage is only captured by early kdump,
This 'now' is meaning the time for the current code, or for the code after this patch applied?
After this patch applied.
and the rest is captured by normal kdump. That to say, when normal kdump service starts, it will load it again and override early kdump. It is helpful to control the logic of early kdump and normal kdump separately in final action(it is called by kdump-capture.service). For example, early kdump always passes the 'rd.earlykdump' to the second kernel when early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' to the second kernel at any time. So they can be distinguished in the second kernel.
Signed-off-by: Lianbo Jiang lijiang@redhat.com
dracut-early-kdump.sh | 2 +- dracut-kdump-error-handler.sh | 7 ++++++- dracut-kdump.sh | 6 +++++- kdump-lib-initramfs.sh | 16 +++++++++++++--- kdump.sysconfig | 2 +- kdump.sysconfig.aarch64 | 2 +- kdump.sysconfig.i386 | 2 +- kdump.sysconfig.ppc64 | 2 +- kdump.sysconfig.ppc64le | 2 +- kdump.sysconfig.s390x | 2 +- kdump.sysconfig.x86_64 | 2 +- kdumpctl | 15 +++++++++++++-- 12 files changed, 45 insertions(+), 15 deletions(-)
diff --git a/dracut-early-kdump.sh b/dracut-early-kdump.sh index 34a9909..a799dbf 100755 --- a/dracut-early-kdump.sh +++ b/dracut-early-kdump.sh @@ -58,7 +58,7 @@ early_kdump_load() fi
$KEXEC ${EARLY_KEXEC_ARGS} $standard_kexec_args \
--command-line="$EARLY_KDUMP_CMDLINE" \
if [ $? == 0 ]; then echo "kexec: loaded early-kdump kernel"--command-line="$EARLY_KDUMP_CMDLINE rd.earlykdump" \ --initrd=$EARLY_KDUMP_INITRD $EARLY_KDUMP_KERNEL
diff --git a/dracut-kdump-error-handler.sh b/dracut-kdump-error-handler.sh index 2f0f1d1..4f0e58c 100755 --- a/dracut-kdump-error-handler.sh +++ b/dracut-kdump-error-handler.sh @@ -1,5 +1,6 @@ #!/bin/sh
+. /lib/dracut-lib.sh . /lib/kdump-lib-initramfs.sh
set -o pipefail @@ -7,4 +8,8 @@ export PATH=$PATH:$KDUMP_SCRIPT_DIR
get_kdump_confs do_default_action -do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi
Can we do it like this? if getargbool 0 rd.earlykdump; then FINAL_ACTION="systemctl poweroff" fi do_final_action
Sure, of course.
diff --git a/dracut-kdump.sh b/dracut-kdump.sh index b75c2a5..119b006 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -201,4 +201,8 @@ if [ $DUMP_RETVAL -ne 0 ]; then exit 1 fi
-do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 7ba99b6..f95cbcf 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -6,7 +6,7 @@ KDUMP_PATH="/var/crash" CORE_COLLECTOR="" DEFAULT_CORE_COLLECTOR="makedumpfile -l --message-level 1 -d 31" DMESG_COLLECTOR="/sbin/vmcore-dmesg" -DEFAULT_ACTION="systemctl reboot -f" +DEFAULT_ACTION="" DATEDIR=`date +%Y-%m-%d-%T` HOST_IP='127.0.0.1' DUMP_INSTRUCTION="" @@ -14,6 +14,7 @@ SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" KDUMP_SCRIPT_DIR="/kdumpscripts" DD_BLKSIZE=512 FINAL_ACTION="systemctl reboot -f" +EARLYKDUMP_FINAL_ACTION="systemctl poweroff" KDUMP_CONF="/etc/kdump.conf" KDUMP_PRE="" KDUMP_POST="" @@ -155,11 +156,20 @@ kdump_emergency_shell()
do_default_action() {
- echo "Kdump: Executing default action $DEFAULT_ACTION"
- eval $DEFAULT_ACTION
- if [ $DEFAULT_ACTION == "" ]; then
echo "Kdump: default action string is null."
- else
echo "Kdump: Executing default action $DEFAULT_ACTION"
eval $DEFAULT_ACTION
- fi
}
do_final_action() { eval $FINAL_ACTION }
+do_earlykdump_final_action() +{
- eval $EARLYKDUMP_FINAL_ACTION
+} diff --git a/kdump.sysconfig b/kdump.sysconfig index ffe1df8..b011c1c 100644 --- a/kdump.sysconfig +++ b/kdump.sysconfig @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
I don't understand. earlykdump get normal kernel's cmdline, then add rd.earlykdump, while normal kdump will reload and get normal kernel's cmdline again, why do you need to remove it here? Normal kdump will inherit early kdump's cmdline?
The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump and normal kdump can get the same parameters from /proc/cmdline in the first kernel.
Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't need it, normal kdump needs to remove the rd.earlykdump.
So which can distinguish early kdump and normal kdump in the second kernel. It helps to control the logic of kdump capture service. For example: default action/final action.
Thanks.
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.aarch64 b/kdump.sysconfig.aarch64 index 0a6b14c..b8b8865 100644 --- a/kdump.sysconfig.aarch64 +++ b/kdump.sysconfig.aarch64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.i386 b/kdump.sysconfig.i386 index 18c407e..b9a2835 100644 --- a/kdump.sysconfig.i386 +++ b/kdump.sysconfig.i386 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64 b/kdump.sysconfig.ppc64 index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64 +++ b/kdump.sysconfig.ppc64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64le b/kdump.sysconfig.ppc64le index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64le +++ b/kdump.sysconfig.ppc64le @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.s390x b/kdump.sysconfig.s390x index b3aec3c..4c5fc00 100644 --- a/kdump.sysconfig.s390x +++ b/kdump.sysconfig.s390x @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.x86_64 b/kdump.sysconfig.x86_64 index f269d02..3cccbfc 100644 --- a/kdump.sysconfig.x86_64 +++ b/kdump.sysconfig.x86_64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdumpctl b/kdumpctl index fe6af22..a8fd53d 100755 --- a/kdumpctl +++ b/kdumpctl @@ -26,7 +26,7 @@ image_time=0 standard_kexec_args="-p"
# Some default values in case /etc/sysconfig/kdump doesn't include -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug rd.earlykdump"
if [ -f /etc/sysconfig/kdump ]; then . /etc/sysconfig/kdump @@ -942,6 +942,11 @@ check_default_config() fi }
+check_rd_earlykdump() +{
- egrep "rd.earlykdump" /proc/cmdline
+}
start() { check_dump_feasibility @@ -969,7 +974,13 @@ start() check_current_status if [ $? == 0 ]; then echo "Kdump already running: [WARNING]"
return 0
check_rd_earlykdump
#if earlykdump loaded, it will stop and start.
if [ $? -eq 0 ]; then
stop
else
return 0
fi
fi
if check_ssh_config; then
-- 2.17.1
On 12/20/18 at 03:31pm, lijiang wrote:
在 2018年12月20日 13:22, Baoquan He 写道:
On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
What:
By default, early kdump reboots the system after capturing the vmcore. If the problematic system is continuously crashing due to some issue during early boot stage, the system may fall into infinite loop restart like this:
boot -----> crash -----> early kdump (dump vmcore) ^ | '.........(reboot).............'
Didn't tell why it will fall into infinite loop in early kdump.
Thanks for your comment.
Why:
Originally, early kdump and normal kdump are not distinguished. Early kdump only loads kernel and initramfs as early as possible, it uses the same logic as normal kdump captures vmcore and executes default/final action.
Plus: earlykernel is captured the crash during kernel initialization, it might be pid 0 process only, so could crash alwasy on that machine.
For normal kdump, the default/final actions are the reboot after capturing the vmcore, so early kdump is also the same action. When the problematic system is crashing at early boot stage, it will switch the kdump-capture service and dump vmcore, and then reboot system, boot, crash, dump vmcore... again and again.
How:
To fix it, the system crash at early stage is only captured by early kdump, and the rest is captured by normal kdump. That to say, when normal kdump ...
But now, the system crash at early stage is only captured by early kdump,
This 'now' is meaning the time for the current code, or for the code after this patch applied?
After this patch applied.
and the rest is captured by normal kdump. That to say, when normal kdump service starts, it will load it again and override early kdump. It is helpful to control the logic of early kdump and normal kdump separately in final action(it is called by kdump-capture.service). For example, early kdump always passes the 'rd.earlykdump' to the second kernel when early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' to the second kernel at any time. So they can be distinguished in the second kernel.
Signed-off-by: Lianbo Jiang lijiang@redhat.com
dracut-early-kdump.sh | 2 +- dracut-kdump-error-handler.sh | 7 ++++++- dracut-kdump.sh | 6 +++++- kdump-lib-initramfs.sh | 16 +++++++++++++--- kdump.sysconfig | 2 +- kdump.sysconfig.aarch64 | 2 +- kdump.sysconfig.i386 | 2 +- kdump.sysconfig.ppc64 | 2 +- kdump.sysconfig.ppc64le | 2 +- kdump.sysconfig.s390x | 2 +- kdump.sysconfig.x86_64 | 2 +- kdumpctl | 15 +++++++++++++-- 12 files changed, 45 insertions(+), 15 deletions(-)
diff --git a/dracut-early-kdump.sh b/dracut-early-kdump.sh index 34a9909..a799dbf 100755 --- a/dracut-early-kdump.sh +++ b/dracut-early-kdump.sh @@ -58,7 +58,7 @@ early_kdump_load() fi
$KEXEC ${EARLY_KEXEC_ARGS} $standard_kexec_args \
--command-line="$EARLY_KDUMP_CMDLINE" \
if [ $? == 0 ]; then echo "kexec: loaded early-kdump kernel"--command-line="$EARLY_KDUMP_CMDLINE rd.earlykdump" \ --initrd=$EARLY_KDUMP_INITRD $EARLY_KDUMP_KERNEL
diff --git a/dracut-kdump-error-handler.sh b/dracut-kdump-error-handler.sh index 2f0f1d1..4f0e58c 100755 --- a/dracut-kdump-error-handler.sh +++ b/dracut-kdump-error-handler.sh @@ -1,5 +1,6 @@ #!/bin/sh
+. /lib/dracut-lib.sh . /lib/kdump-lib-initramfs.sh
set -o pipefail @@ -7,4 +8,8 @@ export PATH=$PATH:$KDUMP_SCRIPT_DIR
get_kdump_confs do_default_action -do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi
Can we do it like this? if getargbool 0 rd.earlykdump; then FINAL_ACTION="systemctl poweroff" fi do_final_action
Sure, of course.
diff --git a/dracut-kdump.sh b/dracut-kdump.sh index b75c2a5..119b006 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -201,4 +201,8 @@ if [ $DUMP_RETVAL -ne 0 ]; then exit 1 fi
-do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 7ba99b6..f95cbcf 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -6,7 +6,7 @@ KDUMP_PATH="/var/crash" CORE_COLLECTOR="" DEFAULT_CORE_COLLECTOR="makedumpfile -l --message-level 1 -d 31" DMESG_COLLECTOR="/sbin/vmcore-dmesg" -DEFAULT_ACTION="systemctl reboot -f" +DEFAULT_ACTION="" DATEDIR=`date +%Y-%m-%d-%T` HOST_IP='127.0.0.1' DUMP_INSTRUCTION="" @@ -14,6 +14,7 @@ SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" KDUMP_SCRIPT_DIR="/kdumpscripts" DD_BLKSIZE=512 FINAL_ACTION="systemctl reboot -f" +EARLYKDUMP_FINAL_ACTION="systemctl poweroff" KDUMP_CONF="/etc/kdump.conf" KDUMP_PRE="" KDUMP_POST="" @@ -155,11 +156,20 @@ kdump_emergency_shell()
do_default_action() {
- echo "Kdump: Executing default action $DEFAULT_ACTION"
- eval $DEFAULT_ACTION
- if [ $DEFAULT_ACTION == "" ]; then
echo "Kdump: default action string is null."
- else
echo "Kdump: Executing default action $DEFAULT_ACTION"
eval $DEFAULT_ACTION
- fi
}
do_final_action() { eval $FINAL_ACTION }
+do_earlykdump_final_action() +{
- eval $EARLYKDUMP_FINAL_ACTION
+} diff --git a/kdump.sysconfig b/kdump.sysconfig index ffe1df8..b011c1c 100644 --- a/kdump.sysconfig +++ b/kdump.sysconfig @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
I don't understand. earlykdump get normal kernel's cmdline, then add rd.earlykdump, while normal kdump will reload and get normal kernel's cmdline again, why do you need to remove it here? Normal kdump will inherit early kdump's cmdline?
The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump and normal kdump can get the same parameters from /proc/cmdline in the first kernel.
Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't need it, normal kdump needs to remove the rd.earlykdump.
So which can distinguish early kdump and normal kdump in the second kernel. It helps to control the logic of kdump capture service. For example: default action/final action.
Thanks.
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.aarch64 b/kdump.sysconfig.aarch64 index 0a6b14c..b8b8865 100644 --- a/kdump.sysconfig.aarch64 +++ b/kdump.sysconfig.aarch64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.i386 b/kdump.sysconfig.i386 index 18c407e..b9a2835 100644 --- a/kdump.sysconfig.i386 +++ b/kdump.sysconfig.i386 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64 b/kdump.sysconfig.ppc64 index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64 +++ b/kdump.sysconfig.ppc64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64le b/kdump.sysconfig.ppc64le index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64le +++ b/kdump.sysconfig.ppc64le @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.s390x b/kdump.sysconfig.s390x index b3aec3c..4c5fc00 100644 --- a/kdump.sysconfig.s390x +++ b/kdump.sysconfig.s390x @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.x86_64 b/kdump.sysconfig.x86_64 index f269d02..3cccbfc 100644 --- a/kdump.sysconfig.x86_64 +++ b/kdump.sysconfig.x86_64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdumpctl b/kdumpctl index fe6af22..a8fd53d 100755 --- a/kdumpctl +++ b/kdumpctl @@ -26,7 +26,7 @@ image_time=0 standard_kexec_args="-p"
# Some default values in case /etc/sysconfig/kdump doesn't include -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug rd.earlykdump"
if [ -f /etc/sysconfig/kdump ]; then . /etc/sysconfig/kdump @@ -942,6 +942,11 @@ check_default_config() fi }
+check_rd_earlykdump() +{
- egrep "rd.earlykdump" /proc/cmdline
+}
start() { check_dump_feasibility @@ -969,7 +974,13 @@ start() check_current_status if [ $? == 0 ]; then echo "Kdump already running: [WARNING]"
return 0
check_rd_earlykdump
#if earlykdump loaded, it will stop and start.
if [ $? -eq 0 ]; then
stop
else
return 0
fi
fi
if check_ssh_config; then
-- 2.17.1
On 12/20/18 at 03:37pm, Baoquan He wrote:
On 12/20/18 at 03:31pm, lijiang wrote:
在 2018年12月20日 13:22, Baoquan He 写道:
On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
What:
By default, early kdump reboots the system after capturing the vmcore. If the problematic system is continuously crashing due to some issue during early boot stage, the system may fall into infinite loop restart like this:
boot -----> crash -----> early kdump (dump vmcore) ^ | '.........(reboot).............'
Didn't tell why it will fall into infinite loop in early kdump.
Thanks for your comment.
Why:
Originally, early kdump and normal kdump are not distinguished. Early kdump only loads kernel and initramfs as early as possible, it uses the same logic as normal kdump captures vmcore and executes default/final action.
Plus: earlykernel is captured the crash during kernel initialization, it
~~~ capturing
might be pid 0 process only, so could crash alwasy on that machine.
For normal kdump, the default/final actions are the reboot after capturing the vmcore, so early kdump is also the same action. When the problematic system is crashing at early boot stage, it will switch the kdump-capture service and dump vmcore, and then reboot system, boot, crash, dump vmcore... again and again.
How:
To fix it, the system crash at early stage is only captured by early kdump, and the rest is captured by normal kdump. That to say, when normal kdump ...
But now, the system crash at early stage is only captured by early kdump,
This 'now' is meaning the time for the current code, or for the code after this patch applied?
After this patch applied.
and the rest is captured by normal kdump. That to say, when normal kdump service starts, it will load it again and override early kdump. It is helpful to control the logic of early kdump and normal kdump separately in final action(it is called by kdump-capture.service). For example, early kdump always passes the 'rd.earlykdump' to the second kernel when early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' to the second kernel at any time. So they can be distinguished in the second kernel.
Signed-off-by: Lianbo Jiang lijiang@redhat.com
dracut-early-kdump.sh | 2 +- dracut-kdump-error-handler.sh | 7 ++++++- dracut-kdump.sh | 6 +++++- kdump-lib-initramfs.sh | 16 +++++++++++++--- kdump.sysconfig | 2 +- kdump.sysconfig.aarch64 | 2 +- kdump.sysconfig.i386 | 2 +- kdump.sysconfig.ppc64 | 2 +- kdump.sysconfig.ppc64le | 2 +- kdump.sysconfig.s390x | 2 +- kdump.sysconfig.x86_64 | 2 +- kdumpctl | 15 +++++++++++++-- 12 files changed, 45 insertions(+), 15 deletions(-)
diff --git a/dracut-early-kdump.sh b/dracut-early-kdump.sh index 34a9909..a799dbf 100755 --- a/dracut-early-kdump.sh +++ b/dracut-early-kdump.sh @@ -58,7 +58,7 @@ early_kdump_load() fi
$KEXEC ${EARLY_KEXEC_ARGS} $standard_kexec_args \
--command-line="$EARLY_KDUMP_CMDLINE" \
if [ $? == 0 ]; then echo "kexec: loaded early-kdump kernel"--command-line="$EARLY_KDUMP_CMDLINE rd.earlykdump" \ --initrd=$EARLY_KDUMP_INITRD $EARLY_KDUMP_KERNEL
diff --git a/dracut-kdump-error-handler.sh b/dracut-kdump-error-handler.sh index 2f0f1d1..4f0e58c 100755 --- a/dracut-kdump-error-handler.sh +++ b/dracut-kdump-error-handler.sh @@ -1,5 +1,6 @@ #!/bin/sh
+. /lib/dracut-lib.sh . /lib/kdump-lib-initramfs.sh
set -o pipefail @@ -7,4 +8,8 @@ export PATH=$PATH:$KDUMP_SCRIPT_DIR
get_kdump_confs do_default_action -do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi
Can we do it like this? if getargbool 0 rd.earlykdump; then FINAL_ACTION="systemctl poweroff" fi do_final_action
Sure, of course.
diff --git a/dracut-kdump.sh b/dracut-kdump.sh index b75c2a5..119b006 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -201,4 +201,8 @@ if [ $DUMP_RETVAL -ne 0 ]; then exit 1 fi
-do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 7ba99b6..f95cbcf 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -6,7 +6,7 @@ KDUMP_PATH="/var/crash" CORE_COLLECTOR="" DEFAULT_CORE_COLLECTOR="makedumpfile -l --message-level 1 -d 31" DMESG_COLLECTOR="/sbin/vmcore-dmesg" -DEFAULT_ACTION="systemctl reboot -f" +DEFAULT_ACTION="" DATEDIR=`date +%Y-%m-%d-%T` HOST_IP='127.0.0.1' DUMP_INSTRUCTION="" @@ -14,6 +14,7 @@ SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" KDUMP_SCRIPT_DIR="/kdumpscripts" DD_BLKSIZE=512 FINAL_ACTION="systemctl reboot -f" +EARLYKDUMP_FINAL_ACTION="systemctl poweroff" KDUMP_CONF="/etc/kdump.conf" KDUMP_PRE="" KDUMP_POST="" @@ -155,11 +156,20 @@ kdump_emergency_shell()
do_default_action() {
- echo "Kdump: Executing default action $DEFAULT_ACTION"
- eval $DEFAULT_ACTION
- if [ $DEFAULT_ACTION == "" ]; then
echo "Kdump: default action string is null."
- else
echo "Kdump: Executing default action $DEFAULT_ACTION"
eval $DEFAULT_ACTION
- fi
}
do_final_action() { eval $FINAL_ACTION }
+do_earlykdump_final_action() +{
- eval $EARLYKDUMP_FINAL_ACTION
+} diff --git a/kdump.sysconfig b/kdump.sysconfig index ffe1df8..b011c1c 100644 --- a/kdump.sysconfig +++ b/kdump.sysconfig @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
I don't understand. earlykdump get normal kernel's cmdline, then add rd.earlykdump, while normal kdump will reload and get normal kernel's cmdline again, why do you need to remove it here? Normal kdump will inherit early kdump's cmdline?
The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump and normal kdump can get the same parameters from /proc/cmdline in the first kernel.
Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't need it, normal kdump needs to remove the rd.earlykdump.
So which can distinguish early kdump and normal kdump in the second kernel. It helps to control the logic of kdump capture service. For example: default action/final action.
Thanks.
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.aarch64 b/kdump.sysconfig.aarch64 index 0a6b14c..b8b8865 100644 --- a/kdump.sysconfig.aarch64 +++ b/kdump.sysconfig.aarch64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.i386 b/kdump.sysconfig.i386 index 18c407e..b9a2835 100644 --- a/kdump.sysconfig.i386 +++ b/kdump.sysconfig.i386 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64 b/kdump.sysconfig.ppc64 index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64 +++ b/kdump.sysconfig.ppc64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64le b/kdump.sysconfig.ppc64le index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64le +++ b/kdump.sysconfig.ppc64le @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.s390x b/kdump.sysconfig.s390x index b3aec3c..4c5fc00 100644 --- a/kdump.sysconfig.s390x +++ b/kdump.sysconfig.s390x @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.x86_64 b/kdump.sysconfig.x86_64 index f269d02..3cccbfc 100644 --- a/kdump.sysconfig.x86_64 +++ b/kdump.sysconfig.x86_64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdumpctl b/kdumpctl index fe6af22..a8fd53d 100755 --- a/kdumpctl +++ b/kdumpctl @@ -26,7 +26,7 @@ image_time=0 standard_kexec_args="-p"
# Some default values in case /etc/sysconfig/kdump doesn't include -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug rd.earlykdump"
if [ -f /etc/sysconfig/kdump ]; then . /etc/sysconfig/kdump @@ -942,6 +942,11 @@ check_default_config() fi }
+check_rd_earlykdump() +{
- egrep "rd.earlykdump" /proc/cmdline
+}
start() { check_dump_feasibility @@ -969,7 +974,13 @@ start() check_current_status if [ $? == 0 ]; then echo "Kdump already running: [WARNING]"
return 0
check_rd_earlykdump
#if earlykdump loaded, it will stop and start.
if [ $? -eq 0 ]; then
stop
else
return 0
fi
fi
if check_ssh_config; then
-- 2.17.1
On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
By default, early kdump reboots the system after capturing the vmcore. If the problematic system is continuously crashing due to some issue during early boot stage, the system may fall into infinite loop restart like this:
boot -----> crash -----> early kdump (dump vmcore) ^ | '.........(reboot).............'
But now, the system crash at early stage is only captured by early kdump, and the rest is captured by normal kdump. That to say, when normal kdump service starts, it will load it again and override early kdump. It is helpful to control the logic of early kdump and normal kdump separately in final action(it is called by kdump-capture.service). For example, early kdump always passes the 'rd.earlykdump' to the second kernel when early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' to the second kernel at any time. So they can be distinguished in the second kernel.
Hmm, I'm confused about the param passing above.
Early or non early just means about the service loading phase, in initramfs or not, I notice dracut/systemd will print some message about they are running in initramfs, so probably you can check how to get it with same way, if this is not initramfs then just unload before the check in kdump loading.
The picture like below:
Kernel boot ->
initramfs --- early kdump load ---- Mark A ---- initramfs switch root
system startup (real root fs) service a service b ... (eg. networking etc.) kdump service start -----Mark B ----- load kdump kernel again
The problem will happen between Mark A and Mark B, during this period, there could be repeated crash -> earlykdump_load, there might be some random crash as well since during the real root fs service startup, for example after network is ready if some network workload cause a panic, it maybe not 100% reproducible, so it seems we still need to make the poweroff configurable. eg.
default is poweroff, but one can choose if he can.
Signed-off-by: Lianbo Jiang lijiang@redhat.com
dracut-early-kdump.sh | 2 +- dracut-kdump-error-handler.sh | 7 ++++++- dracut-kdump.sh | 6 +++++- kdump-lib-initramfs.sh | 16 +++++++++++++--- kdump.sysconfig | 2 +- kdump.sysconfig.aarch64 | 2 +- kdump.sysconfig.i386 | 2 +- kdump.sysconfig.ppc64 | 2 +- kdump.sysconfig.ppc64le | 2 +- kdump.sysconfig.s390x | 2 +- kdump.sysconfig.x86_64 | 2 +- kdumpctl | 15 +++++++++++++-- 12 files changed, 45 insertions(+), 15 deletions(-)
diff --git a/dracut-early-kdump.sh b/dracut-early-kdump.sh index 34a9909..a799dbf 100755 --- a/dracut-early-kdump.sh +++ b/dracut-early-kdump.sh @@ -58,7 +58,7 @@ early_kdump_load() fi
$KEXEC ${EARLY_KEXEC_ARGS} $standard_kexec_args \
--command-line="$EARLY_KDUMP_CMDLINE" \
if [ $? == 0 ]; then echo "kexec: loaded early-kdump kernel"--command-line="$EARLY_KDUMP_CMDLINE rd.earlykdump" \ --initrd=$EARLY_KDUMP_INITRD $EARLY_KDUMP_KERNEL
diff --git a/dracut-kdump-error-handler.sh b/dracut-kdump-error-handler.sh index 2f0f1d1..4f0e58c 100755 --- a/dracut-kdump-error-handler.sh +++ b/dracut-kdump-error-handler.sh @@ -1,5 +1,6 @@ #!/bin/sh
+. /lib/dracut-lib.sh . /lib/kdump-lib-initramfs.sh
set -o pipefail @@ -7,4 +8,8 @@ export PATH=$PATH:$KDUMP_SCRIPT_DIR
get_kdump_confs do_default_action -do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi diff --git a/dracut-kdump.sh b/dracut-kdump.sh index b75c2a5..119b006 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -201,4 +201,8 @@ if [ $DUMP_RETVAL -ne 0 ]; then exit 1 fi
-do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 7ba99b6..f95cbcf 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -6,7 +6,7 @@ KDUMP_PATH="/var/crash" CORE_COLLECTOR="" DEFAULT_CORE_COLLECTOR="makedumpfile -l --message-level 1 -d 31" DMESG_COLLECTOR="/sbin/vmcore-dmesg" -DEFAULT_ACTION="systemctl reboot -f" +DEFAULT_ACTION="" DATEDIR=`date +%Y-%m-%d-%T` HOST_IP='127.0.0.1' DUMP_INSTRUCTION="" @@ -14,6 +14,7 @@ SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" KDUMP_SCRIPT_DIR="/kdumpscripts" DD_BLKSIZE=512 FINAL_ACTION="systemctl reboot -f" +EARLYKDUMP_FINAL_ACTION="systemctl poweroff" KDUMP_CONF="/etc/kdump.conf" KDUMP_PRE="" KDUMP_POST="" @@ -155,11 +156,20 @@ kdump_emergency_shell()
do_default_action() {
- echo "Kdump: Executing default action $DEFAULT_ACTION"
- eval $DEFAULT_ACTION
- if [ $DEFAULT_ACTION == "" ]; then
echo "Kdump: default action string is null."
- else
echo "Kdump: Executing default action $DEFAULT_ACTION"
eval $DEFAULT_ACTION
- fi
}
do_final_action() { eval $FINAL_ACTION }
+do_earlykdump_final_action() +{
- eval $EARLYKDUMP_FINAL_ACTION
+} diff --git a/kdump.sysconfig b/kdump.sysconfig index ffe1df8..b011c1c 100644 --- a/kdump.sysconfig +++ b/kdump.sysconfig @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.aarch64 b/kdump.sysconfig.aarch64 index 0a6b14c..b8b8865 100644 --- a/kdump.sysconfig.aarch64 +++ b/kdump.sysconfig.aarch64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.i386 b/kdump.sysconfig.i386 index 18c407e..b9a2835 100644 --- a/kdump.sysconfig.i386 +++ b/kdump.sysconfig.i386 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64 b/kdump.sysconfig.ppc64 index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64 +++ b/kdump.sysconfig.ppc64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64le b/kdump.sysconfig.ppc64le index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64le +++ b/kdump.sysconfig.ppc64le @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.s390x b/kdump.sysconfig.s390x index b3aec3c..4c5fc00 100644 --- a/kdump.sysconfig.s390x +++ b/kdump.sysconfig.s390x @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.x86_64 b/kdump.sysconfig.x86_64 index f269d02..3cccbfc 100644 --- a/kdump.sysconfig.x86_64 +++ b/kdump.sysconfig.x86_64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdumpctl b/kdumpctl index fe6af22..a8fd53d 100755 --- a/kdumpctl +++ b/kdumpctl @@ -26,7 +26,7 @@ image_time=0 standard_kexec_args="-p"
# Some default values in case /etc/sysconfig/kdump doesn't include -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug rd.earlykdump"
if [ -f /etc/sysconfig/kdump ]; then . /etc/sysconfig/kdump @@ -942,6 +942,11 @@ check_default_config() fi }
+check_rd_earlykdump() +{
- egrep "rd.earlykdump" /proc/cmdline
+}
start() { check_dump_feasibility @@ -969,7 +974,13 @@ start() check_current_status if [ $? == 0 ]; then echo "Kdump already running: [WARNING]"
return 0
check_rd_earlykdump
#if earlykdump loaded, it will stop and start.
if [ $? -eq 0 ]; then
stop
kdumpctl start can run not only by system startup services, one can also run it manually or in udev rule.
The checking of kernel cmdline seems not enough.
else
return 0
fi
fi
if check_ssh_config; then
-- 2.17.1
Thanks Dave
+ Emma On 12/20/18 at 01:57pm, Dave Young wrote:
On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
By default, early kdump reboots the system after capturing the vmcore. If the problematic system is continuously crashing due to some issue during early boot stage, the system may fall into infinite loop restart like this:
boot -----> crash -----> early kdump (dump vmcore) ^ | '.........(reboot).............'
But now, the system crash at early stage is only captured by early kdump, and the rest is captured by normal kdump. That to say, when normal kdump service starts, it will load it again and override early kdump. It is helpful to control the logic of early kdump and normal kdump separately in final action(it is called by kdump-capture.service). For example, early kdump always passes the 'rd.earlykdump' to the second kernel when early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' to the second kernel at any time. So they can be distinguished in the second kernel.
Hmm, I'm confused about the param passing above.
Early or non early just means about the service loading phase, in initramfs or not, I notice dracut/systemd will print some message about they are running in initramfs, so probably you can check how to get it with same way, if this is not initramfs then just unload before the check in kdump loading.
The picture like below:
Kernel boot ->
initramfs --- early kdump load ---- Mark A ---- initramfs switch root
system startup (real root fs) service a service b ... (eg. networking etc.) kdump service start
-----Mark B ----- load kdump kernel again
The problem will happen between Mark A and Mark B, during this period, there could be repeated crash -> earlykdump_load, there might be some random crash as well since during the real root fs service startup, for example after network is ready if some network workload cause a panic, it maybe not 100% reproducible, so it seems we still need to make the poweroff configurable. eg.
default is poweroff, but one can choose if he can.
Signed-off-by: Lianbo Jiang lijiang@redhat.com
dracut-early-kdump.sh | 2 +- dracut-kdump-error-handler.sh | 7 ++++++- dracut-kdump.sh | 6 +++++- kdump-lib-initramfs.sh | 16 +++++++++++++--- kdump.sysconfig | 2 +- kdump.sysconfig.aarch64 | 2 +- kdump.sysconfig.i386 | 2 +- kdump.sysconfig.ppc64 | 2 +- kdump.sysconfig.ppc64le | 2 +- kdump.sysconfig.s390x | 2 +- kdump.sysconfig.x86_64 | 2 +- kdumpctl | 15 +++++++++++++-- 12 files changed, 45 insertions(+), 15 deletions(-)
diff --git a/dracut-early-kdump.sh b/dracut-early-kdump.sh index 34a9909..a799dbf 100755 --- a/dracut-early-kdump.sh +++ b/dracut-early-kdump.sh @@ -58,7 +58,7 @@ early_kdump_load() fi
$KEXEC ${EARLY_KEXEC_ARGS} $standard_kexec_args \
--command-line="$EARLY_KDUMP_CMDLINE" \
if [ $? == 0 ]; then echo "kexec: loaded early-kdump kernel"--command-line="$EARLY_KDUMP_CMDLINE rd.earlykdump" \ --initrd=$EARLY_KDUMP_INITRD $EARLY_KDUMP_KERNEL
diff --git a/dracut-kdump-error-handler.sh b/dracut-kdump-error-handler.sh index 2f0f1d1..4f0e58c 100755 --- a/dracut-kdump-error-handler.sh +++ b/dracut-kdump-error-handler.sh @@ -1,5 +1,6 @@ #!/bin/sh
+. /lib/dracut-lib.sh . /lib/kdump-lib-initramfs.sh
set -o pipefail @@ -7,4 +8,8 @@ export PATH=$PATH:$KDUMP_SCRIPT_DIR
get_kdump_confs do_default_action -do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi diff --git a/dracut-kdump.sh b/dracut-kdump.sh index b75c2a5..119b006 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -201,4 +201,8 @@ if [ $DUMP_RETVAL -ne 0 ]; then exit 1 fi
-do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 7ba99b6..f95cbcf 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -6,7 +6,7 @@ KDUMP_PATH="/var/crash" CORE_COLLECTOR="" DEFAULT_CORE_COLLECTOR="makedumpfile -l --message-level 1 -d 31" DMESG_COLLECTOR="/sbin/vmcore-dmesg" -DEFAULT_ACTION="systemctl reboot -f" +DEFAULT_ACTION="" DATEDIR=`date +%Y-%m-%d-%T` HOST_IP='127.0.0.1' DUMP_INSTRUCTION="" @@ -14,6 +14,7 @@ SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" KDUMP_SCRIPT_DIR="/kdumpscripts" DD_BLKSIZE=512 FINAL_ACTION="systemctl reboot -f" +EARLYKDUMP_FINAL_ACTION="systemctl poweroff" KDUMP_CONF="/etc/kdump.conf" KDUMP_PRE="" KDUMP_POST="" @@ -155,11 +156,20 @@ kdump_emergency_shell()
do_default_action() {
- echo "Kdump: Executing default action $DEFAULT_ACTION"
- eval $DEFAULT_ACTION
- if [ $DEFAULT_ACTION == "" ]; then
echo "Kdump: default action string is null."
- else
echo "Kdump: Executing default action $DEFAULT_ACTION"
eval $DEFAULT_ACTION
- fi
}
do_final_action() { eval $FINAL_ACTION }
+do_earlykdump_final_action() +{
- eval $EARLYKDUMP_FINAL_ACTION
+} diff --git a/kdump.sysconfig b/kdump.sysconfig index ffe1df8..b011c1c 100644 --- a/kdump.sysconfig +++ b/kdump.sysconfig @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.aarch64 b/kdump.sysconfig.aarch64 index 0a6b14c..b8b8865 100644 --- a/kdump.sysconfig.aarch64 +++ b/kdump.sysconfig.aarch64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.i386 b/kdump.sysconfig.i386 index 18c407e..b9a2835 100644 --- a/kdump.sysconfig.i386 +++ b/kdump.sysconfig.i386 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64 b/kdump.sysconfig.ppc64 index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64 +++ b/kdump.sysconfig.ppc64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64le b/kdump.sysconfig.ppc64le index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64le +++ b/kdump.sysconfig.ppc64le @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.s390x b/kdump.sysconfig.s390x index b3aec3c..4c5fc00 100644 --- a/kdump.sysconfig.s390x +++ b/kdump.sysconfig.s390x @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.x86_64 b/kdump.sysconfig.x86_64 index f269d02..3cccbfc 100644 --- a/kdump.sysconfig.x86_64 +++ b/kdump.sysconfig.x86_64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdumpctl b/kdumpctl index fe6af22..a8fd53d 100755 --- a/kdumpctl +++ b/kdumpctl @@ -26,7 +26,7 @@ image_time=0 standard_kexec_args="-p"
# Some default values in case /etc/sysconfig/kdump doesn't include -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug rd.earlykdump"
if [ -f /etc/sysconfig/kdump ]; then . /etc/sysconfig/kdump @@ -942,6 +942,11 @@ check_default_config() fi }
+check_rd_earlykdump() +{
- egrep "rd.earlykdump" /proc/cmdline
+}
start() { check_dump_feasibility @@ -969,7 +974,13 @@ start() check_current_status if [ $? == 0 ]; then echo "Kdump already running: [WARNING]"
return 0
check_rd_earlykdump
#if earlykdump loaded, it will stop and start.
if [ $? -eq 0 ]; then
stop
kdumpctl start can run not only by system startup services, one can also run it manually or in udev rule.
The checking of kernel cmdline seems not enough.
else
return 0
fi
fi
if check_ssh_config; then
-- 2.17.1
Thanks Dave
----- Original Message -----
From: "Dave Young" dyoung@redhat.com To: "Lianbo Jiang" lijiang@redhat.com Cc: kexec@lists.fedoraproject.org, bhe@redhat.com, xiawu@redhat.com Sent: Thursday, December 20, 2018 2:06:17 PM Subject: Re: [PATCH] Avoid falling into infinite loop restart when using a problematic system
- Emma
On 12/20/18 at 01:57pm, Dave Young wrote:
On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
By default, early kdump reboots the system after capturing the vmcore. If the problematic system is continuously crashing due to some issue during early boot stage, the system may fall into infinite loop restart like this:
boot -----> crash -----> early kdump (dump vmcore) ^ | '.........(reboot).............'
But now, the system crash at early stage is only captured by early kdump, and the rest is captured by normal kdump. That to say, when normal kdump service starts, it will load it again and override early kdump. It is helpful to control the logic of early kdump and normal kdump separately in final action(it is called by kdump-capture.service). For example, early kdump always passes the 'rd.earlykdump' to the second kernel when early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' to the second kernel at any time. So they can be distinguished in the second kernel.
So the new logic is to decide to run final poweroff or reboot logic by checking whether it's running the kexec loaded during early boot or by kdump.service? right? By checking if this 'rd.earlykdump' presents in the 2nd kernel boot cmdline.
But what if, early kdump is enabled and kexec load succeeds during at early boot. However, kdump.service is not started successfully by kdump.service for some reaseon during system startup. Then when the sysetm panic, it will poweroff system after saving a vmcore becuase it believes system panic in early boot, which is unexpected.. I am not sure how pratical it is, but it may happen..
Hmm, I'm confused about the param passing above.
Early or non early just means about the service loading phase, in initramfs or not, I notice dracut/systemd will print some message about they are running in initramfs, so probably you can check how to get it with same way, if this is not initramfs then just unload before the check in kdump loading.
The picture like below:
Kernel boot ->
initramfs --- early kdump load ---- Mark A ---- initramfs switch root
system startup (real root fs) service a service b ... (eg. networking etc.) kdump service start
-----Mark B ----- load kdump kernel again
The problem will happen between Mark A and Mark B, during this period, there could be repeated crash -> earlykdump_load, there might be some random crash as well since during the real root fs service startup, for example after network is ready if some network workload cause a panic, it maybe not 100% reproducible, so it seems we still need to make the poweroff configurable. eg.
default is poweroff, but one can choose if he can.
Signed-off-by: Lianbo Jiang lijiang@redhat.com
dracut-early-kdump.sh | 2 +- dracut-kdump-error-handler.sh | 7 ++++++- dracut-kdump.sh | 6 +++++- kdump-lib-initramfs.sh | 16 +++++++++++++--- kdump.sysconfig | 2 +- kdump.sysconfig.aarch64 | 2 +- kdump.sysconfig.i386 | 2 +- kdump.sysconfig.ppc64 | 2 +- kdump.sysconfig.ppc64le | 2 +- kdump.sysconfig.s390x | 2 +- kdump.sysconfig.x86_64 | 2 +- kdumpctl | 15 +++++++++++++-- 12 files changed, 45 insertions(+), 15 deletions(-)
diff --git a/dracut-early-kdump.sh b/dracut-early-kdump.sh index 34a9909..a799dbf 100755 --- a/dracut-early-kdump.sh +++ b/dracut-early-kdump.sh @@ -58,7 +58,7 @@ early_kdump_load() fi
$KEXEC ${EARLY_KEXEC_ARGS} $standard_kexec_args \
--command-line="$EARLY_KDUMP_CMDLINE" \
if [ $? == 0 ]; then echo "kexec: loaded early-kdump kernel"--command-line="$EARLY_KDUMP_CMDLINE rd.earlykdump" \ --initrd=$EARLY_KDUMP_INITRD $EARLY_KDUMP_KERNEL
diff --git a/dracut-kdump-error-handler.sh b/dracut-kdump-error-handler.sh index 2f0f1d1..4f0e58c 100755 --- a/dracut-kdump-error-handler.sh +++ b/dracut-kdump-error-handler.sh @@ -1,5 +1,6 @@ #!/bin/sh
+. /lib/dracut-lib.sh . /lib/kdump-lib-initramfs.sh
set -o pipefail @@ -7,4 +8,8 @@ export PATH=$PATH:$KDUMP_SCRIPT_DIR
get_kdump_confs do_default_action -do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi diff --git a/dracut-kdump.sh b/dracut-kdump.sh index b75c2a5..119b006 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -201,4 +201,8 @@ if [ $DUMP_RETVAL -ne 0 ]; then exit 1 fi
-do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 7ba99b6..f95cbcf 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -6,7 +6,7 @@ KDUMP_PATH="/var/crash" CORE_COLLECTOR="" DEFAULT_CORE_COLLECTOR="makedumpfile -l --message-level 1 -d 31" DMESG_COLLECTOR="/sbin/vmcore-dmesg" -DEFAULT_ACTION="systemctl reboot -f" +DEFAULT_ACTION="" DATEDIR=`date +%Y-%m-%d-%T` HOST_IP='127.0.0.1' DUMP_INSTRUCTION="" @@ -14,6 +14,7 @@ SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" KDUMP_SCRIPT_DIR="/kdumpscripts" DD_BLKSIZE=512 FINAL_ACTION="systemctl reboot -f" +EARLYKDUMP_FINAL_ACTION="systemctl poweroff" KDUMP_CONF="/etc/kdump.conf" KDUMP_PRE="" KDUMP_POST="" @@ -155,11 +156,20 @@ kdump_emergency_shell()
do_default_action() {
- echo "Kdump: Executing default action $DEFAULT_ACTION"
- eval $DEFAULT_ACTION
- if [ $DEFAULT_ACTION == "" ]; then
echo "Kdump: default action string is null."
$DEFAULT_ACTION should be double quoted. Now it reports an error in Kdump Error Handler: Starting Kdump Error Handler... /lib/kdump-lib-initramfs.sh: line 159: [: ==: unary operator expected
And this "Kdump: default action string is null." message is bit confusing. In kdump.conf, it clearly states "The default option is "reboot"". User may wonder this 'string is null' indicates an error or not. I guess it would be better saying like, "Kdump: no default action is specified. Executing default option" ^^ this would probably be replaced by $FINAL_ACTION or $EARLYKDUMP_FINAL_ACTION which will be executed anyway.
- else
echo "Kdump: Executing default action $DEFAULT_ACTION"
eval $DEFAULT_ACTION
- fi
}
do_final_action() { eval $FINAL_ACTION }
+do_earlykdump_final_action() +{
- eval $EARLYKDUMP_FINAL_ACTION
+} diff --git a/kdump.sysconfig b/kdump.sysconfig index ffe1df8..b011c1c 100644 --- a/kdump.sysconfig +++ b/kdump.sysconfig @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.aarch64 b/kdump.sysconfig.aarch64 index 0a6b14c..b8b8865 100644 --- a/kdump.sysconfig.aarch64 +++ b/kdump.sysconfig.aarch64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.i386 b/kdump.sysconfig.i386 index 18c407e..b9a2835 100644 --- a/kdump.sysconfig.i386 +++ b/kdump.sysconfig.i386 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64 b/kdump.sysconfig.ppc64 index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64 +++ b/kdump.sysconfig.ppc64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64le b/kdump.sysconfig.ppc64le index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64le +++ b/kdump.sysconfig.ppc64le @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.s390x b/kdump.sysconfig.s390x index b3aec3c..4c5fc00 100644 --- a/kdump.sysconfig.s390x +++ b/kdump.sysconfig.s390x @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.x86_64 b/kdump.sysconfig.x86_64 index f269d02..3cccbfc 100644 --- a/kdump.sysconfig.x86_64 +++ b/kdump.sysconfig.x86_64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdumpctl b/kdumpctl index fe6af22..a8fd53d 100755 --- a/kdumpctl +++ b/kdumpctl @@ -26,7 +26,7 @@ image_time=0 standard_kexec_args="-p"
# Some default values in case /etc/sysconfig/kdump doesn't include -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug rd.earlykdump"
if [ -f /etc/sysconfig/kdump ]; then . /etc/sysconfig/kdump @@ -942,6 +942,11 @@ check_default_config() fi }
+check_rd_earlykdump() +{
- egrep "rd.earlykdump" /proc/cmdline
+}
start() { check_dump_feasibility @@ -969,7 +974,13 @@ start() check_current_status if [ $? == 0 ]; then echo "Kdump already running: [WARNING]"
return 0
check_rd_earlykdump
#if earlykdump loaded, it will stop and start.
if [ $? -eq 0 ]; then
stop
kdumpctl start can run not only by system startup services, one can also run it manually or in udev rule.
The checking of kernel cmdline seems not enough.
else
return 0
fi
fi
if check_ssh_config; then
-- 2.17.1
Thanks Dave
在 2018年12月20日 15:31, Xiaowu Wu 写道:
----- Original Message -----
From: "Dave Young" dyoung@redhat.com To: "Lianbo Jiang" lijiang@redhat.com Cc: kexec@lists.fedoraproject.org, bhe@redhat.com, xiawu@redhat.com Sent: Thursday, December 20, 2018 2:06:17 PM Subject: Re: [PATCH] Avoid falling into infinite loop restart when using a problematic system
- Emma
On 12/20/18 at 01:57pm, Dave Young wrote:
On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
By default, early kdump reboots the system after capturing the vmcore. If the problematic system is continuously crashing due to some issue during early boot stage, the system may fall into infinite loop restart like this:
boot -----> crash -----> early kdump (dump vmcore) ^ | '.........(reboot).............'
But now, the system crash at early stage is only captured by early kdump, and the rest is captured by normal kdump. That to say, when normal kdump service starts, it will load it again and override early kdump. It is helpful to control the logic of early kdump and normal kdump separately in final action(it is called by kdump-capture.service). For example, early kdump always passes the 'rd.earlykdump' to the second kernel when early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' to the second kernel at any time. So they can be distinguished in the second kernel.
So the new logic is to decide to run final poweroff or reboot logic by checking whether it's running the kexec loaded during early boot or by kdump.service? right? By checking if this 'rd.earlykdump' presents in the 2nd kernel boot cmdline.
But what if, early kdump is enabled and kexec load succeeds during at early boot. However, kdump.service is not started successfully by kdump.service for some reaseon during system startup. Then when the sysetm panic, it will poweroff system after saving a vmcore becuase it believes system panic in early boot, which is unexpected.. I am not sure how pratical it is, but it may happen..
Thanks for your comment.
This is a good case. Let's consider two scenarios: 1. If kdump.service is disabled, it makes no sense distinguish early kdump and normal kdump. Because it has only one kdump. 2. If kdump.service is enabled, but it fails to start, that should be a bug. We should try to fix it.
Actually, the purpose of distinguishing early kdump and normal kdump is to perform the different action in kdump capture service. Anyway, early kdump and normal kdump can get the correct vmcore, what's the difference?
Hmm, I'm confused about the param passing above.
Early or non early just means about the service loading phase, in initramfs or not, I notice dracut/systemd will print some message about they are running in initramfs, so probably you can check how to get it with same way, if this is not initramfs then just unload before the check in kdump loading.
The picture like below:
Kernel boot ->
initramfs --- early kdump load ---- Mark A ---- initramfs switch root
system startup (real root fs) service a service b ... (eg. networking etc.) kdump service start
-----Mark B ----- load kdump kernel again
The problem will happen between Mark A and Mark B, during this period, there could be repeated crash -> earlykdump_load, there might be some random crash as well since during the real root fs service startup, for example after network is ready if some network workload cause a panic, it maybe not 100% reproducible, so it seems we still need to make the poweroff configurable. eg.
default is poweroff, but one can choose if he can.
Signed-off-by: Lianbo Jiang lijiang@redhat.com
dracut-early-kdump.sh | 2 +- dracut-kdump-error-handler.sh | 7 ++++++- dracut-kdump.sh | 6 +++++- kdump-lib-initramfs.sh | 16 +++++++++++++--- kdump.sysconfig | 2 +- kdump.sysconfig.aarch64 | 2 +- kdump.sysconfig.i386 | 2 +- kdump.sysconfig.ppc64 | 2 +- kdump.sysconfig.ppc64le | 2 +- kdump.sysconfig.s390x | 2 +- kdump.sysconfig.x86_64 | 2 +- kdumpctl | 15 +++++++++++++-- 12 files changed, 45 insertions(+), 15 deletions(-)
diff --git a/dracut-early-kdump.sh b/dracut-early-kdump.sh index 34a9909..a799dbf 100755 --- a/dracut-early-kdump.sh +++ b/dracut-early-kdump.sh @@ -58,7 +58,7 @@ early_kdump_load() fi
$KEXEC ${EARLY_KEXEC_ARGS} $standard_kexec_args \
--command-line="$EARLY_KDUMP_CMDLINE" \
if [ $? == 0 ]; then echo "kexec: loaded early-kdump kernel"--command-line="$EARLY_KDUMP_CMDLINE rd.earlykdump" \ --initrd=$EARLY_KDUMP_INITRD $EARLY_KDUMP_KERNEL
diff --git a/dracut-kdump-error-handler.sh b/dracut-kdump-error-handler.sh index 2f0f1d1..4f0e58c 100755 --- a/dracut-kdump-error-handler.sh +++ b/dracut-kdump-error-handler.sh @@ -1,5 +1,6 @@ #!/bin/sh
+. /lib/dracut-lib.sh . /lib/kdump-lib-initramfs.sh
set -o pipefail @@ -7,4 +8,8 @@ export PATH=$PATH:$KDUMP_SCRIPT_DIR
get_kdump_confs do_default_action -do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi diff --git a/dracut-kdump.sh b/dracut-kdump.sh index b75c2a5..119b006 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -201,4 +201,8 @@ if [ $DUMP_RETVAL -ne 0 ]; then exit 1 fi
-do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 7ba99b6..f95cbcf 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -6,7 +6,7 @@ KDUMP_PATH="/var/crash" CORE_COLLECTOR="" DEFAULT_CORE_COLLECTOR="makedumpfile -l --message-level 1 -d 31" DMESG_COLLECTOR="/sbin/vmcore-dmesg" -DEFAULT_ACTION="systemctl reboot -f" +DEFAULT_ACTION="" DATEDIR=`date +%Y-%m-%d-%T` HOST_IP='127.0.0.1' DUMP_INSTRUCTION="" @@ -14,6 +14,7 @@ SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" KDUMP_SCRIPT_DIR="/kdumpscripts" DD_BLKSIZE=512 FINAL_ACTION="systemctl reboot -f" +EARLYKDUMP_FINAL_ACTION="systemctl poweroff" KDUMP_CONF="/etc/kdump.conf" KDUMP_PRE="" KDUMP_POST="" @@ -155,11 +156,20 @@ kdump_emergency_shell()
do_default_action() {
- echo "Kdump: Executing default action $DEFAULT_ACTION"
- eval $DEFAULT_ACTION
- if [ $DEFAULT_ACTION == "" ]; then
echo "Kdump: default action string is null."
$DEFAULT_ACTION should be double quoted. Now it reports an error in Kdump Error Handler: Starting Kdump Error Handler... /lib/kdump-lib-initramfs.sh: line 159: [: ==: unary operator expected
And this "Kdump: default action string is null." message is bit confusing. In kdump.conf, it clearly states "The default option is "reboot"". User may wonder this 'string is null' indicates an error or not. I guess it would be better saying like, "Kdump: no default action is specified. Executing default option" ^^ this would probably be replaced by $FINAL_ACTION or $EARLYKDUMP_FINAL_ACTION which will be executed anyway.
Great. They will be improved in v2.
Thanks.
- else
echo "Kdump: Executing default action $DEFAULT_ACTION"
eval $DEFAULT_ACTION
- fi
}
do_final_action() { eval $FINAL_ACTION }
+do_earlykdump_final_action() +{
- eval $EARLYKDUMP_FINAL_ACTION
+} diff --git a/kdump.sysconfig b/kdump.sysconfig index ffe1df8..b011c1c 100644 --- a/kdump.sysconfig +++ b/kdump.sysconfig @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.aarch64 b/kdump.sysconfig.aarch64 index 0a6b14c..b8b8865 100644 --- a/kdump.sysconfig.aarch64 +++ b/kdump.sysconfig.aarch64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.i386 b/kdump.sysconfig.i386 index 18c407e..b9a2835 100644 --- a/kdump.sysconfig.i386 +++ b/kdump.sysconfig.i386 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64 b/kdump.sysconfig.ppc64 index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64 +++ b/kdump.sysconfig.ppc64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64le b/kdump.sysconfig.ppc64le index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64le +++ b/kdump.sysconfig.ppc64le @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.s390x b/kdump.sysconfig.s390x index b3aec3c..4c5fc00 100644 --- a/kdump.sysconfig.s390x +++ b/kdump.sysconfig.s390x @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.x86_64 b/kdump.sysconfig.x86_64 index f269d02..3cccbfc 100644 --- a/kdump.sysconfig.x86_64 +++ b/kdump.sysconfig.x86_64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdumpctl b/kdumpctl index fe6af22..a8fd53d 100755 --- a/kdumpctl +++ b/kdumpctl @@ -26,7 +26,7 @@ image_time=0 standard_kexec_args="-p"
# Some default values in case /etc/sysconfig/kdump doesn't include -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug rd.earlykdump"
if [ -f /etc/sysconfig/kdump ]; then . /etc/sysconfig/kdump @@ -942,6 +942,11 @@ check_default_config() fi }
+check_rd_earlykdump() +{
- egrep "rd.earlykdump" /proc/cmdline
+}
start() { check_dump_feasibility @@ -969,7 +974,13 @@ start() check_current_status if [ $? == 0 ]; then echo "Kdump already running: [WARNING]"
return 0
check_rd_earlykdump
#if earlykdump loaded, it will stop and start.
if [ $? -eq 0 ]; then
stop
kdumpctl start can run not only by system startup services, one can also run it manually or in udev rule.
The checking of kernel cmdline seems not enough.
else
return 0
fi
fi
if check_ssh_config; then
-- 2.17.1
Thanks Dave
在 2018年12月20日 13:57, Dave Young 写道:
On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
By default, early kdump reboots the system after capturing the vmcore. If the problematic system is continuously crashing due to some issue during early boot stage, the system may fall into infinite loop restart like this:
boot -----> crash -----> early kdump (dump vmcore) ^ | '.........(reboot).............'
But now, the system crash at early stage is only captured by early kdump, and the rest is captured by normal kdump. That to say, when normal kdump service starts, it will load it again and override early kdump. It is helpful to control the logic of early kdump and normal kdump separately in final action(it is called by kdump-capture.service). For example, early kdump always passes the 'rd.earlykdump' to the second kernel when early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' to the second kernel at any time. So they can be distinguished in the second kernel.
Hmm, I'm confused about the param passing above.
I copy some messages from another email, please refer to this one: [---> The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump and normal kdump can get the same parameters from /proc/cmdline in the first kernel.
Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't need it, normal kdump needs to remove the rd.earlykdump.
So which can distinguish early kdump and normal kdump in the second kernel. It helps to control the logic of kdump capture service. For example: default action/final action. ]
Early or non early just means about the service loading phase, in
Yes. This patch used the same method what you said. When normal kdump service starts, it will reload. At the same time, early kdump will be overwritten by normal kdump.
initramfs or not, I notice dracut/systemd will print some message about they are running in initramfs, so probably you can check how to get it with same way, if this is not initramfs then just unload before the check in kdump loading.
The picture like below:
Kernel boot ->
initramfs --- early kdump load ---- Mark A ---- initramfs switch root
system startup (real root fs) service a service b ... (eg. networking etc.) kdump service start
-----Mark B ----- load kdump kernel again
The problem will happen between Mark A and Mark B, during this period, there could be repeated crash -> earlykdump_load, there might be some random crash as well since during the real root fs service startup, for example after network is ready if some network workload cause a panic, it maybe not 100% reproducible, so it seems we still need to make the poweroff configurable. eg.
default is poweroff, but one can choose if he can.
Yes, default is poweroff for early kdump. Unless kdump capture service happens error or enters the emergency service, one can choose the default action.(configure default=xxx in kdump.conf)
Signed-off-by: Lianbo Jiang lijiang@redhat.com
dracut-early-kdump.sh | 2 +- dracut-kdump-error-handler.sh | 7 ++++++- dracut-kdump.sh | 6 +++++- kdump-lib-initramfs.sh | 16 +++++++++++++--- kdump.sysconfig | 2 +- kdump.sysconfig.aarch64 | 2 +- kdump.sysconfig.i386 | 2 +- kdump.sysconfig.ppc64 | 2 +- kdump.sysconfig.ppc64le | 2 +- kdump.sysconfig.s390x | 2 +- kdump.sysconfig.x86_64 | 2 +- kdumpctl | 15 +++++++++++++-- 12 files changed, 45 insertions(+), 15 deletions(-)
diff --git a/dracut-early-kdump.sh b/dracut-early-kdump.sh index 34a9909..a799dbf 100755 --- a/dracut-early-kdump.sh +++ b/dracut-early-kdump.sh @@ -58,7 +58,7 @@ early_kdump_load() fi
$KEXEC ${EARLY_KEXEC_ARGS} $standard_kexec_args \
--command-line="$EARLY_KDUMP_CMDLINE" \
if [ $? == 0 ]; then echo "kexec: loaded early-kdump kernel"--command-line="$EARLY_KDUMP_CMDLINE rd.earlykdump" \ --initrd=$EARLY_KDUMP_INITRD $EARLY_KDUMP_KERNEL
diff --git a/dracut-kdump-error-handler.sh b/dracut-kdump-error-handler.sh index 2f0f1d1..4f0e58c 100755 --- a/dracut-kdump-error-handler.sh +++ b/dracut-kdump-error-handler.sh @@ -1,5 +1,6 @@ #!/bin/sh
+. /lib/dracut-lib.sh . /lib/kdump-lib-initramfs.sh
set -o pipefail @@ -7,4 +8,8 @@ export PATH=$PATH:$KDUMP_SCRIPT_DIR
get_kdump_confs do_default_action -do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi diff --git a/dracut-kdump.sh b/dracut-kdump.sh index b75c2a5..119b006 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -201,4 +201,8 @@ if [ $DUMP_RETVAL -ne 0 ]; then exit 1 fi
-do_final_action +if getargbool 0 rd.earlykdump; then
- do_earlykdump_final_action
+else
- do_final_action
+fi diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 7ba99b6..f95cbcf 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -6,7 +6,7 @@ KDUMP_PATH="/var/crash" CORE_COLLECTOR="" DEFAULT_CORE_COLLECTOR="makedumpfile -l --message-level 1 -d 31" DMESG_COLLECTOR="/sbin/vmcore-dmesg" -DEFAULT_ACTION="systemctl reboot -f" +DEFAULT_ACTION="" DATEDIR=`date +%Y-%m-%d-%T` HOST_IP='127.0.0.1' DUMP_INSTRUCTION="" @@ -14,6 +14,7 @@ SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" KDUMP_SCRIPT_DIR="/kdumpscripts" DD_BLKSIZE=512 FINAL_ACTION="systemctl reboot -f" +EARLYKDUMP_FINAL_ACTION="systemctl poweroff" KDUMP_CONF="/etc/kdump.conf" KDUMP_PRE="" KDUMP_POST="" @@ -155,11 +156,20 @@ kdump_emergency_shell()
do_default_action() {
- echo "Kdump: Executing default action $DEFAULT_ACTION"
- eval $DEFAULT_ACTION
- if [ $DEFAULT_ACTION == "" ]; then
echo "Kdump: default action string is null."
- else
echo "Kdump: Executing default action $DEFAULT_ACTION"
eval $DEFAULT_ACTION
- fi
}
do_final_action() { eval $FINAL_ACTION }
+do_earlykdump_final_action() +{
- eval $EARLYKDUMP_FINAL_ACTION
+} diff --git a/kdump.sysconfig b/kdump.sysconfig index ffe1df8..b011c1c 100644 --- a/kdump.sysconfig +++ b/kdump.sysconfig @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.aarch64 b/kdump.sysconfig.aarch64 index 0a6b14c..b8b8865 100644 --- a/kdump.sysconfig.aarch64 +++ b/kdump.sysconfig.aarch64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.i386 b/kdump.sysconfig.i386 index 18c407e..b9a2835 100644 --- a/kdump.sysconfig.i386 +++ b/kdump.sysconfig.i386 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64 b/kdump.sysconfig.ppc64 index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64 +++ b/kdump.sysconfig.ppc64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.ppc64le b/kdump.sysconfig.ppc64le index 55a01cc..e1ea9c6 100644 --- a/kdump.sysconfig.ppc64le +++ b/kdump.sysconfig.ppc64le @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.s390x b/kdump.sysconfig.s390x index b3aec3c..4c5fc00 100644 --- a/kdump.sysconfig.s390x +++ b/kdump.sysconfig.s390x @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdump.sysconfig.x86_64 b/kdump.sysconfig.x86_64 index f269d02..3cccbfc 100644 --- a/kdump.sysconfig.x86_64 +++ b/kdump.sysconfig.x86_64 @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE="" # This variable lets us remove arguments from the current kdump commandline # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline # NOTE: some arguments such as crashkernel will always be removed -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet rd.earlykdump"
# This variable lets us append arguments to the current kdump commandline # after processed by KDUMP_COMMANDLINE_REMOVE diff --git a/kdumpctl b/kdumpctl index fe6af22..a8fd53d 100755 --- a/kdumpctl +++ b/kdumpctl @@ -26,7 +26,7 @@ image_time=0 standard_kexec_args="-p"
# Some default values in case /etc/sysconfig/kdump doesn't include -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug" +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug rd.earlykdump"
if [ -f /etc/sysconfig/kdump ]; then . /etc/sysconfig/kdump @@ -942,6 +942,11 @@ check_default_config() fi }
+check_rd_earlykdump() +{
- egrep "rd.earlykdump" /proc/cmdline
+}
start() { check_dump_feasibility @@ -969,7 +974,13 @@ start() check_current_status if [ $? == 0 ]; then echo "Kdump already running: [WARNING]"
return 0
check_rd_earlykdump
#if earlykdump loaded, it will stop and start.
if [ $? -eq 0 ]; then
stop
kdumpctl start can run not only by system startup services, one can also run it manually or in udev rule.
The checking of kernel cmdline seems not enough.
Here it means that if kdump has beend loaded, check whether early kdump did it. If yes, let normal kdump load it again, otherwise no need to do anything.
Thanks.
else
return 0
fi
fi
if check_ssh_config; then
-- 2.17.1
Thanks Dave
+ more people On 12/20/18 at 04:49pm, lijiang wrote:
在 2018年12月20日 13:57, Dave Young 写道:
On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
By default, early kdump reboots the system after capturing the vmcore. If the problematic system is continuously crashing due to some issue during early boot stage, the system may fall into infinite loop restart like this:
boot -----> crash -----> early kdump (dump vmcore) ^ | '.........(reboot).............'
But now, the system crash at early stage is only captured by early kdump, and the rest is captured by normal kdump. That to say, when normal kdump service starts, it will load it again and override early kdump. It is helpful to control the logic of early kdump and normal kdump separately in final action(it is called by kdump-capture.service). For example, early kdump always passes the 'rd.earlykdump' to the second kernel when early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' to the second kernel at any time. So they can be distinguished in the second kernel.
Hmm, I'm confused about the param passing above.
I copy some messages from another email, please refer to this one: [---> The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump and normal kdump can get the same parameters from /proc/cmdline in the first kernel.
Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't need it, normal kdump needs to remove the rd.earlykdump.
So which can distinguish early kdump and normal kdump in the second kernel. It helps to control the logic of kdump capture service. For example: default action/final action. ]
The description is confusing, "ealy kdump passes ... to the second kernel", for example about this, the real thing is one person adds the param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd kernel.
Anyway this is patch log issue.
Early or non early just means about the service loading phase, in
Yes. This patch used the same method what you said. When normal kdump service starts, it will reload. At the same time, early kdump will be overwritten by normal kdump.
Probably "early kdump load" is better than "early kdump" in words.
initramfs or not, I notice dracut/systemd will print some message about they are running in initramfs, so probably you can check how to get it with same way, if this is not initramfs then just unload before the check in kdump loading.
The picture like below:
Kernel boot ->
initramfs --- early kdump load ---- Mark A ---- initramfs switch root
system startup (real root fs) service a service b ... (eg. networking etc.) kdump service start
-----Mark B ----- load kdump kernel again
The problem will happen between Mark A and Mark B, during this period, there could be repeated crash -> earlykdump_load, there might be some random crash as well since during the real root fs service startup, for example after network is ready if some network workload cause a panic, it maybe not 100% reproducible, so it seems we still need to make the poweroff configurable. eg.
default is poweroff, but one can choose if he can.
Yes, default is poweroff for early kdump. Unless kdump capture service happens error or enters the emergency service, one can choose the default action.(configure default=xxx in kdump.conf)
For default action instead of final action if you hardcode it, then even if one set default as reboot it still poweroff.
[snip]
+check_rd_earlykdump() +{
- egrep "rd.earlykdump" /proc/cmdline
+}
start() { check_dump_feasibility @@ -969,7 +974,13 @@ start() check_current_status if [ $? == 0 ]; then echo "Kdump already running: [WARNING]"
return 0
check_rd_earlykdump
#if earlykdump loaded, it will stop and start.
if [ $? -eq 0 ]; then
stop
kdumpctl start can run not only by system startup services, one can also run it manually or in udev rule.
The checking of kernel cmdline seems not enough.
Here it means that if kdump has beend loaded, check whether early kdump did it. If yes, let normal kdump load it again, otherwise no need to do anything.
As we discussed you do not get my points here :)
check_rd_earlykdump will be always true once kernel bootup, so there is no way to get the first time of normal kdump load and the later one.
The early boot time panic to address for this patch is the 100% reproducible panic, for this kinds of panic admin should be able to see it when he boot the machine. So rethinking about this the best way may be just a wontfix.
Let's consider below use cases:
* First install:
install os -> ---A reboot -> ... ---B kdump service start -> create kdump initrd ... boot finished recreate default initrd and enable early kdump goto A
Panic happened between A and B, if it is predictable, eg. 100% reproducible, then admin should already see it, then he/she can control and stop the repeating crash/kdump loop if the panic is not 100% reproducible then use reboot as final action is just fine
* Other use cases eg. updating kernel or some other components: It is similar with the intall os use case because if one update kernel or critical components it is likely they need regenerate kdump initrd, and then repack it into early kdump default initrd, in this case admin should be able to see the panic loop and handle it. Also if the panic is not 100% reproducible then we are just fine.
If we choose to split early and late kdump load, there could be other side effects, and make the logic even complicated.
So... the better way may be just leave it as is, and maybe add some documentation.
Thoughts?
Thanks Dave
在 2018年12月21日 10:49, Dave Young 写道:
- more people
On 12/20/18 at 04:49pm, lijiang wrote:
在 2018年12月20日 13:57, Dave Young 写道:
On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
By default, early kdump reboots the system after capturing the vmcore. If the problematic system is continuously crashing due to some issue during early boot stage, the system may fall into infinite loop restart like this:
boot -----> crash -----> early kdump (dump vmcore) ^ | '.........(reboot).............'
But now, the system crash at early stage is only captured by early kdump, and the rest is captured by normal kdump. That to say, when normal kdump service starts, it will load it again and override early kdump. It is helpful to control the logic of early kdump and normal kdump separately in final action(it is called by kdump-capture.service). For example, early kdump always passes the 'rd.earlykdump' to the second kernel when early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' to the second kernel at any time. So they can be distinguished in the second kernel.
Hmm, I'm confused about the param passing above.
I copy some messages from another email, please refer to this one: [---> The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump and normal kdump can get the same parameters from /proc/cmdline in the first kernel.
Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't need it, normal kdump needs to remove the rd.earlykdump.
So which can distinguish early kdump and normal kdump in the second kernel. It helps to control the logic of kdump capture service. For example: default action/final action. ]
The description is confusing, "ealy kdump passes ... to the second kernel", for example about this, the real thing is one person adds the param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd kernel.
Yes. Good point. Thanks for your explanation.
Anyway this is patch log issue.
Early or non early just means about the service loading phase, in
Yes. This patch used the same method what you said. When normal kdump service starts, it will reload. At the same time, early kdump will be overwritten by normal kdump.
Probably "early kdump load" is better than "early kdump" in words.
initramfs or not, I notice dracut/systemd will print some message about they are running in initramfs, so probably you can check how to get it with same way, if this is not initramfs then just unload before the check in kdump loading.
The picture like below:
Kernel boot ->
initramfs --- early kdump load ---- Mark A ---- initramfs switch root
system startup (real root fs) service a service b ... (eg. networking etc.) kdump service start
-----Mark B ----- load kdump kernel again
The problem will happen between Mark A and Mark B, during this period, there could be repeated crash -> earlykdump_load, there might be some random crash as well since during the real root fs service startup, for example after network is ready if some network workload cause a panic, it maybe not 100% reproducible, so it seems we still need to make the poweroff configurable. eg.
default is poweroff, but one can choose if he can.
Yes, default is poweroff for early kdump. Unless kdump capture service happens error or enters the emergency service, one can choose the default action.(configure default=xxx in kdump.conf)
For default action instead of final action if you hardcode it, then even if one set default as reboot it still poweroff.
If really need, that can be improved.
[snip]
+check_rd_earlykdump() +{
- egrep "rd.earlykdump" /proc/cmdline
+}
start() { check_dump_feasibility @@ -969,7 +974,13 @@ start() check_current_status if [ $? == 0 ]; then echo "Kdump already running: [WARNING]"
return 0
check_rd_earlykdump
#if earlykdump loaded, it will stop and start.
if [ $? -eq 0 ]; then
stop
kdumpctl start can run not only by system startup services, one can also run it manually or in udev rule.
The checking of kernel cmdline seems not enough.
Here it means that if kdump has beend loaded, check whether early kdump did it. If yes, let normal kdump load it again, otherwise no need to do anything.
As we discussed you do not get my points here :)
check_rd_earlykdump will be always true once kernel bootup, so there is no way to get the first time of normal kdump load and the later one.
The early boot time panic to address for this patch is the 100% reproducible panic, for this kinds of panic admin should be able to see it when he boot the machine. So rethinking about this the best way may be just a wontfix.
Let's consider below use cases:
- First install:
install os -> ---A reboot -> ... ---B kdump service start -> create kdump initrd ... boot finished recreate default initrd and enable early kdump goto A
Panic happened between A and B, if it is predictable, eg. 100% reproducible, then admin should already see it, then he/she can control and stop the repeating crash/kdump loop if the panic is not 100% reproducible then use reboot as final action is just fine
- Other use cases eg. updating kernel or some other components:
It is similar with the intall os use case because if one update kernel or critical components it is likely they need regenerate kdump initrd, and then repack it into early kdump default initrd, in this case admin should be able to see the panic loop and handle it. Also if the panic is not 100% reproducible then we are just fine.
If we choose to split early and late kdump load, there could be other side effects, and make the logic even complicated.
So... the better way may be just leave it as is, and maybe add some documentation.
It is a good way to document the risks that may exist.
Just like the public transportation, we all know that it has the risk, but we still choose it.
Thanks.
Thoughts?
Thanks Dave
Hi, Dave, Lianbo
My concern is that crash loop may generate tons of dump cores, and the dump target may get filled up by dump cores, that may have larger potential risk. Else I think it's good to leave it as it is.
On Fri, Dec 21, 2018 at 2:05 PM lijiang lijiang@redhat.com wrote:
在 2018年12月21日 10:49, Dave Young 写道:
- more people
On 12/20/18 at 04:49pm, lijiang wrote:
在 2018年12月20日 13:57, Dave Young 写道:
On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
By default, early kdump reboots the system after capturing the vmcore. If the problematic system is continuously crashing due to some issue during early boot stage, the system may fall into infinite loop restart like this:
boot -----> crash -----> early kdump (dump vmcore) ^ | '.........(reboot).............'
But now, the system crash at early stage is only captured by early kdump, and the rest is captured by normal kdump. That to say, when normal kdump service starts, it will load it again and override early kdump. It is helpful to control the logic of early kdump and normal kdump separately in final action(it is called by kdump-capture.service). For example, early kdump always passes the 'rd.earlykdump' to the second kernel when early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' to the second kernel at any time. So they can be distinguished in the second kernel.
Hmm, I'm confused about the param passing above.
I copy some messages from another email, please refer to this one: [---> The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump and normal kdump can get the same parameters from /proc/cmdline in the first kernel.
Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't need it, normal kdump needs to remove the rd.earlykdump.
So which can distinguish early kdump and normal kdump in the second kernel. It helps to control the logic of kdump capture service. For example: default action/final action. ]
The description is confusing, "ealy kdump passes ... to the second kernel", for example about this, the real thing is one person adds the param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd kernel.
Yes. Good point. Thanks for your explanation.
Anyway this is patch log issue.
Early or non early just means about the service loading phase, in
Yes. This patch used the same method what you said. When normal kdump service starts, it will reload. At the same time, early kdump will be overwritten by normal kdump.
Probably "early kdump load" is better than "early kdump" in words.
initramfs or not, I notice dracut/systemd will print some message about they are running in initramfs, so probably you can check how to get it with same way, if this is not initramfs then just unload before the check in kdump loading.
The picture like below:
Kernel boot ->
initramfs --- early kdump load ---- Mark A ---- initramfs switch root
system startup (real root fs) service a service b ... (eg. networking etc.) kdump service start
-----Mark B ----- load kdump kernel again
The problem will happen between Mark A and Mark B, during this period, there could be repeated crash -> earlykdump_load, there might be some random crash as well since during the real root fs service startup, for example after network is ready if some network workload cause a panic, it maybe not 100% reproducible, so it seems we still need to make the poweroff configurable. eg.
default is poweroff, but one can choose if he can.
Yes, default is poweroff for early kdump. Unless kdump capture service happens error or enters the emergency service, one can choose the default action.(configure default=xxx in kdump.conf)
For default action instead of final action if you hardcode it, then even if one set default as reboot it still poweroff.
If really need, that can be improved.
[snip]
+check_rd_earlykdump() +{
- egrep "rd.earlykdump" /proc/cmdline
+}
start() { check_dump_feasibility @@ -969,7 +974,13 @@ start() check_current_status if [ $? == 0 ]; then echo "Kdump already running: [WARNING]"
return 0
check_rd_earlykdump
#if earlykdump loaded, it will stop and start.
if [ $? -eq 0 ]; then
stop
kdumpctl start can run not only by system startup services, one can also run it manually or in udev rule.
The checking of kernel cmdline seems not enough.
Here it means that if kdump has beend loaded, check whether early kdump did it. If yes, let normal kdump load it again, otherwise no need to do anything.
As we discussed you do not get my points here :)
check_rd_earlykdump will be always true once kernel bootup, so there is no way to get the first time of normal kdump load and the later one.
The early boot time panic to address for this patch is the 100% reproducible panic, for this kinds of panic admin should be able to see it when he boot the machine. So rethinking about this the best way may be just a wontfix.
Let's consider below use cases:
- First install:
install os -> ---A reboot -> ... ---B kdump service start -> create kdump initrd ... boot finished recreate default initrd and enable early kdump goto A
Panic happened between A and B, if it is predictable, eg. 100% reproducible, then admin should already see it, then he/she can control and stop the repeating crash/kdump loop if the panic is not 100% reproducible then use reboot as final action is just fine
- Other use cases eg. updating kernel or some other components:
It is similar with the intall os use case because if one update kernel or critical components it is likely they need regenerate kdump initrd, and then repack it into early kdump default initrd, in this case admin should be able to see the panic loop and handle it. Also if the panic is not 100% reproducible then we are just fine.
If we choose to split early and late kdump load, there could be other side effects, and make the logic even complicated.
So... the better way may be just leave it as is, and maybe add some documentation.
It is a good way to document the risks that may exist.
Just like the public transportation, we all know that it has the risk, but we still choose it.
Thanks.
Thoughts?
Thanks Dave
On 12/21/18 12:29 PM, Kairui Song wrote:
Hi, Dave, Lianbo
My concern is that crash loop may generate tons of dump cores, and the dump target may get filled up by dump cores, that may have larger potential risk. Else I think it's good to leave it as it is.
On Fri, Dec 21, 2018 at 2:05 PM lijiang lijiang@redhat.com wrote:
在 2018年12月21日 10:49, Dave Young 写道:
- more people
On 12/20/18 at 04:49pm, lijiang wrote:
在 2018年12月20日 13:57, Dave Young 写道:
On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
By default, early kdump reboots the system after capturing the vmcore. If the problematic system is continuously crashing due to some issue during early boot stage, the system may fall into infinite loop restart like this:
boot -----> crash -----> early kdump (dump vmcore) ^ | '.........(reboot).............'
But now, the system crash at early stage is only captured by early kdump, and the rest is captured by normal kdump. That to say, when normal kdump service starts, it will load it again and override early kdump. It is helpful to control the logic of early kdump and normal kdump separately in final action(it is called by kdump-capture.service). For example, early kdump always passes the 'rd.earlykdump' to the second kernel when early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' to the second kernel at any time. So they can be distinguished in the second kernel.
Hmm, I'm confused about the param passing above.
I copy some messages from another email, please refer to this one: [---> The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump and normal kdump can get the same parameters from /proc/cmdline in the first kernel.
Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't need it, normal kdump needs to remove the rd.earlykdump.
So which can distinguish early kdump and normal kdump in the second kernel. It helps to control the logic of kdump capture service. For example: default action/final action. ]
The description is confusing, "ealy kdump passes ... to the second kernel", for example about this, the real thing is one person adds the param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd kernel.
Yes. Good point. Thanks for your explanation.
Anyway this is patch log issue.
Early or non early just means about the service loading phase, in
Yes. This patch used the same method what you said. When normal kdump service starts, it will reload. At the same time, early kdump will be overwritten by normal kdump.
Probably "early kdump load" is better than "early kdump" in words.
initramfs or not, I notice dracut/systemd will print some message about they are running in initramfs, so probably you can check how to get it with same way, if this is not initramfs then just unload before the check in kdump loading.
The picture like below:
Kernel boot ->
initramfs --- early kdump load
---- Mark A ---- initramfs switch root
system startup (real root fs) service a service b ... (eg. networking etc.) kdump service start
-----Mark B ----- load kdump kernel again
The problem will happen between Mark A and Mark B, during this period, there could be repeated crash -> earlykdump_load, there might be some random crash as well since during the real root fs service startup, for example after network is ready if some network workload cause a panic, it maybe not 100% reproducible, so it seems we still need to make the poweroff configurable. eg.
default is poweroff, but one can choose if he can.
Yes, default is poweroff for early kdump. Unless kdump capture service happens error or enters the emergency service, one can choose the default action.(configure default=xxx in kdump.conf)
For default action instead of final action if you hardcode it, then even if one set default as reboot it still poweroff.
If really need, that can be improved.
[snip]
+check_rd_earlykdump() +{
- egrep "rd.earlykdump" /proc/cmdline
+}
- start() { check_dump_feasibility
@@ -969,7 +974,13 @@ start() check_current_status if [ $? == 0 ]; then echo "Kdump already running: [WARNING]"
return 0
check_rd_earlykdump
#if earlykdump loaded, it will stop and start.
if [ $? -eq 0 ]; then
stop
kdumpctl start can run not only by system startup services, one can also run it manually or in udev rule.
The checking of kernel cmdline seems not enough.
Here it means that if kdump has beend loaded, check whether early kdump did it. If yes, let normal kdump load it again, otherwise no need to do anything.
As we discussed you do not get my points here :)
check_rd_earlykdump will be always true once kernel bootup, so there is no way to get the first time of normal kdump load and the later one.
The early boot time panic to address for this patch is the 100% reproducible panic, for this kinds of panic admin should be able to see it when he boot the machine. So rethinking about this the best way may be just a wontfix.
Let's consider below use cases:
- First install:
install os -> ---A reboot -> ... ---B kdump service start -> create kdump initrd ... boot finished recreate default initrd and enable early kdump goto A
Panic happened between A and B, if it is predictable, eg. 100% reproducible, then admin should already see it, then he/she can control and stop the repeating crash/kdump loop if the panic is not 100% reproducible then use reboot as final action is just fine
- Other use cases eg. updating kernel or some other components:
It is similar with the intall os use case because if one update kernel or critical components it is likely they need regenerate kdump initrd, and then repack it into early kdump default initrd, in this case admin should be able to see the panic loop and handle it. Also if the panic is not 100% reproducible then we are just fine.
If we choose to split early and late kdump load, there could be other side effects, and make the logic even complicated.
So... the better way may be just leave it as is, and maybe add some documentation.
It is a good way to document the risks that may exist.
Just like the public transportation, we all know that it has the risk, but we still choose it.
Thanks.
Thoughts?
Thanks Dave
Hello Dave et al.
Kindly check the below condition, assumption and share your thoughts.
Condition:
[0x1] System is up and running with the kernel version X. [0x2] Admin performed the kernel Y upgrade. [0x3] Running kernel X crashed. [0x4] Normal kdump rebooted the system and captured the kernel crash dump of the kernel X. [0x5] System rebooted with the newly installed kernel Y. [0x6] Let's assume that due to some unknown reason the booting kernel Y also crashed (assume that the panic is 100% reproducible). [0x7] Early kdump started dumping the kernel crash dump of the booting kernel Y. [0x8] Early kdump rebooted the system and stuck in the loop.
Assumption: 1
What if the problematic system is in data center and admin is not aware of this situation?
[0x1] The dump target will be filled with the multiple copies of the kernel crash dump?
[ 1 kernel crash dump of the kernel X and 'n' kernel crash dump of the kernel Y]
[0x2] The system will reboot in the loop?
Assumption: 2
What if the dump target in on the local disk?
[0x1] Admin needs to power off the system manually to retrieve the kernel crash dump of the kernel X and Y from the resume mode. [0x2] Admin needs to remove the multiple copies of the kernel crash dump of the booting kernel Y. [0x3] Admin might get confused while differentiating between the kernel crash dump of the kernel X and Kernel Y.
Regards, Buland
On 12/21/18 at 12:59pm, Buland Singh wrote:
On 12/21/18 12:29 PM, Kairui Song wrote:
Hi, Dave, Lianbo
My concern is that crash loop may generate tons of dump cores, and the dump target may get filled up by dump cores, that may have larger potential risk. Else I think it's good to leave it as it is.
On Fri, Dec 21, 2018 at 2:05 PM lijiang lijiang@redhat.com wrote:
在 2018年12月21日 10:49, Dave Young 写道:
- more people
On 12/20/18 at 04:49pm, lijiang wrote:
在 2018年12月20日 13:57, Dave Young 写道:
On 12/20/18 at 01:06pm, Lianbo Jiang wrote: > By default, early kdump reboots the system after capturing the vmcore. > If the problematic system is continuously crashing due to some issue > during early boot stage, the system may fall into infinite loop restart > like this: > > boot -----> crash -----> early kdump (dump vmcore) > ^ | > '.........(reboot).............' > > But now, the system crash at early stage is only captured by early kdump, > and the rest is captured by normal kdump. That to say, when normal kdump > service starts, it will load it again and override early kdump. It is > helpful to control the logic of early kdump and normal kdump separately > in final action(it is called by kdump-capture.service). For example, > early kdump always passes the 'rd.earlykdump' to the second kernel when > early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' > to the second kernel at any time. So they can be distinguished in the > second kernel.
Hmm, I'm confused about the param passing above.
I copy some messages from another email, please refer to this one: [---> The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump and normal kdump can get the same parameters from /proc/cmdline in the first kernel.
Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't need it, normal kdump needs to remove the rd.earlykdump.
So which can distinguish early kdump and normal kdump in the second kernel. It helps to control the logic of kdump capture service. For example: default action/final action. ]
The description is confusing, "ealy kdump passes ... to the second kernel", for example about this, the real thing is one person adds the param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd kernel.
Yes. Good point. Thanks for your explanation.
Anyway this is patch log issue.
Early or non early just means about the service loading phase, in
Yes. This patch used the same method what you said. When normal kdump service starts, it will reload. At the same time, early kdump will be overwritten by normal kdump.
Probably "early kdump load" is better than "early kdump" in words.
initramfs or not, I notice dracut/systemd will print some message about they are running in initramfs, so probably you can check how to get it with same way, if this is not initramfs then just unload before the check in kdump loading.
The picture like below:
Kernel boot ->
initramfs --- early kdump load
---- Mark A ---- initramfs switch root
system startup (real root fs) service a service b ... (eg. networking etc.) kdump service start
-----Mark B ----- load kdump kernel again
The problem will happen between Mark A and Mark B, during this period, there could be repeated crash -> earlykdump_load, there might be some random crash as well since during the real root fs service startup, for example after network is ready if some network workload cause a panic, it maybe not 100% reproducible, so it seems we still need to make the poweroff configurable. eg.
default is poweroff, but one can choose if he can.
Yes, default is poweroff for early kdump. Unless kdump capture service happens error or enters the emergency service, one can choose the default action.(configure default=xxx in kdump.conf)
For default action instead of final action if you hardcode it, then even if one set default as reboot it still poweroff.
If really need, that can be improved.
[snip]
> > +check_rd_earlykdump() > +{ > + egrep "rd.earlykdump" /proc/cmdline > +} > + > start() > { > check_dump_feasibility > @@ -969,7 +974,13 @@ start() > check_current_status > if [ $? == 0 ]; then > echo "Kdump already running: [WARNING]" > - return 0 > + check_rd_earlykdump > + #if earlykdump loaded, it will stop and start. > + if [ $? -eq 0 ]; then > + stop
kdumpctl start can run not only by system startup services, one can also run it manually or in udev rule.
The checking of kernel cmdline seems not enough.
Here it means that if kdump has beend loaded, check whether early kdump did it. If yes, let normal kdump load it again, otherwise no need to do anything.
As we discussed you do not get my points here :)
check_rd_earlykdump will be always true once kernel bootup, so there is no way to get the first time of normal kdump load and the later one.
The early boot time panic to address for this patch is the 100% reproducible panic, for this kinds of panic admin should be able to see it when he boot the machine. So rethinking about this the best way may be just a wontfix.
Let's consider below use cases:
- First install:
install os -> ---A reboot -> ... ---B kdump service start -> create kdump initrd ... boot finished recreate default initrd and enable early kdump goto A
Panic happened between A and B, if it is predictable, eg. 100% reproducible, then admin should already see it, then he/she can control and stop the repeating crash/kdump loop if the panic is not 100% reproducible then use reboot as final action is just fine
- Other use cases eg. updating kernel or some other components:
It is similar with the intall os use case because if one update kernel or critical components it is likely they need regenerate kdump initrd, and then repack it into early kdump default initrd, in this case admin should be able to see the panic loop and handle it. Also if the panic is not 100% reproducible then we are just fine.
If we choose to split early and late kdump load, there could be other side effects, and make the logic even complicated.
So... the better way may be just leave it as is, and maybe add some documentation.
It is a good way to document the risks that may exist.
Just like the public transportation, we all know that it has the risk, but we still choose it.
Thanks.
Thoughts?
Thanks Dave
Hello Dave et al.
Kindly check the below condition, assumption and share your thoughts.
Buland, thank you for the reply.
Condition:
[0x1] System is up and running with the kernel version X. [0x2] Admin performed the kernel Y upgrade. [0x3] Running kernel X crashed. [0x4] Normal kdump rebooted the system and captured the kernel crash dump of the kernel X. [0x5] System rebooted with the newly installed kernel Y. [0x6] Let's assume that due to some unknown reason the booting kernel Y also crashed (assume that the panic is 100% reproducible). [0x7] Early kdump started dumping the kernel crash dump of the booting kernel Y. [0x8] Early kdump rebooted the system and stuck in the loop.
Assumption: 1
What if the problematic system is in data center and admin is not aware of this situation?
[0x1] The dump target will be filled with the multiple copies of the kernel crash dump?
[ 1 kernel crash dump of the kernel X and 'n' kernel crash dump of the kernel Y]
[0x2] The system will reboot in the loop?
It is hard to define, for original kdump service without early kdump load, it is also possible after one replaced a kernel the new kernel panics during boot phase just after kdump get loaded.
Admin at least should test a reboot after updating a kernel?
Assumption: 2
What if the dump target in on the local disk?
[0x1] Admin needs to power off the system manually to retrieve the kernel crash dump of the kernel X and Y from the resume mode. [0x2] Admin needs to remove the multiple copies of the kernel crash dump of the booting kernel Y. [0x3] Admin might get confused while differentiating between the kernel crash dump of the kernel X and Kernel Y.
As for the worst case we all admit this is a problem, we are more than happy to make admin be easier and fix it :)
As I said we are exporting about this problem see if we have a good solution.
But if we can assume predictable panic can be avoid then the situation will be better.
Regards, Buland
Thanks Dave
On 12/21/18 4:14 PM, Dave Young wrote:
On 12/21/18 at 12:59pm, Buland Singh wrote:
On 12/21/18 12:29 PM, Kairui Song wrote:
Hi, Dave, Lianbo
My concern is that crash loop may generate tons of dump cores, and the dump target may get filled up by dump cores, that may have larger potential risk. Else I think it's good to leave it as it is.
On Fri, Dec 21, 2018 at 2:05 PM lijiang lijiang@redhat.com wrote:
在 2018年12月21日 10:49, Dave Young 写道:
- more people
On 12/20/18 at 04:49pm, lijiang wrote:
在 2018年12月20日 13:57, Dave Young 写道: > On 12/20/18 at 01:06pm, Lianbo Jiang wrote: >> By default, early kdump reboots the system after capturing the vmcore. >> If the problematic system is continuously crashing due to some issue >> during early boot stage, the system may fall into infinite loop restart >> like this: >> >> boot -----> crash -----> early kdump (dump vmcore) >> ^ | >> '.........(reboot).............' >> >> But now, the system crash at early stage is only captured by early kdump, >> and the rest is captured by normal kdump. That to say, when normal kdump >> service starts, it will load it again and override early kdump. It is >> helpful to control the logic of early kdump and normal kdump separately >> in final action(it is called by kdump-capture.service). For example, >> early kdump always passes the 'rd.earlykdump' to the second kernel when >> early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' >> to the second kernel at any time. So they can be distinguished in the >> second kernel. > > Hmm, I'm confused about the param passing above. >
I copy some messages from another email, please refer to this one: [---> The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump and normal kdump can get the same parameters from /proc/cmdline in the first kernel.
Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't need it, normal kdump needs to remove the rd.earlykdump.
So which can distinguish early kdump and normal kdump in the second kernel. It helps to control the logic of kdump capture service. For example: default action/final action. ]
The description is confusing, "ealy kdump passes ... to the second kernel", for example about this, the real thing is one person adds the param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd kernel.
Yes. Good point. Thanks for your explanation.
Anyway this is patch log issue.
> Early or non early just means about the service loading phase, in
Yes. This patch used the same method what you said. When normal kdump service starts, it will reload. At the same time, early kdump will be overwritten by normal kdump.
Probably "early kdump load" is better than "early kdump" in words.
> initramfs or not, I notice dracut/systemd will print some message about > they are running in initramfs, so probably you can check how to get it > with same way, if this is not initramfs then just unload before the > check in kdump loading. > > The picture like below: > > Kernel boot -> > > initramfs --- > early kdump load > ---- Mark A ---- > initramfs switch root > > system startup (real root fs) > service a > service b ... (eg. networking etc.) > kdump service start > -----Mark B ----- > load kdump kernel again > > > The problem will happen between Mark A and Mark B, during this period, > there could be repeated crash -> earlykdump_load, there might be some > random crash as well since during the real root fs service startup, > for example after network is ready if some network workload cause a > panic, it maybe not 100% reproducible, so it seems we still need to > make the poweroff configurable. eg. > > default is poweroff, but one can choose if he can. >
Yes, default is poweroff for early kdump. Unless kdump capture service happens error or enters the emergency service, one can choose the default action.(configure default=xxx in kdump.conf)
For default action instead of final action if you hardcode it, then even if one set default as reboot it still poweroff.
If really need, that can be improved.
[snip]
>> >> +check_rd_earlykdump() >> +{ >> + egrep "rd.earlykdump" /proc/cmdline >> +} >> + >> start() >> { >> check_dump_feasibility >> @@ -969,7 +974,13 @@ start() >> check_current_status >> if [ $? == 0 ]; then >> echo "Kdump already running: [WARNING]" >> - return 0 >> + check_rd_earlykdump >> + #if earlykdump loaded, it will stop and start. >> + if [ $? -eq 0 ]; then >> + stop > > kdumpctl start can run not only by system startup services, one can also > run it manually or in udev rule. > > The checking of kernel cmdline seems not enough. >
Here it means that if kdump has beend loaded, check whether early kdump did it. If yes, let normal kdump load it again, otherwise no need to do anything.
As we discussed you do not get my points here :)
check_rd_earlykdump will be always true once kernel bootup, so there is no way to get the first time of normal kdump load and the later one.
The early boot time panic to address for this patch is the 100% reproducible panic, for this kinds of panic admin should be able to see it when he boot the machine. So rethinking about this the best way may be just a wontfix.
Let's consider below use cases:
- First install:
install os -> ---A reboot -> ... ---B kdump service start -> create kdump initrd ... boot finished recreate default initrd and enable early kdump goto A
Panic happened between A and B, if it is predictable, eg. 100% reproducible, then admin should already see it, then he/she can control and stop the repeating crash/kdump loop if the panic is not 100% reproducible then use reboot as final action is just fine
- Other use cases eg. updating kernel or some other components:
It is similar with the intall os use case because if one update kernel or critical components it is likely they need regenerate kdump initrd, and then repack it into early kdump default initrd, in this case admin should be able to see the panic loop and handle it. Also if the panic is not 100% reproducible then we are just fine.
If we choose to split early and late kdump load, there could be other side effects, and make the logic even complicated.
So... the better way may be just leave it as is, and maybe add some documentation.
It is a good way to document the risks that may exist.
Just like the public transportation, we all know that it has the risk, but we still choose it.
Thanks.
Thoughts?
Thanks Dave
Hello Dave et al.
Kindly check the below condition, assumption and share your thoughts.
Buland, thank you for the reply.
Condition:
[0x1] System is up and running with the kernel version X. [0x2] Admin performed the kernel Y upgrade. [0x3] Running kernel X crashed. [0x4] Normal kdump rebooted the system and captured the kernel crash dump of the kernel X. [0x5] System rebooted with the newly installed kernel Y. [0x6] Let's assume that due to some unknown reason the booting kernel Y also crashed (assume that the panic is 100% reproducible). [0x7] Early kdump started dumping the kernel crash dump of the booting kernel Y. [0x8] Early kdump rebooted the system and stuck in the loop.
Assumption: 1
What if the problematic system is in data center and admin is not aware of this situation?
[0x1] The dump target will be filled with the multiple copies of the kernel crash dump?
[ 1 kernel crash dump of the kernel X and 'n' kernel crash dump of the kernel Y]
[0x2] The system will reboot in the loop?
It is hard to define, for original kdump service without early kdump load, it is also possible after one replaced a kernel the new kernel panics during boot phase just after kdump get loaded.
Admin at least should test a reboot after updating a kernel?
Agree, but not sure if all the admins will follow this rule :)
Assumption: 2
What if the dump target in on the local disk?
[0x1] Admin needs to power off the system manually to retrieve the kernel crash dump of the kernel X and Y from the resume mode. [0x2] Admin needs to remove the multiple copies of the kernel crash dump of the booting kernel Y. [0x3] Admin might get confused while differentiating between the kernel crash dump of the kernel X and Kernel Y.
As for the worst case we all admit this is a problem, we are more than happy to make admin be easier and fix it :)
As I said we are exporting about this problem see if we have a good solution.
But if we can assume predictable panic can be avoid then the situation will be better.
One suggestion, can we have a separate default behavior for normal kdump and early kdump?
Eg:
In /etc/kdump.conf file. default reboot (Default action to be taken by normal kdump and early kdump if dumping fails) ndefault reboot (Default action to be taken once the dumping is successful by normal kdump) edefault poweroff (Default action to be taken once the dumping is successful by early kdump)
Note: Create 'ndefault' & 'edefault' options identical to 'default' option so that an admin can alter the behavior as per the requirement.
[0x1] If kernel crash dump is captured by normal kdump then 'reboot' the system (by default) or take action as per 'ndefault' value. [0x2] If kernel crash dump is captured by early kdump then 'poweroff' the system 9by default) or take action as per 'edefault' value.
-- Buland
On 12/21/18 at 04:47pm, Buland Singh wrote:
On 12/21/18 4:14 PM, Dave Young wrote:
On 12/21/18 at 12:59pm, Buland Singh wrote:
On 12/21/18 12:29 PM, Kairui Song wrote:
Hi, Dave, Lianbo
My concern is that crash loop may generate tons of dump cores, and the dump target may get filled up by dump cores, that may have larger potential risk. Else I think it's good to leave it as it is.
On Fri, Dec 21, 2018 at 2:05 PM lijiang lijiang@redhat.com wrote:
在 2018年12月21日 10:49, Dave Young 写道:
- more people
On 12/20/18 at 04:49pm, lijiang wrote: > 在 2018年12月20日 13:57, Dave Young 写道: > > On 12/20/18 at 01:06pm, Lianbo Jiang wrote: > > > By default, early kdump reboots the system after capturing the vmcore. > > > If the problematic system is continuously crashing due to some issue > > > during early boot stage, the system may fall into infinite loop restart > > > like this: > > > > > > boot -----> crash -----> early kdump (dump vmcore) > > > ^ | > > > '.........(reboot).............' > > > > > > But now, the system crash at early stage is only captured by early kdump, > > > and the rest is captured by normal kdump. That to say, when normal kdump > > > service starts, it will load it again and override early kdump. It is > > > helpful to control the logic of early kdump and normal kdump separately > > > in final action(it is called by kdump-capture.service). For example, > > > early kdump always passes the 'rd.earlykdump' to the second kernel when > > > early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' > > > to the second kernel at any time. So they can be distinguished in the > > > second kernel. > > > > Hmm, I'm confused about the param passing above. > > > > I copy some messages from another email, please refer to this one: > [---> > The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump > and normal kdump can get the same parameters from /proc/cmdline in the first kernel. > > Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't > need it, normal kdump needs to remove the rd.earlykdump. > > So which can distinguish early kdump and normal kdump in the second kernel. It helps > to control the logic of kdump capture service. For example: default action/final action. > ]
The description is confusing, "ealy kdump passes ... to the second kernel", for example about this, the real thing is one person adds the param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd kernel.
Yes. Good point. Thanks for your explanation.
Anyway this is patch log issue.
> > > Early or non early just means about the service loading phase, in > > Yes. This patch used the same method what you said. When normal kdump service starts, > it will reload. At the same time, early kdump will be overwritten by normal kdump.
Probably "early kdump load" is better than "early kdump" in words.
> > > initramfs or not, I notice dracut/systemd will print some message about > > they are running in initramfs, so probably you can check how to get it > > with same way, if this is not initramfs then just unload before the > > check in kdump loading. > > > > The picture like below: > > > > Kernel boot -> > > > > initramfs --- > > early kdump load > > ---- Mark A ---- > > initramfs switch root > > > > system startup (real root fs) > > service a > > service b ... (eg. networking etc.) > > kdump service start > > -----Mark B ----- > > load kdump kernel again > > > > > > The problem will happen between Mark A and Mark B, during this period, > > there could be repeated crash -> earlykdump_load, there might be some > > random crash as well since during the real root fs service startup, > > for example after network is ready if some network workload cause a > > panic, it maybe not 100% reproducible, so it seems we still need to > > make the poweroff configurable. eg. > > > > default is poweroff, but one can choose if he can. > > > > Yes, default is poweroff for early kdump. Unless kdump capture service > happens error or enters the emergency service, one can choose the default > action.(configure default=xxx in kdump.conf)
For default action instead of final action if you hardcode it, then even if one set default as reboot it still poweroff.
If really need, that can be improved.
[snip]
> > > > > > +check_rd_earlykdump() > > > +{ > > > + egrep "rd.earlykdump" /proc/cmdline > > > +} > > > + > > > start() > > > { > > > check_dump_feasibility > > > @@ -969,7 +974,13 @@ start() > > > check_current_status > > > if [ $? == 0 ]; then > > > echo "Kdump already running: [WARNING]" > > > - return 0 > > > + check_rd_earlykdump > > > + #if earlykdump loaded, it will stop and start. > > > + if [ $? -eq 0 ]; then > > > + stop > > > > kdumpctl start can run not only by system startup services, one can also > > run it manually or in udev rule. > > > > The checking of kernel cmdline seems not enough. > > > > Here it means that if kdump has beend loaded, check whether early kdump did it. > If yes, let normal kdump load it again, otherwise no need to do anything.
As we discussed you do not get my points here :)
check_rd_earlykdump will be always true once kernel bootup, so there is no way to get the first time of normal kdump load and the later one.
The early boot time panic to address for this patch is the 100% reproducible panic, for this kinds of panic admin should be able to see it when he boot the machine. So rethinking about this the best way may be just a wontfix.
Let's consider below use cases:
- First install:
install os -> ---A reboot -> ... ---B kdump service start -> create kdump initrd ... boot finished recreate default initrd and enable early kdump goto A
Panic happened between A and B, if it is predictable, eg. 100% reproducible, then admin should already see it, then he/she can control and stop the repeating crash/kdump loop if the panic is not 100% reproducible then use reboot as final action is just fine
- Other use cases eg. updating kernel or some other components:
It is similar with the intall os use case because if one update kernel or critical components it is likely they need regenerate kdump initrd, and then repack it into early kdump default initrd, in this case admin should be able to see the panic loop and handle it. Also if the panic is not 100% reproducible then we are just fine.
If we choose to split early and late kdump load, there could be other side effects, and make the logic even complicated.
So... the better way may be just leave it as is, and maybe add some documentation.
It is a good way to document the risks that may exist.
Just like the public transportation, we all know that it has the risk, but we still choose it.
Thanks.
Thoughts?
Thanks Dave
Hello Dave et al.
Kindly check the below condition, assumption and share your thoughts.
Buland, thank you for the reply.
Condition:
[0x1] System is up and running with the kernel version X. [0x2] Admin performed the kernel Y upgrade. [0x3] Running kernel X crashed. [0x4] Normal kdump rebooted the system and captured the kernel crash dump of the kernel X. [0x5] System rebooted with the newly installed kernel Y. [0x6] Let's assume that due to some unknown reason the booting kernel Y also crashed (assume that the panic is 100% reproducible). [0x7] Early kdump started dumping the kernel crash dump of the booting kernel Y. [0x8] Early kdump rebooted the system and stuck in the loop.
Assumption: 1
What if the problematic system is in data center and admin is not aware of this situation?
[0x1] The dump target will be filled with the multiple copies of the kernel crash dump?
[ 1 kernel crash dump of the kernel X and 'n' kernel crash dump of the kernel Y]
[0x2] The system will reboot in the loop?
It is hard to define, for original kdump service without early kdump load, it is also possible after one replaced a kernel the new kernel panics during boot phase just after kdump get loaded.
Admin at least should test a reboot after updating a kernel?
Agree, but not sure if all the admins will follow this rule :)
Assumption: 2
What if the dump target in on the local disk?
[0x1] Admin needs to power off the system manually to retrieve the kernel crash dump of the kernel X and Y from the resume mode. [0x2] Admin needs to remove the multiple copies of the kernel crash dump of the booting kernel Y. [0x3] Admin might get confused while differentiating between the kernel crash dump of the kernel X and Kernel Y.
As for the worst case we all admit this is a problem, we are more than happy to make admin be easier and fix it :)
As I said we are exporting about this problem see if we have a good solution.
But if we can assume predictable panic can be avoid then the situation will be better.
One suggestion, can we have a separate default behavior for normal kdump and early kdump?
Yes if we can.
Kdump service start either early or late we just use a syscall to load another kernel/initrd into pre-reserved memory.
So once kdump service started we can not differenciate kdump kernel was loaded early or late.
Eg:
In /etc/kdump.conf file. default reboot (Default action to be taken by normal kdump and early kdump if dumping fails) ndefault reboot (Default action to be taken once the dumping is successful by normal kdump) edefault poweroff (Default action to be taken once the dumping is successful by early kdump)
Note: Create 'ndefault' & 'edefault' options identical to 'default' option so that an admin can alter the behavior as per the requirement.
[0x1] If kernel crash dump is captured by normal kdump then 'reboot' the system (by default) or take action as per 'ndefault' value. [0x2] If kernel crash dump is captured by early kdump then 'poweroff' the system 9by default) or take action as per 'edefault' value.
We can introduce something so that early kdump service load and set poweroff as default, but when normal kdump service start we have two choices:
1. go ahead to use the early loaded setup without reload like we have in the code now, in this case normal kdump will also use poweroff no matter what is set in /etc/kdump.conf
2. reload kernel/initrd with normal setup in /etc/kdump.conf. In this way, we need get if this is a late service startup. Because if we blindly reload then it will affect later udev triggered kdump restart.
For example manually start the service like kdumpctl start, origianlly it will just print a msg about the service is already running. But now the "start" == "stop, then start", this may be not a big problem but looks odd.
Correct my self about the udev hotplug triggered events, it should be ok because it will call restart that means stop then start, this change will not affects it.
BTW, the default action here should be "final action", there is a "default" action can be configured in kdump.conf which is used for kdump kernel to do after vmcore saving failed, it is more like a failsafe action. But the default "default" is also "reboot". When kdump successfully saves a vmcore it will go to "final action" (==reboot) which is not configurable.
-- Buland
Thanks Dave
On 12/21/18 at 07:38pm, Dave Young wrote:
On 12/21/18 at 04:47pm, Buland Singh wrote:
On 12/21/18 4:14 PM, Dave Young wrote:
On 12/21/18 at 12:59pm, Buland Singh wrote:
On 12/21/18 12:29 PM, Kairui Song wrote:
Hi, Dave, Lianbo
My concern is that crash loop may generate tons of dump cores, and the dump target may get filled up by dump cores, that may have larger potential risk. Else I think it's good to leave it as it is.
On Fri, Dec 21, 2018 at 2:05 PM lijiang lijiang@redhat.com wrote:
在 2018年12月21日 10:49, Dave Young 写道: > + more people > On 12/20/18 at 04:49pm, lijiang wrote: > > 在 2018年12月20日 13:57, Dave Young 写道: > > > On 12/20/18 at 01:06pm, Lianbo Jiang wrote: > > > > By default, early kdump reboots the system after capturing the vmcore. > > > > If the problematic system is continuously crashing due to some issue > > > > during early boot stage, the system may fall into infinite loop restart > > > > like this: > > > > > > > > boot -----> crash -----> early kdump (dump vmcore) > > > > ^ | > > > > '.........(reboot).............' > > > > > > > > But now, the system crash at early stage is only captured by early kdump, > > > > and the rest is captured by normal kdump. That to say, when normal kdump > > > > service starts, it will load it again and override early kdump. It is > > > > helpful to control the logic of early kdump and normal kdump separately > > > > in final action(it is called by kdump-capture.service). For example, > > > > early kdump always passes the 'rd.earlykdump' to the second kernel when > > > > early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' > > > > to the second kernel at any time. So they can be distinguished in the > > > > second kernel. > > > > > > Hmm, I'm confused about the param passing above. > > > > > > > I copy some messages from another email, please refer to this one: > > [---> > > The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump > > and normal kdump can get the same parameters from /proc/cmdline in the first kernel. > > > > Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't > > need it, normal kdump needs to remove the rd.earlykdump. > > > > So which can distinguish early kdump and normal kdump in the second kernel. It helps > > to control the logic of kdump capture service. For example: default action/final action. > > ] > > The description is confusing, "ealy kdump passes ... to the second > kernel", for example about this, the real thing is one person adds the > param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd > kernel. >
Yes. Good point. Thanks for your explanation.
> Anyway this is patch log issue. > > > > > > Early or non early just means about the service loading phase, in > > > > Yes. This patch used the same method what you said. When normal kdump service starts, > > it will reload. At the same time, early kdump will be overwritten by normal kdump. > > Probably "early kdump load" is better than "early kdump" in words. > > > > > > initramfs or not, I notice dracut/systemd will print some message about > > > they are running in initramfs, so probably you can check how to get it > > > with same way, if this is not initramfs then just unload before the > > > check in kdump loading. > > > > > > The picture like below: > > > > > > Kernel boot -> > > > > > > initramfs --- > > > early kdump load > > > ---- Mark A ---- > > > initramfs switch root > > > > > > system startup (real root fs) > > > service a > > > service b ... (eg. networking etc.) > > > kdump service start > > > -----Mark B ----- > > > load kdump kernel again > > > > > > > > > The problem will happen between Mark A and Mark B, during this period, > > > there could be repeated crash -> earlykdump_load, there might be some > > > random crash as well since during the real root fs service startup, > > > for example after network is ready if some network workload cause a > > > panic, it maybe not 100% reproducible, so it seems we still need to > > > make the poweroff configurable. eg. > > > > > > default is poweroff, but one can choose if he can. > > > > > > > Yes, default is poweroff for early kdump. Unless kdump capture service > > happens error or enters the emergency service, one can choose the default > > action.(configure default=xxx in kdump.conf) > > For default action instead of final action if you hardcode it, then even if > one set default as reboot it still poweroff. >
If really need, that can be improved.
> [snip] > > > > > > > > > +check_rd_earlykdump() > > > > +{ > > > > + egrep "rd.earlykdump" /proc/cmdline > > > > +} > > > > + > > > > start() > > > > { > > > > check_dump_feasibility > > > > @@ -969,7 +974,13 @@ start() > > > > check_current_status > > > > if [ $? == 0 ]; then > > > > echo "Kdump already running: [WARNING]" > > > > - return 0 > > > > + check_rd_earlykdump > > > > + #if earlykdump loaded, it will stop and start. > > > > + if [ $? -eq 0 ]; then > > > > + stop > > > > > > kdumpctl start can run not only by system startup services, one can also > > > run it manually or in udev rule. > > > > > > The checking of kernel cmdline seems not enough. > > > > > > > Here it means that if kdump has beend loaded, check whether early kdump did it. > > If yes, let normal kdump load it again, otherwise no need to do anything. > > As we discussed you do not get my points here :) > > check_rd_earlykdump will be always true once kernel bootup, so there is > no way to get the first time of normal kdump load and the later one. > > The early boot time panic to address for this patch is the 100% > reproducible panic, for this kinds of panic admin should be able to see > it when he boot the machine. So rethinking about this the best way may > be just a wontfix. > > Let's consider below use cases: > > * First install: > > install os -> > ---A > reboot -> > ... > ---B > kdump service start > -> create kdump initrd > ... > boot finished > recreate default initrd and enable early kdump > goto A > > Panic happened between A and B, if it is predictable, eg. 100% > reproducible, then admin should already see it, then he/she can control > and stop the repeating crash/kdump loop > if the panic is not 100% reproducible then use reboot as final action is just fine > > * Other use cases eg. updating kernel or some other components: > It is similar with the intall os use case because if one update kernel > or critical components it is likely they need regenerate kdump initrd, > and then repack it into early kdump default initrd, in this case admin > should be able to see the panic loop and handle it. > Also if the panic is not 100% reproducible then we are just fine. > > If we choose to split early and late kdump load, there could be other > side effects, and make the logic even complicated. > > So... the better way may be just leave it as is, and maybe add some > documentation. >
It is a good way to document the risks that may exist.
Just like the public transportation, we all know that it has the risk, but we still choose it.
Thanks.
> Thoughts? > > Thanks > Dave >
Hello Dave et al.
Kindly check the below condition, assumption and share your thoughts.
Buland, thank you for the reply.
Condition:
[0x1] System is up and running with the kernel version X. [0x2] Admin performed the kernel Y upgrade. [0x3] Running kernel X crashed. [0x4] Normal kdump rebooted the system and captured the kernel crash dump of the kernel X. [0x5] System rebooted with the newly installed kernel Y. [0x6] Let's assume that due to some unknown reason the booting kernel Y also crashed (assume that the panic is 100% reproducible). [0x7] Early kdump started dumping the kernel crash dump of the booting kernel Y. [0x8] Early kdump rebooted the system and stuck in the loop.
Assumption: 1
What if the problematic system is in data center and admin is not aware of this situation?
[0x1] The dump target will be filled with the multiple copies of the kernel crash dump?
[ 1 kernel crash dump of the kernel X and 'n' kernel crash dump of the kernel Y]
[0x2] The system will reboot in the loop?
It is hard to define, for original kdump service without early kdump load, it is also possible after one replaced a kernel the new kernel panics during boot phase just after kdump get loaded.
Admin at least should test a reboot after updating a kernel?
Agree, but not sure if all the admins will follow this rule :)
Assumption: 2
What if the dump target in on the local disk?
[0x1] Admin needs to power off the system manually to retrieve the kernel crash dump of the kernel X and Y from the resume mode. [0x2] Admin needs to remove the multiple copies of the kernel crash dump of the booting kernel Y. [0x3] Admin might get confused while differentiating between the kernel crash dump of the kernel X and Kernel Y.
As for the worst case we all admit this is a problem, we are more than happy to make admin be easier and fix it :)
As I said we are exporting about this problem see if we have a good solution.
But if we can assume predictable panic can be avoid then the situation will be better.
One suggestion, can we have a separate default behavior for normal kdump and early kdump?
Yes if we can.
Kdump service start either early or late we just use a syscall to load another kernel/initrd into pre-reserved memory.
So once kdump service started we can not differenciate kdump kernel was loaded early or late.
Eg:
In /etc/kdump.conf file. default reboot (Default action to be taken by normal kdump and early kdump if dumping fails) ndefault reboot (Default action to be taken once the dumping is successful by normal kdump) edefault poweroff (Default action to be taken once the dumping is successful by early kdump)
Note: Create 'ndefault' & 'edefault' options identical to 'default' option so that an admin can alter the behavior as per the requirement.
[0x1] If kernel crash dump is captured by normal kdump then 'reboot' the system (by default) or take action as per 'ndefault' value. [0x2] If kernel crash dump is captured by early kdump then 'poweroff' the system 9by default) or take action as per 'edefault' value.
We can introduce something so that early kdump service load and set poweroff as default, but when normal kdump service start we have two choices:
- go ahead to use the early loaded setup without reload like we have in
the code now, in this case normal kdump will also use poweroff no matter what is set in /etc/kdump.conf
- reload kernel/initrd with normal setup in /etc/kdump.conf. In this
way, we need get if this is a late service startup. Because if we blindly reload then it will affect later udev triggered kdump restart.
I have just corrected myself but forgot here, s/udev/triggered//
For example manually start the service like kdumpctl start, origianlly it will just print a msg about the service is already running. But now the "start" == "stop, then start", this may be not a big problem but looks odd.
Correct my self about the udev hotplug triggered events, it should be ok because it will call restart that means stop then start, this change will not affects it.
BTW, the default action here should be "final action", there is a "default" action can be configured in kdump.conf which is used for kdump kernel to do after vmcore saving failed, it is more like a failsafe action. But the default "default" is also "reboot". When kdump successfully saves a vmcore it will go to "final action" (==reboot) which is not configurable.
-- Buland
Thanks Dave
On 12/21/18 at 07:38pm, Dave Young wrote:
On 12/21/18 at 04:47pm, Buland Singh wrote:
On 12/21/18 4:14 PM, Dave Young wrote:
On 12/21/18 at 12:59pm, Buland Singh wrote:
On 12/21/18 12:29 PM, Kairui Song wrote:
Hi, Dave, Lianbo
My concern is that crash loop may generate tons of dump cores, and the dump target may get filled up by dump cores, that may have larger potential risk. Else I think it's good to leave it as it is.
On Fri, Dec 21, 2018 at 2:05 PM lijiang lijiang@redhat.com wrote:
在 2018年12月21日 10:49, Dave Young 写道: > + more people > On 12/20/18 at 04:49pm, lijiang wrote: > > 在 2018年12月20日 13:57, Dave Young 写道: > > > On 12/20/18 at 01:06pm, Lianbo Jiang wrote: > > > > By default, early kdump reboots the system after capturing the vmcore. > > > > If the problematic system is continuously crashing due to some issue > > > > during early boot stage, the system may fall into infinite loop restart > > > > like this: > > > > > > > > boot -----> crash -----> early kdump (dump vmcore) > > > > ^ | > > > > '.........(reboot).............' > > > > > > > > But now, the system crash at early stage is only captured by early kdump, > > > > and the rest is captured by normal kdump. That to say, when normal kdump > > > > service starts, it will load it again and override early kdump. It is > > > > helpful to control the logic of early kdump and normal kdump separately > > > > in final action(it is called by kdump-capture.service). For example, > > > > early kdump always passes the 'rd.earlykdump' to the second kernel when > > > > early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' > > > > to the second kernel at any time. So they can be distinguished in the > > > > second kernel. > > > > > > Hmm, I'm confused about the param passing above. > > > > > > > I copy some messages from another email, please refer to this one: > > [---> > > The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump > > and normal kdump can get the same parameters from /proc/cmdline in the first kernel. > > > > Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't > > need it, normal kdump needs to remove the rd.earlykdump. > > > > So which can distinguish early kdump and normal kdump in the second kernel. It helps > > to control the logic of kdump capture service. For example: default action/final action. > > ] > > The description is confusing, "ealy kdump passes ... to the second > kernel", for example about this, the real thing is one person adds the > param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd > kernel. >
Yes. Good point. Thanks for your explanation.
> Anyway this is patch log issue. > > > > > > Early or non early just means about the service loading phase, in > > > > Yes. This patch used the same method what you said. When normal kdump service starts, > > it will reload. At the same time, early kdump will be overwritten by normal kdump. > > Probably "early kdump load" is better than "early kdump" in words. > > > > > > initramfs or not, I notice dracut/systemd will print some message about > > > they are running in initramfs, so probably you can check how to get it > > > with same way, if this is not initramfs then just unload before the > > > check in kdump loading. > > > > > > The picture like below: > > > > > > Kernel boot -> > > > > > > initramfs --- > > > early kdump load > > > ---- Mark A ---- > > > initramfs switch root > > > > > > system startup (real root fs) > > > service a > > > service b ... (eg. networking etc.) > > > kdump service start > > > -----Mark B ----- > > > load kdump kernel again > > > > > > > > > The problem will happen between Mark A and Mark B, during this period, > > > there could be repeated crash -> earlykdump_load, there might be some > > > random crash as well since during the real root fs service startup, > > > for example after network is ready if some network workload cause a > > > panic, it maybe not 100% reproducible, so it seems we still need to > > > make the poweroff configurable. eg. > > > > > > default is poweroff, but one can choose if he can. > > > > > > > Yes, default is poweroff for early kdump. Unless kdump capture service > > happens error or enters the emergency service, one can choose the default > > action.(configure default=xxx in kdump.conf) > > For default action instead of final action if you hardcode it, then even if > one set default as reboot it still poweroff. >
If really need, that can be improved.
> [snip] > > > > > > > > > +check_rd_earlykdump() > > > > +{ > > > > + egrep "rd.earlykdump" /proc/cmdline > > > > +} > > > > + > > > > start() > > > > { > > > > check_dump_feasibility > > > > @@ -969,7 +974,13 @@ start() > > > > check_current_status > > > > if [ $? == 0 ]; then > > > > echo "Kdump already running: [WARNING]" > > > > - return 0 > > > > + check_rd_earlykdump > > > > + #if earlykdump loaded, it will stop and start. > > > > + if [ $? -eq 0 ]; then > > > > + stop > > > > > > kdumpctl start can run not only by system startup services, one can also > > > run it manually or in udev rule. > > > > > > The checking of kernel cmdline seems not enough. > > > > > > > Here it means that if kdump has beend loaded, check whether early kdump did it. > > If yes, let normal kdump load it again, otherwise no need to do anything. > > As we discussed you do not get my points here :) > > check_rd_earlykdump will be always true once kernel bootup, so there is > no way to get the first time of normal kdump load and the later one. > > The early boot time panic to address for this patch is the 100% > reproducible panic, for this kinds of panic admin should be able to see > it when he boot the machine. So rethinking about this the best way may > be just a wontfix. > > Let's consider below use cases: > > * First install: > > install os -> > ---A > reboot -> > ... > ---B > kdump service start > -> create kdump initrd > ... > boot finished > recreate default initrd and enable early kdump > goto A > > Panic happened between A and B, if it is predictable, eg. 100% > reproducible, then admin should already see it, then he/she can control > and stop the repeating crash/kdump loop > if the panic is not 100% reproducible then use reboot as final action is just fine > > * Other use cases eg. updating kernel or some other components: > It is similar with the intall os use case because if one update kernel > or critical components it is likely they need regenerate kdump initrd, > and then repack it into early kdump default initrd, in this case admin > should be able to see the panic loop and handle it. > Also if the panic is not 100% reproducible then we are just fine. > > If we choose to split early and late kdump load, there could be other > side effects, and make the logic even complicated. > > So... the better way may be just leave it as is, and maybe add some > documentation. >
It is a good way to document the risks that may exist.
Just like the public transportation, we all know that it has the risk, but we still choose it.
Thanks.
> Thoughts? > > Thanks > Dave >
Hello Dave et al.
Kindly check the below condition, assumption and share your thoughts.
Buland, thank you for the reply.
Condition:
[0x1] System is up and running with the kernel version X. [0x2] Admin performed the kernel Y upgrade. [0x3] Running kernel X crashed. [0x4] Normal kdump rebooted the system and captured the kernel crash dump of the kernel X. [0x5] System rebooted with the newly installed kernel Y. [0x6] Let's assume that due to some unknown reason the booting kernel Y also crashed (assume that the panic is 100% reproducible). [0x7] Early kdump started dumping the kernel crash dump of the booting kernel Y. [0x8] Early kdump rebooted the system and stuck in the loop.
Assumption: 1
What if the problematic system is in data center and admin is not aware of this situation?
[0x1] The dump target will be filled with the multiple copies of the kernel crash dump?
[ 1 kernel crash dump of the kernel X and 'n' kernel crash dump of the kernel Y]
[0x2] The system will reboot in the loop?
It is hard to define, for original kdump service without early kdump load, it is also possible after one replaced a kernel the new kernel panics during boot phase just after kdump get loaded.
Admin at least should test a reboot after updating a kernel?
Agree, but not sure if all the admins will follow this rule :)
Assumption: 2
What if the dump target in on the local disk?
[0x1] Admin needs to power off the system manually to retrieve the kernel crash dump of the kernel X and Y from the resume mode. [0x2] Admin needs to remove the multiple copies of the kernel crash dump of the booting kernel Y. [0x3] Admin might get confused while differentiating between the kernel crash dump of the kernel X and Kernel Y.
As for the worst case we all admit this is a problem, we are more than happy to make admin be easier and fix it :)
As I said we are exporting about this problem see if we have a good solution.
But if we can assume predictable panic can be avoid then the situation will be better.
One suggestion, can we have a separate default behavior for normal kdump and early kdump?
Yes if we can.
Kdump service start either early or late we just use a syscall to load another kernel/initrd into pre-reserved memory.
So once kdump service started we can not differenciate kdump kernel was loaded early or late.
But still not sure about it. As we can see for normal kdump there will be similar issue existed, eg. between C and D if a reproducible panic also happens every time during late boot phase
So it is hard to define this is early kdump only, just more likely for early kdump.
---A initramfs kdump load switch root other services startup ---B kdump service start ---C other services start up ... ---D boot finished
If we consider this as a early kdump only/must-fix issue, thinking about it we should split into two issues:
1. how to determine early load and then reload while kdumpctl start
If "kdump" service is not active but kdump kernel loaded then it should have been "early loaded". Then something will like this:
kdumpctl start() if kdump is loaded: if systemd kdump.service is active # early loaded stop and continue to load again else print a warning service is already running and then return else go ahead to load and start
2. how to set the reboot action for early and late loading:
use an cmdline like rd.earlykdump.noreboot, default value is true, if one want to use alternate he/she can add rd.earlykdump.noreboot=0
Add extra config optioins is also a choice but seems much complicated, not only final action, also default action, for default action in case partially saved vmcore it still will occupy the whole disk. So seems we just should replace reboot with shutdown for any cases.
Eg:
In /etc/kdump.conf file. default reboot (Default action to be taken by normal kdump and early kdump if dumping fails) ndefault reboot (Default action to be taken once the dumping is successful by normal kdump) edefault poweroff (Default action to be taken once the dumping is successful by early kdump)
Note: Create 'ndefault' & 'edefault' options identical to 'default' option so that an admin can alter the behavior as per the requirement.
[0x1] If kernel crash dump is captured by normal kdump then 'reboot' the system (by default) or take action as per 'ndefault' value. [0x2] If kernel crash dump is captured by early kdump then 'poweroff' the system 9by default) or take action as per 'edefault' value.
We can introduce something so that early kdump service load and set poweroff as default, but when normal kdump service start we have two choices:
- go ahead to use the early loaded setup without reload like we have in
the code now, in this case normal kdump will also use poweroff no matter what is set in /etc/kdump.conf
- reload kernel/initrd with normal setup in /etc/kdump.conf. In this
way, we need get if this is a late service startup. Because if we blindly reload then it will affect later udev triggered kdump restart.
For example manually start the service like kdumpctl start, origianlly it will just print a msg about the service is already running. But now the "start" == "stop, then start", this may be not a big problem but looks odd.
Correct my self about the udev hotplug triggered events, it should be ok because it will call restart that means stop then start, this change will not affects it.
BTW, the default action here should be "final action", there is a "default" action can be configured in kdump.conf which is used for kdump kernel to do after vmcore saving failed, it is more like a failsafe action. But the default "default" is also "reboot". When kdump successfully saves a vmcore it will go to "final action" (==reboot) which is not configurable.
-- Buland
Thanks Dave
在 2018年12月24日 12:17, Dave Young 写道:
On 12/21/18 at 07:38pm, Dave Young wrote:
On 12/21/18 at 04:47pm, Buland Singh wrote:
On 12/21/18 4:14 PM, Dave Young wrote:
On 12/21/18 at 12:59pm, Buland Singh wrote:
On 12/21/18 12:29 PM, Kairui Song wrote:
Hi, Dave, Lianbo
My concern is that crash loop may generate tons of dump cores, and the dump target may get filled up by dump cores, that may have larger potential risk. Else I think it's good to leave it as it is.
On Fri, Dec 21, 2018 at 2:05 PM lijiang lijiang@redhat.com wrote: > > 在 2018年12月21日 10:49, Dave Young 写道: >> + more people >> On 12/20/18 at 04:49pm, lijiang wrote: >>> 在 2018年12月20日 13:57, Dave Young 写道: >>>> On 12/20/18 at 01:06pm, Lianbo Jiang wrote: >>>>> By default, early kdump reboots the system after capturing the vmcore. >>>>> If the problematic system is continuously crashing due to some issue >>>>> during early boot stage, the system may fall into infinite loop restart >>>>> like this: >>>>> >>>>> boot -----> crash -----> early kdump (dump vmcore) >>>>> ^ | >>>>> '.........(reboot).............' >>>>> >>>>> But now, the system crash at early stage is only captured by early kdump, >>>>> and the rest is captured by normal kdump. That to say, when normal kdump >>>>> service starts, it will load it again and override early kdump. It is >>>>> helpful to control the logic of early kdump and normal kdump separately >>>>> in final action(it is called by kdump-capture.service). For example, >>>>> early kdump always passes the 'rd.earlykdump' to the second kernel when >>>>> early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' >>>>> to the second kernel at any time. So they can be distinguished in the >>>>> second kernel. >>>> >>>> Hmm, I'm confused about the param passing above. >>>> >>> >>> I copy some messages from another email, please refer to this one: >>> [---> >>> The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump >>> and normal kdump can get the same parameters from /proc/cmdline in the first kernel. >>> >>> Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't >>> need it, normal kdump needs to remove the rd.earlykdump. >>> >>> So which can distinguish early kdump and normal kdump in the second kernel. It helps >>> to control the logic of kdump capture service. For example: default action/final action. >>> ] >> >> The description is confusing, "ealy kdump passes ... to the second >> kernel", for example about this, the real thing is one person adds the >> param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd >> kernel. >> > > Yes. Good point. Thanks for your explanation. > >> Anyway this is patch log issue. >> >>> >>>> Early or non early just means about the service loading phase, in >>> >>> Yes. This patch used the same method what you said. When normal kdump service starts, >>> it will reload. At the same time, early kdump will be overwritten by normal kdump. >> >> Probably "early kdump load" is better than "early kdump" in words. >> >>> >>>> initramfs or not, I notice dracut/systemd will print some message about >>>> they are running in initramfs, so probably you can check how to get it >>>> with same way, if this is not initramfs then just unload before the >>>> check in kdump loading. >>>> >>>> The picture like below: >>>> >>>> Kernel boot -> >>>> >>>> initramfs --- >>>> early kdump load >>>> ---- Mark A ---- >>>> initramfs switch root >>>> >>>> system startup (real root fs) >>>> service a >>>> service b ... (eg. networking etc.) >>>> kdump service start >>>> -----Mark B ----- >>>> load kdump kernel again >>>> >>>> >>>> The problem will happen between Mark A and Mark B, during this period, >>>> there could be repeated crash -> earlykdump_load, there might be some >>>> random crash as well since during the real root fs service startup, >>>> for example after network is ready if some network workload cause a >>>> panic, it maybe not 100% reproducible, so it seems we still need to >>>> make the poweroff configurable. eg. >>>> >>>> default is poweroff, but one can choose if he can. >>>> >>> >>> Yes, default is poweroff for early kdump. Unless kdump capture service >>> happens error or enters the emergency service, one can choose the default >>> action.(configure default=xxx in kdump.conf) >> >> For default action instead of final action if you hardcode it, then even if >> one set default as reboot it still poweroff. >> > > If really need, that can be improved. > >> [snip] >> >>>>> >>>>> +check_rd_earlykdump() >>>>> +{ >>>>> + egrep "rd.earlykdump" /proc/cmdline >>>>> +} >>>>> + >>>>> start() >>>>> { >>>>> check_dump_feasibility >>>>> @@ -969,7 +974,13 @@ start() >>>>> check_current_status >>>>> if [ $? == 0 ]; then >>>>> echo "Kdump already running: [WARNING]" >>>>> - return 0 >>>>> + check_rd_earlykdump >>>>> + #if earlykdump loaded, it will stop and start. >>>>> + if [ $? -eq 0 ]; then >>>>> + stop >>>> >>>> kdumpctl start can run not only by system startup services, one can also >>>> run it manually or in udev rule. >>>> >>>> The checking of kernel cmdline seems not enough. >>>> >>> >>> Here it means that if kdump has beend loaded, check whether early kdump did it. >>> If yes, let normal kdump load it again, otherwise no need to do anything. >> >> As we discussed you do not get my points here :) >> >> check_rd_earlykdump will be always true once kernel bootup, so there is >> no way to get the first time of normal kdump load and the later one. >> >> The early boot time panic to address for this patch is the 100% >> reproducible panic, for this kinds of panic admin should be able to see >> it when he boot the machine. So rethinking about this the best way may >> be just a wontfix. >> >> Let's consider below use cases: >> >> * First install: >> >> install os -> >> ---A >> reboot -> >> ... >> ---B >> kdump service start >> -> create kdump initrd >> ... >> boot finished >> recreate default initrd and enable early kdump >> goto A >> >> Panic happened between A and B, if it is predictable, eg. 100% >> reproducible, then admin should already see it, then he/she can control >> and stop the repeating crash/kdump loop >> if the panic is not 100% reproducible then use reboot as final action is just fine >> >> * Other use cases eg. updating kernel or some other components: >> It is similar with the intall os use case because if one update kernel >> or critical components it is likely they need regenerate kdump initrd, >> and then repack it into early kdump default initrd, in this case admin >> should be able to see the panic loop and handle it. >> Also if the panic is not 100% reproducible then we are just fine. >> >> If we choose to split early and late kdump load, there could be other >> side effects, and make the logic even complicated. >> >> So... the better way may be just leave it as is, and maybe add some >> documentation. >> > > It is a good way to document the risks that may exist. > > Just like the public transportation, we all know that it has the risk, but we still choose it. > > Thanks. > >> Thoughts? >> >> Thanks >> Dave >>
Hello Dave et al.
Kindly check the below condition, assumption and share your thoughts.
Buland, thank you for the reply.
Condition:
[0x1] System is up and running with the kernel version X. [0x2] Admin performed the kernel Y upgrade. [0x3] Running kernel X crashed. [0x4] Normal kdump rebooted the system and captured the kernel crash dump of the kernel X. [0x5] System rebooted with the newly installed kernel Y. [0x6] Let's assume that due to some unknown reason the booting kernel Y also crashed (assume that the panic is 100% reproducible). [0x7] Early kdump started dumping the kernel crash dump of the booting kernel Y. [0x8] Early kdump rebooted the system and stuck in the loop.
Assumption: 1
What if the problematic system is in data center and admin is not aware of this situation?
[0x1] The dump target will be filled with the multiple copies of the kernel crash dump?
[ 1 kernel crash dump of the kernel X and 'n' kernel crash dump of the kernel Y]
[0x2] The system will reboot in the loop?
It is hard to define, for original kdump service without early kdump load, it is also possible after one replaced a kernel the new kernel panics during boot phase just after kdump get loaded.
Admin at least should test a reboot after updating a kernel?
Agree, but not sure if all the admins will follow this rule :)
Assumption: 2
What if the dump target in on the local disk?
[0x1] Admin needs to power off the system manually to retrieve the kernel crash dump of the kernel X and Y from the resume mode. [0x2] Admin needs to remove the multiple copies of the kernel crash dump of the booting kernel Y. [0x3] Admin might get confused while differentiating between the kernel crash dump of the kernel X and Kernel Y.
As for the worst case we all admit this is a problem, we are more than happy to make admin be easier and fix it :)
As I said we are exporting about this problem see if we have a good solution.
But if we can assume predictable panic can be avoid then the situation will be better.
One suggestion, can we have a separate default behavior for normal kdump and early kdump?
Yes if we can.
Kdump service start either early or late we just use a syscall to load another kernel/initrd into pre-reserved memory.
So once kdump service started we can not differenciate kdump kernel was loaded early or late.
But still not sure about it. As we can see for normal kdump there will be similar issue existed, eg. between C and D if a reproducible panic also happens every time during late boot phase
So it is hard to define this is early kdump only, just more likely for early kdump.
---A initramfs kdump load switch root other services startup ---B kdump service start ---C other services start up ... ---D boot finished
If we consider this as a early kdump only/must-fix issue, thinking about it we should split into two issues:
- how to determine early load and then reload while kdumpctl start
If "kdump" service is not active but kdump kernel loaded then it should have been "early loaded". Then something will like this:
Thanks for your suggestion.
If we executed the command: systemctl restart kdump.service and then start kdump.service again, it won't exactly distinguish early kdump and normal kdump.
I'm also looking for other solutions.
kdumpctl start() if kdump is loaded: if systemd kdump.service is active # early loaded stop and continue to load again else print a warning service is already running and then return else go ahead to load and start
- how to set the reboot action for early and late loading:
use an cmdline like rd.earlykdump.noreboot, default value is true, if one want to use alternate he/she can add rd.earlykdump.noreboot=0
Add extra config optioins is also a choice but seems much complicated, not only final action, also default action, for default action in case partially saved vmcore it still will occupy the whole disk. So seems we just should replace reboot with shutdown for any cases.
Eg:
In /etc/kdump.conf file. default reboot (Default action to be taken by normal kdump and early kdump if dumping fails) ndefault reboot (Default action to be taken once the dumping is successful by normal kdump) edefault poweroff (Default action to be taken once the dumping is successful by early kdump)
Note: Create 'ndefault' & 'edefault' options identical to 'default' option so that an admin can alter the behavior as per the requirement.
[0x1] If kernel crash dump is captured by normal kdump then 'reboot' the system (by default) or take action as per 'ndefault' value. [0x2] If kernel crash dump is captured by early kdump then 'poweroff' the system 9by default) or take action as per 'edefault' value.
We can introduce something so that early kdump service load and set poweroff as default, but when normal kdump service start we have two choices:
- go ahead to use the early loaded setup without reload like we have in
the code now, in this case normal kdump will also use poweroff no matter what is set in /etc/kdump.conf
- reload kernel/initrd with normal setup in /etc/kdump.conf. In this
way, we need get if this is a late service startup. Because if we blindly reload then it will affect later udev triggered kdump restart.
For example manually start the service like kdumpctl start, origianlly it will just print a msg about the service is already running. But now the "start" == "stop, then start", this may be not a big problem but looks odd.
Correct my self about the udev hotplug triggered events, it should be ok because it will call restart that means stop then start, this change will not affects it.
BTW, the default action here should be "final action", there is a "default" action can be configured in kdump.conf which is used for kdump kernel to do after vmcore saving failed, it is more like a failsafe action. But the default "default" is also "reboot". When kdump successfully saves a vmcore it will go to "final action" (==reboot) which is not configurable.
-- Buland
Thanks Dave
On 12/25/18 at 10:24am, lijiang wrote:
在 2018年12月24日 12:17, Dave Young 写道:
On 12/21/18 at 07:38pm, Dave Young wrote:
On 12/21/18 at 04:47pm, Buland Singh wrote:
On 12/21/18 4:14 PM, Dave Young wrote:
On 12/21/18 at 12:59pm, Buland Singh wrote:
On 12/21/18 12:29 PM, Kairui Song wrote: > Hi, Dave, Lianbo > > My concern is that crash loop may generate tons of dump cores, and the > dump target may get filled up by dump cores, > that may have larger potential risk. Else I think it's good to leave > it as it is. > > On Fri, Dec 21, 2018 at 2:05 PM lijiang lijiang@redhat.com wrote: >> >> 在 2018年12月21日 10:49, Dave Young 写道: >>> + more people >>> On 12/20/18 at 04:49pm, lijiang wrote: >>>> 在 2018年12月20日 13:57, Dave Young 写道: >>>>> On 12/20/18 at 01:06pm, Lianbo Jiang wrote: >>>>>> By default, early kdump reboots the system after capturing the vmcore. >>>>>> If the problematic system is continuously crashing due to some issue >>>>>> during early boot stage, the system may fall into infinite loop restart >>>>>> like this: >>>>>> >>>>>> boot -----> crash -----> early kdump (dump vmcore) >>>>>> ^ | >>>>>> '.........(reboot).............' >>>>>> >>>>>> But now, the system crash at early stage is only captured by early kdump, >>>>>> and the rest is captured by normal kdump. That to say, when normal kdump >>>>>> service starts, it will load it again and override early kdump. It is >>>>>> helpful to control the logic of early kdump and normal kdump separately >>>>>> in final action(it is called by kdump-capture.service). For example, >>>>>> early kdump always passes the 'rd.earlykdump' to the second kernel when >>>>>> early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' >>>>>> to the second kernel at any time. So they can be distinguished in the >>>>>> second kernel. >>>>> >>>>> Hmm, I'm confused about the param passing above. >>>>> >>>> >>>> I copy some messages from another email, please refer to this one: >>>> [---> >>>> The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump >>>> and normal kdump can get the same parameters from /proc/cmdline in the first kernel. >>>> >>>> Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't >>>> need it, normal kdump needs to remove the rd.earlykdump. >>>> >>>> So which can distinguish early kdump and normal kdump in the second kernel. It helps >>>> to control the logic of kdump capture service. For example: default action/final action. >>>> ] >>> >>> The description is confusing, "ealy kdump passes ... to the second >>> kernel", for example about this, the real thing is one person adds the >>> param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd >>> kernel. >>> >> >> Yes. Good point. Thanks for your explanation. >> >>> Anyway this is patch log issue. >>> >>>> >>>>> Early or non early just means about the service loading phase, in >>>> >>>> Yes. This patch used the same method what you said. When normal kdump service starts, >>>> it will reload. At the same time, early kdump will be overwritten by normal kdump. >>> >>> Probably "early kdump load" is better than "early kdump" in words. >>> >>>> >>>>> initramfs or not, I notice dracut/systemd will print some message about >>>>> they are running in initramfs, so probably you can check how to get it >>>>> with same way, if this is not initramfs then just unload before the >>>>> check in kdump loading. >>>>> >>>>> The picture like below: >>>>> >>>>> Kernel boot -> >>>>> >>>>> initramfs --- >>>>> early kdump load >>>>> ---- Mark A ---- >>>>> initramfs switch root >>>>> >>>>> system startup (real root fs) >>>>> service a >>>>> service b ... (eg. networking etc.) >>>>> kdump service start >>>>> -----Mark B ----- >>>>> load kdump kernel again >>>>> >>>>> >>>>> The problem will happen between Mark A and Mark B, during this period, >>>>> there could be repeated crash -> earlykdump_load, there might be some >>>>> random crash as well since during the real root fs service startup, >>>>> for example after network is ready if some network workload cause a >>>>> panic, it maybe not 100% reproducible, so it seems we still need to >>>>> make the poweroff configurable. eg. >>>>> >>>>> default is poweroff, but one can choose if he can. >>>>> >>>> >>>> Yes, default is poweroff for early kdump. Unless kdump capture service >>>> happens error or enters the emergency service, one can choose the default >>>> action.(configure default=xxx in kdump.conf) >>> >>> For default action instead of final action if you hardcode it, then even if >>> one set default as reboot it still poweroff. >>> >> >> If really need, that can be improved. >> >>> [snip] >>> >>>>>> >>>>>> +check_rd_earlykdump() >>>>>> +{ >>>>>> + egrep "rd.earlykdump" /proc/cmdline >>>>>> +} >>>>>> + >>>>>> start() >>>>>> { >>>>>> check_dump_feasibility >>>>>> @@ -969,7 +974,13 @@ start() >>>>>> check_current_status >>>>>> if [ $? == 0 ]; then >>>>>> echo "Kdump already running: [WARNING]" >>>>>> - return 0 >>>>>> + check_rd_earlykdump >>>>>> + #if earlykdump loaded, it will stop and start. >>>>>> + if [ $? -eq 0 ]; then >>>>>> + stop >>>>> >>>>> kdumpctl start can run not only by system startup services, one can also >>>>> run it manually or in udev rule. >>>>> >>>>> The checking of kernel cmdline seems not enough. >>>>> >>>> >>>> Here it means that if kdump has beend loaded, check whether early kdump did it. >>>> If yes, let normal kdump load it again, otherwise no need to do anything. >>> >>> As we discussed you do not get my points here :) >>> >>> check_rd_earlykdump will be always true once kernel bootup, so there is >>> no way to get the first time of normal kdump load and the later one. >>> >>> The early boot time panic to address for this patch is the 100% >>> reproducible panic, for this kinds of panic admin should be able to see >>> it when he boot the machine. So rethinking about this the best way may >>> be just a wontfix. >>> >>> Let's consider below use cases: >>> >>> * First install: >>> >>> install os -> >>> ---A >>> reboot -> >>> ... >>> ---B >>> kdump service start >>> -> create kdump initrd >>> ... >>> boot finished >>> recreate default initrd and enable early kdump >>> goto A >>> >>> Panic happened between A and B, if it is predictable, eg. 100% >>> reproducible, then admin should already see it, then he/she can control >>> and stop the repeating crash/kdump loop >>> if the panic is not 100% reproducible then use reboot as final action is just fine >>> >>> * Other use cases eg. updating kernel or some other components: >>> It is similar with the intall os use case because if one update kernel >>> or critical components it is likely they need regenerate kdump initrd, >>> and then repack it into early kdump default initrd, in this case admin >>> should be able to see the panic loop and handle it. >>> Also if the panic is not 100% reproducible then we are just fine. >>> >>> If we choose to split early and late kdump load, there could be other >>> side effects, and make the logic even complicated. >>> >>> So... the better way may be just leave it as is, and maybe add some >>> documentation. >>> >> >> It is a good way to document the risks that may exist. >> >> Just like the public transportation, we all know that it has the risk, but we still choose it. >> >> Thanks. >> >>> Thoughts? >>> >>> Thanks >>> Dave >>> >
Hello Dave et al.
Kindly check the below condition, assumption and share your thoughts.
Buland, thank you for the reply.
Condition:
[0x1] System is up and running with the kernel version X. [0x2] Admin performed the kernel Y upgrade. [0x3] Running kernel X crashed. [0x4] Normal kdump rebooted the system and captured the kernel crash dump of the kernel X. [0x5] System rebooted with the newly installed kernel Y. [0x6] Let's assume that due to some unknown reason the booting kernel Y also crashed (assume that the panic is 100% reproducible). [0x7] Early kdump started dumping the kernel crash dump of the booting kernel Y. [0x8] Early kdump rebooted the system and stuck in the loop.
Assumption: 1
What if the problematic system is in data center and admin is not aware of this situation?
[0x1] The dump target will be filled with the multiple copies of the kernel crash dump?
[ 1 kernel crash dump of the kernel X and 'n' kernel crash dump of the kernel Y]
[0x2] The system will reboot in the loop?
It is hard to define, for original kdump service without early kdump load, it is also possible after one replaced a kernel the new kernel panics during boot phase just after kdump get loaded.
Admin at least should test a reboot after updating a kernel?
Agree, but not sure if all the admins will follow this rule :)
Assumption: 2
What if the dump target in on the local disk?
[0x1] Admin needs to power off the system manually to retrieve the kernel crash dump of the kernel X and Y from the resume mode. [0x2] Admin needs to remove the multiple copies of the kernel crash dump of the booting kernel Y. [0x3] Admin might get confused while differentiating between the kernel crash dump of the kernel X and Kernel Y.
As for the worst case we all admit this is a problem, we are more than happy to make admin be easier and fix it :)
As I said we are exporting about this problem see if we have a good solution.
But if we can assume predictable panic can be avoid then the situation will be better.
One suggestion, can we have a separate default behavior for normal kdump and early kdump?
Yes if we can.
Kdump service start either early or late we just use a syscall to load another kernel/initrd into pre-reserved memory.
So once kdump service started we can not differenciate kdump kernel was loaded early or late.
But still not sure about it. As we can see for normal kdump there will be similar issue existed, eg. between C and D if a reproducible panic also happens every time during late boot phase
So it is hard to define this is early kdump only, just more likely for early kdump.
---A initramfs kdump load switch root other services startup ---B kdump service start ---C other services start up ... ---D boot finished
If we consider this as a early kdump only/must-fix issue, thinking about it we should split into two issues:
- how to determine early load and then reload while kdumpctl start
If "kdump" service is not active but kdump kernel loaded then it should have been "early loaded". Then something will like this:
Thanks for your suggestion.
If we executed the command: systemctl restart kdump.service and then start kdump.service again, it won't exactly distinguish early kdump and normal kdump.
I'm also looking for other solutions.
As we talked in meeting, I means "is not active" below pseudo code missed a "!"
It should be good if no better way.
kdumpctl start() if kdump is loaded: if systemd kdump.service is active # early loaded stop and continue to load again else print a warning service is already running and then return else go ahead to load and start
- how to set the reboot action for early and late loading:
use an cmdline like rd.earlykdump.noreboot, default value is true, if one want to use alternate he/she can add rd.earlykdump.noreboot=0
Add extra config optioins is also a choice but seems much complicated, not only final action, also default action, for default action in case partially saved vmcore it still will occupy the whole disk. So seems we just should replace reboot with shutdown for any cases.
Eg:
In /etc/kdump.conf file. default reboot (Default action to be taken by normal kdump and early kdump if dumping fails) ndefault reboot (Default action to be taken once the dumping is successful by normal kdump) edefault poweroff (Default action to be taken once the dumping is successful by early kdump)
Note: Create 'ndefault' & 'edefault' options identical to 'default' option so that an admin can alter the behavior as per the requirement.
[0x1] If kernel crash dump is captured by normal kdump then 'reboot' the system (by default) or take action as per 'ndefault' value. [0x2] If kernel crash dump is captured by early kdump then 'poweroff' the system 9by default) or take action as per 'edefault' value.
We can introduce something so that early kdump service load and set poweroff as default, but when normal kdump service start we have two choices:
- go ahead to use the early loaded setup without reload like we have in
the code now, in this case normal kdump will also use poweroff no matter what is set in /etc/kdump.conf
- reload kernel/initrd with normal setup in /etc/kdump.conf. In this
way, we need get if this is a late service startup. Because if we blindly reload then it will affect later udev triggered kdump restart.
For example manually start the service like kdumpctl start, origianlly it will just print a msg about the service is already running. But now the "start" == "stop, then start", this may be not a big problem but looks odd.
Correct my self about the udev hotplug triggered events, it should be ok because it will call restart that means stop then start, this change will not affects it.
BTW, the default action here should be "final action", there is a "default" action can be configured in kdump.conf which is used for kdump kernel to do after vmcore saving failed, it is more like a failsafe action. But the default "default" is also "reboot". When kdump successfully saves a vmcore it will go to "final action" (==reboot) which is not configurable.
-- Buland
Thanks Dave
在 2018年12月26日 09:17, Dave Young 写道:
On 12/25/18 at 10:24am, lijiang wrote:
在 2018年12月24日 12:17, Dave Young 写道:
On 12/21/18 at 07:38pm, Dave Young wrote:
On 12/21/18 at 04:47pm, Buland Singh wrote:
On 12/21/18 4:14 PM, Dave Young wrote:
On 12/21/18 at 12:59pm, Buland Singh wrote: > On 12/21/18 12:29 PM, Kairui Song wrote: >> Hi, Dave, Lianbo >> >> My concern is that crash loop may generate tons of dump cores, and the >> dump target may get filled up by dump cores, >> that may have larger potential risk. Else I think it's good to leave >> it as it is. >> >> On Fri, Dec 21, 2018 at 2:05 PM lijiang lijiang@redhat.com wrote: >>> >>> 在 2018年12月21日 10:49, Dave Young 写道: >>>> + more people >>>> On 12/20/18 at 04:49pm, lijiang wrote: >>>>> 在 2018年12月20日 13:57, Dave Young 写道: >>>>>> On 12/20/18 at 01:06pm, Lianbo Jiang wrote: >>>>>>> By default, early kdump reboots the system after capturing the vmcore. >>>>>>> If the problematic system is continuously crashing due to some issue >>>>>>> during early boot stage, the system may fall into infinite loop restart >>>>>>> like this: >>>>>>> >>>>>>> boot -----> crash -----> early kdump (dump vmcore) >>>>>>> ^ | >>>>>>> '.........(reboot).............' >>>>>>> >>>>>>> But now, the system crash at early stage is only captured by early kdump, >>>>>>> and the rest is captured by normal kdump. That to say, when normal kdump >>>>>>> service starts, it will load it again and override early kdump. It is >>>>>>> helpful to control the logic of early kdump and normal kdump separately >>>>>>> in final action(it is called by kdump-capture.service). For example, >>>>>>> early kdump always passes the 'rd.earlykdump' to the second kernel when >>>>>>> early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' >>>>>>> to the second kernel at any time. So they can be distinguished in the >>>>>>> second kernel. >>>>>> >>>>>> Hmm, I'm confused about the param passing above. >>>>>> >>>>> >>>>> I copy some messages from another email, please refer to this one: >>>>> [---> >>>>> The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump >>>>> and normal kdump can get the same parameters from /proc/cmdline in the first kernel. >>>>> >>>>> Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't >>>>> need it, normal kdump needs to remove the rd.earlykdump. >>>>> >>>>> So which can distinguish early kdump and normal kdump in the second kernel. It helps >>>>> to control the logic of kdump capture service. For example: default action/final action. >>>>> ] >>>> >>>> The description is confusing, "ealy kdump passes ... to the second >>>> kernel", for example about this, the real thing is one person adds the >>>> param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd >>>> kernel. >>>> >>> >>> Yes. Good point. Thanks for your explanation. >>> >>>> Anyway this is patch log issue. >>>> >>>>> >>>>>> Early or non early just means about the service loading phase, in >>>>> >>>>> Yes. This patch used the same method what you said. When normal kdump service starts, >>>>> it will reload. At the same time, early kdump will be overwritten by normal kdump. >>>> >>>> Probably "early kdump load" is better than "early kdump" in words. >>>> >>>>> >>>>>> initramfs or not, I notice dracut/systemd will print some message about >>>>>> they are running in initramfs, so probably you can check how to get it >>>>>> with same way, if this is not initramfs then just unload before the >>>>>> check in kdump loading. >>>>>> >>>>>> The picture like below: >>>>>> >>>>>> Kernel boot -> >>>>>> >>>>>> initramfs --- >>>>>> early kdump load >>>>>> ---- Mark A ---- >>>>>> initramfs switch root >>>>>> >>>>>> system startup (real root fs) >>>>>> service a >>>>>> service b ... (eg. networking etc.) >>>>>> kdump service start >>>>>> -----Mark B ----- >>>>>> load kdump kernel again >>>>>> >>>>>> >>>>>> The problem will happen between Mark A and Mark B, during this period, >>>>>> there could be repeated crash -> earlykdump_load, there might be some >>>>>> random crash as well since during the real root fs service startup, >>>>>> for example after network is ready if some network workload cause a >>>>>> panic, it maybe not 100% reproducible, so it seems we still need to >>>>>> make the poweroff configurable. eg. >>>>>> >>>>>> default is poweroff, but one can choose if he can. >>>>>> >>>>> >>>>> Yes, default is poweroff for early kdump. Unless kdump capture service >>>>> happens error or enters the emergency service, one can choose the default >>>>> action.(configure default=xxx in kdump.conf) >>>> >>>> For default action instead of final action if you hardcode it, then even if >>>> one set default as reboot it still poweroff. >>>> >>> >>> If really need, that can be improved. >>> >>>> [snip] >>>> >>>>>>> >>>>>>> +check_rd_earlykdump() >>>>>>> +{ >>>>>>> + egrep "rd.earlykdump" /proc/cmdline >>>>>>> +} >>>>>>> + >>>>>>> start() >>>>>>> { >>>>>>> check_dump_feasibility >>>>>>> @@ -969,7 +974,13 @@ start() >>>>>>> check_current_status >>>>>>> if [ $? == 0 ]; then >>>>>>> echo "Kdump already running: [WARNING]" >>>>>>> - return 0 >>>>>>> + check_rd_earlykdump >>>>>>> + #if earlykdump loaded, it will stop and start. >>>>>>> + if [ $? -eq 0 ]; then >>>>>>> + stop >>>>>> >>>>>> kdumpctl start can run not only by system startup services, one can also >>>>>> run it manually or in udev rule. >>>>>> >>>>>> The checking of kernel cmdline seems not enough. >>>>>> >>>>> >>>>> Here it means that if kdump has beend loaded, check whether early kdump did it. >>>>> If yes, let normal kdump load it again, otherwise no need to do anything. >>>> >>>> As we discussed you do not get my points here :) >>>> >>>> check_rd_earlykdump will be always true once kernel bootup, so there is >>>> no way to get the first time of normal kdump load and the later one. >>>> >>>> The early boot time panic to address for this patch is the 100% >>>> reproducible panic, for this kinds of panic admin should be able to see >>>> it when he boot the machine. So rethinking about this the best way may >>>> be just a wontfix. >>>> >>>> Let's consider below use cases: >>>> >>>> * First install: >>>> >>>> install os -> >>>> ---A >>>> reboot -> >>>> ... >>>> ---B >>>> kdump service start >>>> -> create kdump initrd >>>> ... >>>> boot finished >>>> recreate default initrd and enable early kdump >>>> goto A >>>> >>>> Panic happened between A and B, if it is predictable, eg. 100% >>>> reproducible, then admin should already see it, then he/she can control >>>> and stop the repeating crash/kdump loop >>>> if the panic is not 100% reproducible then use reboot as final action is just fine >>>> >>>> * Other use cases eg. updating kernel or some other components: >>>> It is similar with the intall os use case because if one update kernel >>>> or critical components it is likely they need regenerate kdump initrd, >>>> and then repack it into early kdump default initrd, in this case admin >>>> should be able to see the panic loop and handle it. >>>> Also if the panic is not 100% reproducible then we are just fine. >>>> >>>> If we choose to split early and late kdump load, there could be other >>>> side effects, and make the logic even complicated. >>>> >>>> So... the better way may be just leave it as is, and maybe add some >>>> documentation. >>>> >>> >>> It is a good way to document the risks that may exist. >>> >>> Just like the public transportation, we all know that it has the risk, but we still choose it. >>> >>> Thanks. >>> >>>> Thoughts? >>>> >>>> Thanks >>>> Dave >>>> >> > > Hello Dave et al. > > Kindly check the below condition, assumption and share your thoughts.
Buland, thank you for the reply.
> > Condition: > > [0x1] System is up and running with the kernel version X. > [0x2] Admin performed the kernel Y upgrade. > [0x3] Running kernel X crashed. > [0x4] Normal kdump rebooted the system and captured the kernel crash dump of the kernel X. > [0x5] System rebooted with the newly installed kernel Y. > [0x6] Let's assume that due to some unknown reason the booting kernel Y also crashed (assume that the panic is 100% reproducible). > [0x7] Early kdump started dumping the kernel crash dump of the booting kernel Y. > [0x8] Early kdump rebooted the system and stuck in the loop. > > Assumption: 1 > > What if the problematic system is in data center and admin is not aware of this situation? > > [0x1] The dump target will be filled with the multiple copies of the kernel crash dump? > > [ 1 kernel crash dump of the kernel X and 'n' kernel crash dump of the kernel Y] > > [0x2] The system will reboot in the loop?
It is hard to define, for original kdump service without early kdump load, it is also possible after one replaced a kernel the new kernel panics during boot phase just after kdump get loaded.
Admin at least should test a reboot after updating a kernel?
Agree, but not sure if all the admins will follow this rule :)
> Assumption: 2 > > What if the dump target in on the local disk? > > [0x1] Admin needs to power off the system manually to retrieve the kernel crash dump of the kernel X and Y from the resume mode. > [0x2] Admin needs to remove the multiple copies of the kernel crash dump of the booting kernel Y. > [0x3] Admin might get confused while differentiating between the kernel crash dump of the kernel X and Kernel Y.
As for the worst case we all admit this is a problem, we are more than happy to make admin be easier and fix it :)
As I said we are exporting about this problem see if we have a good solution.
But if we can assume predictable panic can be avoid then the situation will be better.
One suggestion, can we have a separate default behavior for normal kdump and early kdump?
Yes if we can.
Kdump service start either early or late we just use a syscall to load another kernel/initrd into pre-reserved memory.
So once kdump service started we can not differenciate kdump kernel was loaded early or late.
But still not sure about it. As we can see for normal kdump there will be similar issue existed, eg. between C and D if a reproducible panic also happens every time during late boot phase
So it is hard to define this is early kdump only, just more likely for early kdump.
---A initramfs kdump load switch root other services startup ---B kdump service start ---C other services start up ... ---D boot finished
If we consider this as a early kdump only/must-fix issue, thinking about it we should split into two issues:
- how to determine early load and then reload while kdumpctl start
If "kdump" service is not active but kdump kernel loaded then it should have been "early loaded". Then something will like this:
Thanks for your suggestion.
If we executed the command: systemctl restart kdump.service and then start kdump.service again, it won't exactly distinguish early kdump and normal kdump.
I'm also looking for other solutions.
As we talked in meeting, I means "is not active" below pseudo code missed a "!"
It should be good if no better way.
Previously, early kdump was called by the dracut cmdline hook, systemd doesn't know early kdump's status. The call trace is like this: systemd -> dracut cmdline hook -> early kdump.
Now, i plan to do some tests, and let systemd directly call early kdump like this: systemd -> early kdump
systemd-journal.socket systemd-journal.socket | | v v dracut-cmdline.service ->early kdump ------> early kdump | | v v dracut-pre-udev.service dracut-cmdline.service | | v v systemd-udevd.service dracut-pre-udev.service ...... | v systemd-udevd.service ...... So systemd can know early kdump's status, when normal kdump starts, we can check early kdump's status by systemctl status earlykdump.
Above two solutions, i still need to do some test.
Thanks.
kdumpctl start() if kdump is loaded: if systemd kdump.service is active # early loaded stop and continue to load again else print a warning service is already running and then return else go ahead to load and start
- how to set the reboot action for early and late loading:
use an cmdline like rd.earlykdump.noreboot, default value is true, if one want to use alternate he/she can add rd.earlykdump.noreboot=0
Add extra config optioins is also a choice but seems much complicated, not only final action, also default action, for default action in case partially saved vmcore it still will occupy the whole disk. So seems we just should replace reboot with shutdown for any cases.
Eg:
In /etc/kdump.conf file. default reboot (Default action to be taken by normal kdump and early kdump if dumping fails) ndefault reboot (Default action to be taken once the dumping is successful by normal kdump) edefault poweroff (Default action to be taken once the dumping is successful by early kdump)
Note: Create 'ndefault' & 'edefault' options identical to 'default' option so that an admin can alter the behavior as per the requirement.
[0x1] If kernel crash dump is captured by normal kdump then 'reboot' the system (by default) or take action as per 'ndefault' value. [0x2] If kernel crash dump is captured by early kdump then 'poweroff' the system 9by default) or take action as per 'edefault' value.
We can introduce something so that early kdump service load and set poweroff as default, but when normal kdump service start we have two choices:
- go ahead to use the early loaded setup without reload like we have in
the code now, in this case normal kdump will also use poweroff no matter what is set in /etc/kdump.conf
- reload kernel/initrd with normal setup in /etc/kdump.conf. In this
way, we need get if this is a late service startup. Because if we blindly reload then it will affect later udev triggered kdump restart.
For example manually start the service like kdumpctl start, origianlly it will just print a msg about the service is already running. But now the "start" == "stop, then start", this may be not a big problem but looks odd.
Correct my self about the udev hotplug triggered events, it should be ok because it will call restart that means stop then start, this change will not affects it.
BTW, the default action here should be "final action", there is a "default" action can be configured in kdump.conf which is used for kdump kernel to do after vmcore saving failed, it is more like a failsafe action. But the default "default" is also "reboot". When kdump successfully saves a vmcore it will go to "final action" (==reboot) which is not configurable.
-- Buland
Thanks Dave
On 12/26/18 at 10:01am, lijiang wrote:
在 2018年12月26日 09:17, Dave Young 写道:
On 12/25/18 at 10:24am, lijiang wrote:
在 2018年12月24日 12:17, Dave Young 写道:
On 12/21/18 at 07:38pm, Dave Young wrote:
On 12/21/18 at 04:47pm, Buland Singh wrote:
On 12/21/18 4:14 PM, Dave Young wrote: > On 12/21/18 at 12:59pm, Buland Singh wrote: >> On 12/21/18 12:29 PM, Kairui Song wrote: >>> Hi, Dave, Lianbo >>> >>> My concern is that crash loop may generate tons of dump cores, and the >>> dump target may get filled up by dump cores, >>> that may have larger potential risk. Else I think it's good to leave >>> it as it is. >>> >>> On Fri, Dec 21, 2018 at 2:05 PM lijiang lijiang@redhat.com wrote: >>>> >>>> 在 2018年12月21日 10:49, Dave Young 写道: >>>>> + more people >>>>> On 12/20/18 at 04:49pm, lijiang wrote: >>>>>> 在 2018年12月20日 13:57, Dave Young 写道: >>>>>>> On 12/20/18 at 01:06pm, Lianbo Jiang wrote: >>>>>>>> By default, early kdump reboots the system after capturing the vmcore. >>>>>>>> If the problematic system is continuously crashing due to some issue >>>>>>>> during early boot stage, the system may fall into infinite loop restart >>>>>>>> like this: >>>>>>>> >>>>>>>> boot -----> crash -----> early kdump (dump vmcore) >>>>>>>> ^ | >>>>>>>> '.........(reboot).............' >>>>>>>> >>>>>>>> But now, the system crash at early stage is only captured by early kdump, >>>>>>>> and the rest is captured by normal kdump. That to say, when normal kdump >>>>>>>> service starts, it will load it again and override early kdump. It is >>>>>>>> helpful to control the logic of early kdump and normal kdump separately >>>>>>>> in final action(it is called by kdump-capture.service). For example, >>>>>>>> early kdump always passes the 'rd.earlykdump' to the second kernel when >>>>>>>> early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' >>>>>>>> to the second kernel at any time. So they can be distinguished in the >>>>>>>> second kernel. >>>>>>> >>>>>>> Hmm, I'm confused about the param passing above. >>>>>>> >>>>>> >>>>>> I copy some messages from another email, please refer to this one: >>>>>> [---> >>>>>> The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump >>>>>> and normal kdump can get the same parameters from /proc/cmdline in the first kernel. >>>>>> >>>>>> Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't >>>>>> need it, normal kdump needs to remove the rd.earlykdump. >>>>>> >>>>>> So which can distinguish early kdump and normal kdump in the second kernel. It helps >>>>>> to control the logic of kdump capture service. For example: default action/final action. >>>>>> ] >>>>> >>>>> The description is confusing, "ealy kdump passes ... to the second >>>>> kernel", for example about this, the real thing is one person adds the >>>>> param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd >>>>> kernel. >>>>> >>>> >>>> Yes. Good point. Thanks for your explanation. >>>> >>>>> Anyway this is patch log issue. >>>>> >>>>>> >>>>>>> Early or non early just means about the service loading phase, in >>>>>> >>>>>> Yes. This patch used the same method what you said. When normal kdump service starts, >>>>>> it will reload. At the same time, early kdump will be overwritten by normal kdump. >>>>> >>>>> Probably "early kdump load" is better than "early kdump" in words. >>>>> >>>>>> >>>>>>> initramfs or not, I notice dracut/systemd will print some message about >>>>>>> they are running in initramfs, so probably you can check how to get it >>>>>>> with same way, if this is not initramfs then just unload before the >>>>>>> check in kdump loading. >>>>>>> >>>>>>> The picture like below: >>>>>>> >>>>>>> Kernel boot -> >>>>>>> >>>>>>> initramfs --- >>>>>>> early kdump load >>>>>>> ---- Mark A ---- >>>>>>> initramfs switch root >>>>>>> >>>>>>> system startup (real root fs) >>>>>>> service a >>>>>>> service b ... (eg. networking etc.) >>>>>>> kdump service start >>>>>>> -----Mark B ----- >>>>>>> load kdump kernel again >>>>>>> >>>>>>> >>>>>>> The problem will happen between Mark A and Mark B, during this period, >>>>>>> there could be repeated crash -> earlykdump_load, there might be some >>>>>>> random crash as well since during the real root fs service startup, >>>>>>> for example after network is ready if some network workload cause a >>>>>>> panic, it maybe not 100% reproducible, so it seems we still need to >>>>>>> make the poweroff configurable. eg. >>>>>>> >>>>>>> default is poweroff, but one can choose if he can. >>>>>>> >>>>>> >>>>>> Yes, default is poweroff for early kdump. Unless kdump capture service >>>>>> happens error or enters the emergency service, one can choose the default >>>>>> action.(configure default=xxx in kdump.conf) >>>>> >>>>> For default action instead of final action if you hardcode it, then even if >>>>> one set default as reboot it still poweroff. >>>>> >>>> >>>> If really need, that can be improved. >>>> >>>>> [snip] >>>>> >>>>>>>> >>>>>>>> +check_rd_earlykdump() >>>>>>>> +{ >>>>>>>> + egrep "rd.earlykdump" /proc/cmdline >>>>>>>> +} >>>>>>>> + >>>>>>>> start() >>>>>>>> { >>>>>>>> check_dump_feasibility >>>>>>>> @@ -969,7 +974,13 @@ start() >>>>>>>> check_current_status >>>>>>>> if [ $? == 0 ]; then >>>>>>>> echo "Kdump already running: [WARNING]" >>>>>>>> - return 0 >>>>>>>> + check_rd_earlykdump >>>>>>>> + #if earlykdump loaded, it will stop and start. >>>>>>>> + if [ $? -eq 0 ]; then >>>>>>>> + stop >>>>>>> >>>>>>> kdumpctl start can run not only by system startup services, one can also >>>>>>> run it manually or in udev rule. >>>>>>> >>>>>>> The checking of kernel cmdline seems not enough. >>>>>>> >>>>>> >>>>>> Here it means that if kdump has beend loaded, check whether early kdump did it. >>>>>> If yes, let normal kdump load it again, otherwise no need to do anything. >>>>> >>>>> As we discussed you do not get my points here :) >>>>> >>>>> check_rd_earlykdump will be always true once kernel bootup, so there is >>>>> no way to get the first time of normal kdump load and the later one. >>>>> >>>>> The early boot time panic to address for this patch is the 100% >>>>> reproducible panic, for this kinds of panic admin should be able to see >>>>> it when he boot the machine. So rethinking about this the best way may >>>>> be just a wontfix. >>>>> >>>>> Let's consider below use cases: >>>>> >>>>> * First install: >>>>> >>>>> install os -> >>>>> ---A >>>>> reboot -> >>>>> ... >>>>> ---B >>>>> kdump service start >>>>> -> create kdump initrd >>>>> ... >>>>> boot finished >>>>> recreate default initrd and enable early kdump >>>>> goto A >>>>> >>>>> Panic happened between A and B, if it is predictable, eg. 100% >>>>> reproducible, then admin should already see it, then he/she can control >>>>> and stop the repeating crash/kdump loop >>>>> if the panic is not 100% reproducible then use reboot as final action is just fine >>>>> >>>>> * Other use cases eg. updating kernel or some other components: >>>>> It is similar with the intall os use case because if one update kernel >>>>> or critical components it is likely they need regenerate kdump initrd, >>>>> and then repack it into early kdump default initrd, in this case admin >>>>> should be able to see the panic loop and handle it. >>>>> Also if the panic is not 100% reproducible then we are just fine. >>>>> >>>>> If we choose to split early and late kdump load, there could be other >>>>> side effects, and make the logic even complicated. >>>>> >>>>> So... the better way may be just leave it as is, and maybe add some >>>>> documentation. >>>>> >>>> >>>> It is a good way to document the risks that may exist. >>>> >>>> Just like the public transportation, we all know that it has the risk, but we still choose it. >>>> >>>> Thanks. >>>> >>>>> Thoughts? >>>>> >>>>> Thanks >>>>> Dave >>>>> >>> >> >> Hello Dave et al. >> >> Kindly check the below condition, assumption and share your thoughts. > > Buland, thank you for the reply. > >> >> Condition: >> >> [0x1] System is up and running with the kernel version X. >> [0x2] Admin performed the kernel Y upgrade. >> [0x3] Running kernel X crashed. >> [0x4] Normal kdump rebooted the system and captured the kernel crash dump of the kernel X. >> [0x5] System rebooted with the newly installed kernel Y. >> [0x6] Let's assume that due to some unknown reason the booting kernel Y also crashed (assume that the panic is 100% reproducible). >> [0x7] Early kdump started dumping the kernel crash dump of the booting kernel Y. >> [0x8] Early kdump rebooted the system and stuck in the loop. >> >> Assumption: 1 >> >> What if the problematic system is in data center and admin is not aware of this situation? >> >> [0x1] The dump target will be filled with the multiple copies of the kernel crash dump? >> >> [ 1 kernel crash dump of the kernel X and 'n' kernel crash dump of the kernel Y] >> >> [0x2] The system will reboot in the loop? > > It is hard to define, for original kdump service without early kdump > load, it is also possible after one replaced a kernel the new kernel panics > during boot phase just after kdump get loaded. > > Admin at least should test a reboot after updating a kernel?
Agree, but not sure if all the admins will follow this rule :)
>> Assumption: 2 >> >> What if the dump target in on the local disk? >> >> [0x1] Admin needs to power off the system manually to retrieve the kernel crash dump of the kernel X and Y from the resume mode. >> [0x2] Admin needs to remove the multiple copies of the kernel crash dump of the booting kernel Y. >> [0x3] Admin might get confused while differentiating between the kernel crash dump of the kernel X and Kernel Y. > > As for the worst case we all admit this is a problem, we are more than > happy to make admin be easier and fix it :) > > As I said we are exporting about this problem see if we have a good > solution. > > But if we can assume predictable panic can be avoid then the situation > will be better.
One suggestion, can we have a separate default behavior for normal kdump and early kdump?
Yes if we can.
Kdump service start either early or late we just use a syscall to load another kernel/initrd into pre-reserved memory.
So once kdump service started we can not differenciate kdump kernel was loaded early or late.
But still not sure about it. As we can see for normal kdump there will be similar issue existed, eg. between C and D if a reproducible panic also happens every time during late boot phase
So it is hard to define this is early kdump only, just more likely for early kdump.
---A initramfs kdump load switch root other services startup ---B kdump service start ---C other services start up ... ---D boot finished
If we consider this as a early kdump only/must-fix issue, thinking about it we should split into two issues:
- how to determine early load and then reload while kdumpctl start
If "kdump" service is not active but kdump kernel loaded then it should have been "early loaded". Then something will like this:
Thanks for your suggestion.
If we executed the command: systemctl restart kdump.service and then start kdump.service again, it won't exactly distinguish early kdump and normal kdump.
I'm also looking for other solutions.
As we talked in meeting, I means "is not active" below pseudo code missed a "!"
It should be good if no better way.
Previously, early kdump was called by the dracut cmdline hook, systemd doesn't know early kdump's status. The call trace is like this: systemd -> dracut cmdline hook -> early kdump.
Now, i plan to do some tests, and let systemd directly call early kdump like this: systemd -> early kdump
systemd-journal.socket systemd-journal.socket | | v v dracut-cmdline.service ->early kdump ------> early kdump | | v v dracut-pre-udev.service dracut-cmdline.service | | v v systemd-udevd.service dracut-pre-udev.service ...... | v systemd-udevd.service ......
So systemd can know early kdump's status, when normal kdump starts, we can check early kdump's status by systemctl status earlykdump.
Thanks for exlporing more approach. Just not sure if systemd can get the early (initramfs) service status after switching root, if it is doable then it will be a good way.
Above two solutions, i still need to do some test.
Thanks.
kdumpctl start() if kdump is loaded: if systemd kdump.service is active # early loaded stop and continue to load again else print a warning service is already running and then return else go ahead to load and start
- how to set the reboot action for early and late loading:
use an cmdline like rd.earlykdump.noreboot, default value is true, if one want to use alternate he/she can add rd.earlykdump.noreboot=0
Add extra config optioins is also a choice but seems much complicated, not only final action, also default action, for default action in case partially saved vmcore it still will occupy the whole disk. So seems we just should replace reboot with shutdown for any cases.
Eg:
In /etc/kdump.conf file. default reboot (Default action to be taken by normal kdump and early kdump if dumping fails) ndefault reboot (Default action to be taken once the dumping is successful by normal kdump) edefault poweroff (Default action to be taken once the dumping is successful by early kdump)
Note: Create 'ndefault' & 'edefault' options identical to 'default' option so that an admin can alter the behavior as per the requirement.
[0x1] If kernel crash dump is captured by normal kdump then 'reboot' the system (by default) or take action as per 'ndefault' value. [0x2] If kernel crash dump is captured by early kdump then 'poweroff' the system 9by default) or take action as per 'edefault' value.
We can introduce something so that early kdump service load and set poweroff as default, but when normal kdump service start we have two choices:
- go ahead to use the early loaded setup without reload like we have in
the code now, in this case normal kdump will also use poweroff no matter what is set in /etc/kdump.conf
- reload kernel/initrd with normal setup in /etc/kdump.conf. In this
way, we need get if this is a late service startup. Because if we blindly reload then it will affect later udev triggered kdump restart.
For example manually start the service like kdumpctl start, origianlly it will just print a msg about the service is already running. But now the "start" == "stop, then start", this may be not a big problem but looks odd.
Correct my self about the udev hotplug triggered events, it should be ok because it will call restart that means stop then start, this change will not affects it.
BTW, the default action here should be "final action", there is a "default" action can be configured in kdump.conf which is used for kdump kernel to do after vmcore saving failed, it is more like a failsafe action. But the default "default" is also "reboot". When kdump successfully saves a vmcore it will go to "final action" (==reboot) which is not configurable.
-- Buland
Thanks Dave
On 12/24/18 at 12:17pm, Dave Young wrote:
On 12/21/18 at 07:38pm, Dave Young wrote:
On 12/21/18 at 04:47pm, Buland Singh wrote:
On 12/21/18 4:14 PM, Dave Young wrote:
On 12/21/18 at 12:59pm, Buland Singh wrote:
On 12/21/18 12:29 PM, Kairui Song wrote:
Hi, Dave, Lianbo
My concern is that crash loop may generate tons of dump cores, and the dump target may get filled up by dump cores, that may have larger potential risk. Else I think it's good to leave it as it is.
On Fri, Dec 21, 2018 at 2:05 PM lijiang lijiang@redhat.com wrote: > > 在 2018年12月21日 10:49, Dave Young 写道: > > + more people > > On 12/20/18 at 04:49pm, lijiang wrote: > > > 在 2018年12月20日 13:57, Dave Young 写道: > > > > On 12/20/18 at 01:06pm, Lianbo Jiang wrote: > > > > > By default, early kdump reboots the system after capturing the vmcore. > > > > > If the problematic system is continuously crashing due to some issue > > > > > during early boot stage, the system may fall into infinite loop restart > > > > > like this: > > > > > > > > > > boot -----> crash -----> early kdump (dump vmcore) > > > > > ^ | > > > > > '.........(reboot).............' > > > > > > > > > > But now, the system crash at early stage is only captured by early kdump, > > > > > and the rest is captured by normal kdump. That to say, when normal kdump > > > > > service starts, it will load it again and override early kdump. It is > > > > > helpful to control the logic of early kdump and normal kdump separately > > > > > in final action(it is called by kdump-capture.service). For example, > > > > > early kdump always passes the 'rd.earlykdump' to the second kernel when > > > > > early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' > > > > > to the second kernel at any time. So they can be distinguished in the > > > > > second kernel. > > > > > > > > Hmm, I'm confused about the param passing above. > > > > > > > > > > I copy some messages from another email, please refer to this one: > > > [---> > > > The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump > > > and normal kdump can get the same parameters from /proc/cmdline in the first kernel. > > > > > > Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't > > > need it, normal kdump needs to remove the rd.earlykdump. > > > > > > So which can distinguish early kdump and normal kdump in the second kernel. It helps > > > to control the logic of kdump capture service. For example: default action/final action. > > > ] > > > > The description is confusing, "ealy kdump passes ... to the second > > kernel", for example about this, the real thing is one person adds the > > param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd > > kernel. > > > > Yes. Good point. Thanks for your explanation. > > > Anyway this is patch log issue. > > > > > > > > > Early or non early just means about the service loading phase, in > > > > > > Yes. This patch used the same method what you said. When normal kdump service starts, > > > it will reload. At the same time, early kdump will be overwritten by normal kdump. > > > > Probably "early kdump load" is better than "early kdump" in words. > > > > > > > > > initramfs or not, I notice dracut/systemd will print some message about > > > > they are running in initramfs, so probably you can check how to get it > > > > with same way, if this is not initramfs then just unload before the > > > > check in kdump loading. > > > > > > > > The picture like below: > > > > > > > > Kernel boot -> > > > > > > > > initramfs --- > > > > early kdump load > > > > ---- Mark A ---- > > > > initramfs switch root > > > > > > > > system startup (real root fs) > > > > service a > > > > service b ... (eg. networking etc.) > > > > kdump service start > > > > -----Mark B ----- > > > > load kdump kernel again > > > > > > > > > > > > The problem will happen between Mark A and Mark B, during this period, > > > > there could be repeated crash -> earlykdump_load, there might be some > > > > random crash as well since during the real root fs service startup, > > > > for example after network is ready if some network workload cause a > > > > panic, it maybe not 100% reproducible, so it seems we still need to > > > > make the poweroff configurable. eg. > > > > > > > > default is poweroff, but one can choose if he can. > > > > > > > > > > Yes, default is poweroff for early kdump. Unless kdump capture service > > > happens error or enters the emergency service, one can choose the default > > > action.(configure default=xxx in kdump.conf) > > > > For default action instead of final action if you hardcode it, then even if > > one set default as reboot it still poweroff. > > > > If really need, that can be improved. > > > [snip] > > > > > > > > > > > > +check_rd_earlykdump() > > > > > +{ > > > > > + egrep "rd.earlykdump" /proc/cmdline > > > > > +} > > > > > + > > > > > start() > > > > > { > > > > > check_dump_feasibility > > > > > @@ -969,7 +974,13 @@ start() > > > > > check_current_status > > > > > if [ $? == 0 ]; then > > > > > echo "Kdump already running: [WARNING]" > > > > > - return 0 > > > > > + check_rd_earlykdump > > > > > + #if earlykdump loaded, it will stop and start. > > > > > + if [ $? -eq 0 ]; then > > > > > + stop > > > > > > > > kdumpctl start can run not only by system startup services, one can also > > > > run it manually or in udev rule. > > > > > > > > The checking of kernel cmdline seems not enough. > > > > > > > > > > Here it means that if kdump has beend loaded, check whether early kdump did it. > > > If yes, let normal kdump load it again, otherwise no need to do anything. > > > > As we discussed you do not get my points here :) > > > > check_rd_earlykdump will be always true once kernel bootup, so there is > > no way to get the first time of normal kdump load and the later one. > > > > The early boot time panic to address for this patch is the 100% > > reproducible panic, for this kinds of panic admin should be able to see > > it when he boot the machine. So rethinking about this the best way may > > be just a wontfix. > > > > Let's consider below use cases: > > > > * First install: > > > > install os -> > > ---A > > reboot -> > > ... > > ---B > > kdump service start > > -> create kdump initrd > > ... > > boot finished > > recreate default initrd and enable early kdump > > goto A > > > > Panic happened between A and B, if it is predictable, eg. 100% > > reproducible, then admin should already see it, then he/she can control > > and stop the repeating crash/kdump loop > > if the panic is not 100% reproducible then use reboot as final action is just fine > > > > * Other use cases eg. updating kernel or some other components: > > It is similar with the intall os use case because if one update kernel > > or critical components it is likely they need regenerate kdump initrd, > > and then repack it into early kdump default initrd, in this case admin > > should be able to see the panic loop and handle it. > > Also if the panic is not 100% reproducible then we are just fine. > > > > If we choose to split early and late kdump load, there could be other > > side effects, and make the logic even complicated. > > > > So... the better way may be just leave it as is, and maybe add some > > documentation. > > > > It is a good way to document the risks that may exist. > > Just like the public transportation, we all know that it has the risk, but we still choose it. > > Thanks. > > > Thoughts? > > > > Thanks > > Dave > >
Hello Dave et al.
Kindly check the below condition, assumption and share your thoughts.
Buland, thank you for the reply.
Condition:
[0x1] System is up and running with the kernel version X. [0x2] Admin performed the kernel Y upgrade. [0x3] Running kernel X crashed. [0x4] Normal kdump rebooted the system and captured the kernel crash dump of the kernel X. [0x5] System rebooted with the newly installed kernel Y. [0x6] Let's assume that due to some unknown reason the booting kernel Y also crashed (assume that the panic is 100% reproducible). [0x7] Early kdump started dumping the kernel crash dump of the booting kernel Y. [0x8] Early kdump rebooted the system and stuck in the loop.
Assumption: 1
What if the problematic system is in data center and admin is not aware of this situation?
[0x1] The dump target will be filled with the multiple copies of the kernel crash dump?
[ 1 kernel crash dump of the kernel X and 'n' kernel crash dump of the kernel Y]
[0x2] The system will reboot in the loop?
It is hard to define, for original kdump service without early kdump load, it is also possible after one replaced a kernel the new kernel panics during boot phase just after kdump get loaded.
Admin at least should test a reboot after updating a kernel?
Agree, but not sure if all the admins will follow this rule :)
Assumption: 2
What if the dump target in on the local disk?
[0x1] Admin needs to power off the system manually to retrieve the kernel crash dump of the kernel X and Y from the resume mode. [0x2] Admin needs to remove the multiple copies of the kernel crash dump of the booting kernel Y. [0x3] Admin might get confused while differentiating between the kernel crash dump of the kernel X and Kernel Y.
As for the worst case we all admit this is a problem, we are more than happy to make admin be easier and fix it :)
As I said we are exporting about this problem see if we have a good solution.
But if we can assume predictable panic can be avoid then the situation will be better.
One suggestion, can we have a separate default behavior for normal kdump and early kdump?
Yes if we can.
Kdump service start either early or late we just use a syscall to load another kernel/initrd into pre-reserved memory.
So once kdump service started we can not differenciate kdump kernel was loaded early or late.
But still not sure about it. As we can see for normal kdump there will be similar issue existed, eg. between C and D if a reproducible panic also happens every time during late boot phase
So it is hard to define this is early kdump only, just more likely for early kdump.
---A initramfs kdump load switch root other services startup ---B kdump service start ---C other services start up ... ---D boot finished
If we consider this as a early kdump only/must-fix issue, thinking about it we should split into two issues:
- how to determine early load and then reload while kdumpctl start
If "kdump" service is not active but kdump kernel loaded then it should have been "early loaded". Then something will like this:
kdumpctl start() if kdump is loaded: if systemd kdump.service is active
Typo, I means the inverse logic: if !systemd kdump.service is active
# early loaded stop and continue to load again
else print a warning service is already running and then return else go ahead to load and start
- how to set the reboot action for early and late loading:
use an cmdline like rd.earlykdump.noreboot, default value is true, if one want to use alternate he/she can add rd.earlykdump.noreboot=0
Add extra config optioins is also a choice but seems much complicated, not only final action, also default action, for default action in case partially saved vmcore it still will occupy the whole disk. So seems we just should replace reboot with shutdown for any cases.
Eg:
In /etc/kdump.conf file. default reboot (Default action to be taken by normal kdump and early kdump if dumping fails) ndefault reboot (Default action to be taken once the dumping is successful by normal kdump) edefault poweroff (Default action to be taken once the dumping is successful by early kdump)
Note: Create 'ndefault' & 'edefault' options identical to 'default' option so that an admin can alter the behavior as per the requirement.
[0x1] If kernel crash dump is captured by normal kdump then 'reboot' the system (by default) or take action as per 'ndefault' value. [0x2] If kernel crash dump is captured by early kdump then 'poweroff' the system 9by default) or take action as per 'edefault' value.
We can introduce something so that early kdump service load and set poweroff as default, but when normal kdump service start we have two choices:
- go ahead to use the early loaded setup without reload like we have in
the code now, in this case normal kdump will also use poweroff no matter what is set in /etc/kdump.conf
- reload kernel/initrd with normal setup in /etc/kdump.conf. In this
way, we need get if this is a late service startup. Because if we blindly reload then it will affect later udev triggered kdump restart.
For example manually start the service like kdumpctl start, origianlly it will just print a msg about the service is already running. But now the "start" == "stop, then start", this may be not a big problem but looks odd.
Correct my self about the udev hotplug triggered events, it should be ok because it will call restart that means stop then start, this change will not affects it.
BTW, the default action here should be "final action", there is a "default" action can be configured in kdump.conf which is used for kdump kernel to do after vmcore saving failed, it is more like a failsafe action. But the default "default" is also "reboot". When kdump successfully saves a vmcore it will go to "final action" (==reboot) which is not configurable.
-- Buland
Thanks Dave
Hi,
On 12/23/2018 11:17 PM, Dave Young wrote:
But still not sure about it. As we can see for normal kdump there will be similar issue existed, eg. between C and D if a reproducible panic also happens every time during late boot phase
So it is hard to define this is early kdump only, just more likely for early kdump.
I agree on this. And I guess that having behavior differences between early and normal kdump will make kexec-tools complicated and confusing for users.
If so, having a common 'final_action' option in kdump.conf also might be good. We can recommend users set 'final_action poweroff' for early kdump to avoid crash loop in the early kdump document.
As for the vmcore-filled disk issue, which also occurs with normal kdump and even an one-shot panic, something like 'remove_on_fail' option might be helpful as a fail-safe.
Thanks, Kazu
On 01/04/19 at 04:45pm, Kazuhito Hagio wrote:
Hi,
On 12/23/2018 11:17 PM, Dave Young wrote:
But still not sure about it. As we can see for normal kdump there will be similar issue existed, eg. between C and D if a reproducible panic also happens every time during late boot phase
So it is hard to define this is early kdump only, just more likely for early kdump.
I agree on this. And I guess that having behavior differences between early and normal kdump will make kexec-tools complicated and confusing for users.
I also hesitate to make the kdump logic more complicated, it will become even confusing..
If so, having a common 'final_action' option in kdump.conf also might be good. We can recommend users set 'final_action poweroff' for early kdump to avoid crash loop in the early kdump document.
Good suggestion, it might be the better.
As for the vmcore-filled disk issue, which also occurs with normal kdump and even an one-shot panic, something like 'remove_on_fail' option might be helpful as a fail-safe.
Not very clear about "remove_on_fail", could you elaborate about the suggestion? remove old vmcores or remove partial dumped files?
Thanks, Kazu
Thanks Dave
On 1/6/2019 9:59 PM, Dave Young wrote:
On 01/04/19 at 04:45pm, Kazuhito Hagio wrote:
Hi,
On 12/23/2018 11:17 PM, Dave Young wrote:
But still not sure about it. As we can see for normal kdump there will be similar issue existed, eg. between C and D if a reproducible panic also happens every time during late boot phase
So it is hard to define this is early kdump only, just more likely for early kdump.
I agree on this. And I guess that having behavior differences between early and normal kdump will make kexec-tools complicated and confusing for users.
I also hesitate to make the kdump logic more complicated, it will become even confusing..
Early kdump is very optional (most kdump users rarely use it), so I'm afraid that making the kdump logic more complicated because of it doesn't pay for now.. I think we can implement it when a request or a need to have different actions between early and normal kdump comes.
If so, having a common 'final_action' option in kdump.conf also might be good. We can recommend users set 'final_action poweroff' for early kdump to avoid crash loop in the early kdump document.
Good suggestion, it might be the better.
This would be simple and give a new choice to normal kdump users, too. I'll write a patch for this.
As for the vmcore-filled disk issue, which also occurs with normal kdump and even an one-shot panic, something like 'remove_on_fail' option might be helpful as a fail-safe.
Not very clear about "remove_on_fail", could you elaborate about the suggestion? remove old vmcores or remove partial dumped files?
I mean it removes the partial dumped file when the core_collector fails and the target disk is full. (so 'remove_on_full' is better?) Users can select to keep the failed dump file (maybe useless) or keep some disk space vacant.
or, something like 'disk_margin <size>' option would be safer to keep enough space for user applications. It can remove the vmcore when dump finished and the space of disk is less than the specified margin value.
Ideally, users should prepare a target disk only for kdump separately from system disk and we should recommend them do so, but some of them cannot or don't. So having such an option might be helpful for some users.
Thanks, Kazu
On 01/07/19 at 04:20pm, Kazuhito Hagio wrote:
On 1/6/2019 9:59 PM, Dave Young wrote:
On 01/04/19 at 04:45pm, Kazuhito Hagio wrote:
Hi,
On 12/23/2018 11:17 PM, Dave Young wrote:
But still not sure about it. As we can see for normal kdump there will be similar issue existed, eg. between C and D if a reproducible panic also happens every time during late boot phase
So it is hard to define this is early kdump only, just more likely for early kdump.
I agree on this. And I guess that having behavior differences between early and normal kdump will make kexec-tools complicated and confusing for users.
I also hesitate to make the kdump logic more complicated, it will become even confusing..
Early kdump is very optional (most kdump users rarely use it), so I'm afraid that making the kdump logic more complicated because of it doesn't pay for now.. I think we can implement it when a request or a need to have different actions between early and normal kdump comes.
Agreed.
If so, having a common 'final_action' option in kdump.conf also might be good. We can recommend users set 'final_action poweroff' for early kdump to avoid crash loop in the early kdump document.
Good suggestion, it might be the better.
This would be simple and give a new choice to normal kdump users, too. I'll write a patch for this.
Great, thank you for the effort.
As for the vmcore-filled disk issue, which also occurs with normal kdump and even an one-shot panic, something like 'remove_on_fail' option might be helpful as a fail-safe.
Not very clear about "remove_on_fail", could you elaborate about the suggestion? remove old vmcores or remove partial dumped files?
I mean it removes the partial dumped file when the core_collector fails and the target disk is full. (so 'remove_on_full' is better?) Users can select to keep the failed dump file (maybe useless) or keep some disk space vacant.
or, something like 'disk_margin <size>' option would be safer to keep enough space for user applications. It can remove the vmcore when dump finished and the space of disk is less than the specified margin value.
Ideally, users should prepare a target disk only for kdump separately from system disk and we should recommend them do so, but some of them cannot or don't. So having such an option might be helpful for some users.
Hmm, maybe it should be leaved to admin, using something like kdump_post script to cleanup the disk space
Thanks, Kazu
Thanks Dave
On 1/8/2019 4:46 AM, Dave Young wrote:
On 01/07/19 at 04:20pm, Kazuhito Hagio wrote:
I mean it removes the partial dumped file when the core_collector fails and the target disk is full. (so 'remove_on_full' is better?) Users can select to keep the failed dump file (maybe useless) or keep some disk space vacant.
or, something like 'disk_margin <size>' option would be safer to keep enough space for user applications. It can remove the vmcore when dump finished and the space of disk is less than the specified margin value.
Ideally, users should prepare a target disk only for kdump separately from system disk and we should recommend them do so, but some of them cannot or don't. So having such an option might be helpful for some users.
Hmm, maybe it should be leaved to admin, using something like kdump_post script to cleanup the disk space
OK. I thought it would become easier for admins to do that than using kdump_post, but yes, they can do the same thing with it and there might be not much demand so far.
Thanks for your comments. Kazu
Hi Kairui,
On 12/21/18 at 02:59pm, Kairui Song wrote:
Hi, Dave, Lianbo
My concern is that crash loop may generate tons of dump cores, and the dump target may get filled up by dump cores, that may have larger potential risk. Else I think it's good to leave it as it is.
This is also what we are concerning. But no easy way to handle yet, feel free to provide suggestions if have any.
On Fri, Dec 21, 2018 at 2:05 PM lijiang lijiang@redhat.com wrote:
在 2018年12月21日 10:49, Dave Young 写道:
- more people
On 12/20/18 at 04:49pm, lijiang wrote:
在 2018年12月20日 13:57, Dave Young 写道:
On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
By default, early kdump reboots the system after capturing the vmcore. If the problematic system is continuously crashing due to some issue during early boot stage, the system may fall into infinite loop restart like this:
boot -----> crash -----> early kdump (dump vmcore) ^ | '.........(reboot).............'
But now, the system crash at early stage is only captured by early kdump, and the rest is captured by normal kdump. That to say, when normal kdump service starts, it will load it again and override early kdump. It is helpful to control the logic of early kdump and normal kdump separately in final action(it is called by kdump-capture.service). For example, early kdump always passes the 'rd.earlykdump' to the second kernel when early kdump is enabled, but normal kdump doesn't pass the 'rd.earlykdump' to the second kernel at any time. So they can be distinguished in the second kernel.
Hmm, I'm confused about the param passing above.
I copy some messages from another email, please refer to this one: [---> The rd.earlykdump is added to kernel command line in grub.cfg. However, early kdump and normal kdump can get the same parameters from /proc/cmdline in the first kernel.
Early kdump passes the rd.earlykdump to the second kernel, but normal kdump doesn't need it, normal kdump needs to remove the rd.earlykdump.
So which can distinguish early kdump and normal kdump in the second kernel. It helps to control the logic of kdump capture service. For example: default action/final action. ]
The description is confusing, "ealy kdump passes ... to the second kernel", for example about this, the real thing is one person adds the param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd kernel.
Yes. Good point. Thanks for your explanation.
Anyway this is patch log issue.
Early or non early just means about the service loading phase, in
Yes. This patch used the same method what you said. When normal kdump service starts, it will reload. At the same time, early kdump will be overwritten by normal kdump.
Probably "early kdump load" is better than "early kdump" in words.
initramfs or not, I notice dracut/systemd will print some message about they are running in initramfs, so probably you can check how to get it with same way, if this is not initramfs then just unload before the check in kdump loading.
The picture like below:
Kernel boot ->
initramfs --- early kdump load ---- Mark A ---- initramfs switch root
system startup (real root fs) service a service b ... (eg. networking etc.) kdump service start
-----Mark B ----- load kdump kernel again
The problem will happen between Mark A and Mark B, during this period, there could be repeated crash -> earlykdump_load, there might be some random crash as well since during the real root fs service startup, for example after network is ready if some network workload cause a panic, it maybe not 100% reproducible, so it seems we still need to make the poweroff configurable. eg.
default is poweroff, but one can choose if he can.
Yes, default is poweroff for early kdump. Unless kdump capture service happens error or enters the emergency service, one can choose the default action.(configure default=xxx in kdump.conf)
For default action instead of final action if you hardcode it, then even if one set default as reboot it still poweroff.
If really need, that can be improved.
[snip]
+check_rd_earlykdump() +{
- egrep "rd.earlykdump" /proc/cmdline
+}
start() { check_dump_feasibility @@ -969,7 +974,13 @@ start() check_current_status if [ $? == 0 ]; then echo "Kdump already running: [WARNING]"
return 0
check_rd_earlykdump
#if earlykdump loaded, it will stop and start.
if [ $? -eq 0 ]; then
stop
kdumpctl start can run not only by system startup services, one can also run it manually or in udev rule.
The checking of kernel cmdline seems not enough.
Here it means that if kdump has beend loaded, check whether early kdump did it. If yes, let normal kdump load it again, otherwise no need to do anything.
As we discussed you do not get my points here :)
check_rd_earlykdump will be always true once kernel bootup, so there is no way to get the first time of normal kdump load and the later one.
The early boot time panic to address for this patch is the 100% reproducible panic, for this kinds of panic admin should be able to see it when he boot the machine. So rethinking about this the best way may be just a wontfix.
Let's consider below use cases:
- First install:
install os -> ---A reboot -> ... ---B kdump service start -> create kdump initrd ... boot finished recreate default initrd and enable early kdump goto A
Panic happened between A and B, if it is predictable, eg. 100% reproducible, then admin should already see it, then he/she can control and stop the repeating crash/kdump loop if the panic is not 100% reproducible then use reboot as final action is just fine
- Other use cases eg. updating kernel or some other components:
It is similar with the intall os use case because if one update kernel or critical components it is likely they need regenerate kdump initrd, and then repack it into early kdump default initrd, in this case admin should be able to see the panic loop and handle it. Also if the panic is not 100% reproducible then we are just fine.
If we choose to split early and late kdump load, there could be other side effects, and make the logic even complicated.
So... the better way may be just leave it as is, and maybe add some documentation.
It is a good way to document the risks that may exist.
Just like the public transportation, we all know that it has the risk, but we still choose it.
Thanks.
Thoughts?
Thanks Dave
-- Best Regards, Kairui Song
Thanks Dave