Add recommendations on how much memory is required for FADump. Also, mention the optimizations applied to default initrd when FADump is used and how to workaround it.
Signed-off-by: Hari Bathini hbathini@linux.ibm.com ---
* The 4th point added in "Things to remember" section can be dropped, if the fix proposed in Redhat Bug 1796941 is accepted upstream.
fadump-howto.txt | 86 +++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 78 insertions(+), 8 deletions(-)
diff --git a/fadump-howto.txt b/fadump-howto.txt index 7061f02..89675a1 100644 --- a/fadump-howto.txt +++ b/fadump-howto.txt @@ -84,19 +84,25 @@ kexec-tools installed. If not, you install it via the following command:
# yum install kexec-tools
-To be able to do much of anything interesting in the way of debug analysis, -you'll also need to install the kernel-debuginfo package, of the same arch -as your running kernel, and the crash utility: +Make the kernel to be configured with FADump as the default boot entry, if +it isn't already: + + # grubby --set-default=/boot/vmlinuz-<kver> + +Boot into the kernel to be configured for FADump. To be able to do much of +anything interesting in the way of debug analysis, you'll also need to install +the kernel-debuginfo package, of the same arch as your running kernel, and the +crash utility:
# yum --enablerepo=*debuginfo install kernel-debuginfo.$(uname -m) crash
Next up, we need to modify some boot parameters to enable firmware assisted dump. With the help of grubby, it's very easy to append "fadump=on" to the end -of your kernel boot parameters. Optionally, user can also append -'fadump_reserve_mem=X' kernel cmdline to specify size of the memory to reserve -for boot memory dump preservation. +of your kernel boot parameters. To reserve the appropriate amount of memory +for boot memory preservation, pass 'crashkernel=X' kernel cmdline parameter. +For the recommended value of X, see 'FADump Memory Requirements' section.
- # grubby --args="fadump=on" --update-kernel=/boot/vmlinuz-`uname -r` + # grubby --args="fadump=on crashkernel=6G" --update-kernel=/boot/vmlinuz-`uname -r`
The term 'boot memory' means size of the low memory chunk that is required for a kernel to boot successfully when booted with restricted memory. By default, @@ -145,7 +151,7 @@ perform postmortem analysis: and so on...
Saving vmcore-dmesg.txt ----------------------- +----------------------- Kernel log bufferes are one of the most important information available in vmcore. Now before saving vmcore, kernel log bufferes are extracted from /proc/vmcore and saved into a file vmcore-dmesg.txt. After @@ -153,6 +159,57 @@ vmcore-dmesg.txt, vmcore is saved. Destination disk and directory for vmcore-dmesg.txt is same as vmcore. Note that kernel log buffers will not be available if dump target is raw device.
+FADump Memory Requirements: + + System Memory Recommended memory +--------------------- ---------------------- + 4 GB - 16 GB : 768 MB + 16 GB - 64 GB : 1024 MB + 64 GB - 128 GB : 2 GB + 128 GB - 1 TB : 4 GB + 1 TB - 2 TB : 6 GB + 2 TB - 4 TB : 12 GB + 4 TB - 8 TB : 20 GB + 8 TB - 16 TB : 36 GB + 16 TB - 32 TB : 64 GB + 32 TB - 64 TB : 128 GB + 64 TB & above : 180 GB + +Things to remember: + +1) The memory required to boot capture Kernel is a moving target that depends + on many factors like hardware attached to the system, kernel and modules in + use, packages installed and services enabled, there is no one-size-fits-all. + But the above recommendations are based on system memory. So, the above + recommendations for FADump come with a few assumptions, based on available + system memory, about the resources the system could have. So, please take + the recommendations with a pinch of salt and remember to try capturing dump + a few times to confirm that the system is configured successfully with dump + capturing support. + +2) Though the memory requirements for FADump seem high, this memory is not + completely set aside but made available for userspace applications to use, + through the CMA allocator. + +3) As the same initrd is used for booting production kernel as well as capture + kernel and with dump being captured in a restricted memory environment, few + optimizations (like not inclding network dracut module, disabling multipath + and such) are applied while building the initrd. In case, the production + environment needs these optimizations to be avoided, dracut_args option in + /etc/kdump.conf file could be leveraged. For example, if a user wishes for + network module to be included in the initrd, adding the below entry in + /etc/kdump.conf file and restarting kdump service would take care of it. + + dracut_args --add "network" + +4) If FADump is configured to capture vmcore to a remote dump target using SSH + or NFS protocol, the network interface is renamed to kdump-<interface-name> + if <interface-name> is generic, for example, *eth#, or net#. This problem + occurs because the vmcore capture scripts in the initial RAM disk (initrd) + add the kdump- prefix to the network interface name to secure persistent + naming. As the same initrd is used for production kernel boot, the interface + name is changed for the production kernel too. + Dump Triggering methods:
This section talks about the various ways, other than a Kernel Panic, in which @@ -250,3 +307,16 @@ initramfs for the time being. If you need modify "KDUMP_COMMANDLINE=" in options are copied from /proc/cmdline. In general it is best to append command line options using "KDUMP_COMMANDLINE_APPEND=" instead of replacing the original command line completely. + +How to disable FADump: + +Remove "fadump=on" from kernel cmdline parameters: + + # grubby --update-kernel=/boot/vmlinuz-`uname -r` --remove-args="fadump=on" + +If KDump is to be used as the dump capturing mechanism, update the crashkernel +parameter (Else, remove "crashkernel=" parameter too, using grubby): + + # grubby --update-kernel=/boot/vmlinuz-$kver --args="crashkernl=auto" + +Reboot the system for the settings to take effect.
Thanks for it.
Acked-by: Pingfan Liu piliu@redhat.com
On 03/23/2020 09:28 PM, Hari Bathini wrote:
Add recommendations on how much memory is required for FADump. Also, mention the optimizations applied to default initrd when FADump is used and how to workaround it.
Signed-off-by: Hari Bathini hbathini@linux.ibm.com
- The 4th point added in "Things to remember" section can be dropped, if the fix proposed in Redhat Bug 1796941 is accepted upstream.
fadump-howto.txt | 86 +++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 78 insertions(+), 8 deletions(-)
diff --git a/fadump-howto.txt b/fadump-howto.txt index 7061f02..89675a1 100644 --- a/fadump-howto.txt +++ b/fadump-howto.txt @@ -84,19 +84,25 @@ kexec-tools installed. If not, you install it via the following command:
# yum install kexec-tools
-To be able to do much of anything interesting in the way of debug analysis, -you'll also need to install the kernel-debuginfo package, of the same arch -as your running kernel, and the crash utility: +Make the kernel to be configured with FADump as the default boot entry, if +it isn't already:
- # grubby --set-default=/boot/vmlinuz-<kver>
+Boot into the kernel to be configured for FADump. To be able to do much of +anything interesting in the way of debug analysis, you'll also need to install +the kernel-debuginfo package, of the same arch as your running kernel, and the +crash utility:
# yum --enablerepo=\*debuginfo install kernel-debuginfo.$(uname -m) crash
Next up, we need to modify some boot parameters to enable firmware assisted dump. With the help of grubby, it's very easy to append "fadump=on" to the end -of your kernel boot parameters. Optionally, user can also append -'fadump_reserve_mem=X' kernel cmdline to specify size of the memory to reserve -for boot memory dump preservation. +of your kernel boot parameters. To reserve the appropriate amount of memory +for boot memory preservation, pass 'crashkernel=X' kernel cmdline parameter. +For the recommended value of X, see 'FADump Memory Requirements' section.
- # grubby --args="fadump=on" --update-kernel=/boot/vmlinuz-`uname -r`
- # grubby --args="fadump=on crashkernel=6G" --update-kernel=/boot/vmlinuz-`uname -r`
The term 'boot memory' means size of the low memory chunk that is required for a kernel to boot successfully when booted with restricted memory. By default, @@ -145,7 +151,7 @@ perform postmortem analysis: and so on...
Saving vmcore-dmesg.txt
+----------------------- Kernel log bufferes are one of the most important information available in vmcore. Now before saving vmcore, kernel log bufferes are extracted from /proc/vmcore and saved into a file vmcore-dmesg.txt. After @@ -153,6 +159,57 @@ vmcore-dmesg.txt, vmcore is saved. Destination disk and directory for vmcore-dmesg.txt is same as vmcore. Note that kernel log buffers will not be available if dump target is raw device.
+FADump Memory Requirements:
- System Memory Recommended memory
+--------------------- ----------------------
- 4 GB - 16 GB : 768 MB
- 16 GB - 64 GB : 1024 MB
- 64 GB - 128 GB : 2 GB
- 128 GB - 1 TB : 4 GB
- 1 TB - 2 TB : 6 GB
- 2 TB - 4 TB : 12 GB
- 4 TB - 8 TB : 20 GB
- 8 TB - 16 TB : 36 GB
- 16 TB - 32 TB : 64 GB
- 32 TB - 64 TB : 128 GB
- 64 TB & above : 180 GB
+Things to remember:
+1) The memory required to boot capture Kernel is a moving target that depends
- on many factors like hardware attached to the system, kernel and modules in
- use, packages installed and services enabled, there is no one-size-fits-all.
- But the above recommendations are based on system memory. So, the above
- recommendations for FADump come with a few assumptions, based on available
- system memory, about the resources the system could have. So, please take
- the recommendations with a pinch of salt and remember to try capturing dump
- a few times to confirm that the system is configured successfully with dump
- capturing support.
+2) Though the memory requirements for FADump seem high, this memory is not
- completely set aside but made available for userspace applications to use,
- through the CMA allocator.
+3) As the same initrd is used for booting production kernel as well as capture
- kernel and with dump being captured in a restricted memory environment, few
- optimizations (like not inclding network dracut module, disabling multipath
- and such) are applied while building the initrd. In case, the production
- environment needs these optimizations to be avoided, dracut_args option in
- /etc/kdump.conf file could be leveraged. For example, if a user wishes for
- network module to be included in the initrd, adding the below entry in
- /etc/kdump.conf file and restarting kdump service would take care of it.
- dracut_args --add "network"
+4) If FADump is configured to capture vmcore to a remote dump target using SSH
- or NFS protocol, the network interface is renamed to kdump-<interface-name>
- if <interface-name> is generic, for example, *eth#, or net#. This problem
- occurs because the vmcore capture scripts in the initial RAM disk (initrd)
- add the kdump- prefix to the network interface name to secure persistent
- naming. As the same initrd is used for production kernel boot, the interface
- name is changed for the production kernel too.
Dump Triggering methods:
This section talks about the various ways, other than a Kernel Panic, in which @@ -250,3 +307,16 @@ initramfs for the time being. If you need modify "KDUMP_COMMANDLINE=" in options are copied from /proc/cmdline. In general it is best to append command line options using "KDUMP_COMMANDLINE_APPEND=" instead of replacing the original command line completely.
+How to disable FADump:
+Remove "fadump=on" from kernel cmdline parameters:
- # grubby --update-kernel=/boot/vmlinuz-`uname -r` --remove-args="fadump=on"
+If KDump is to be used as the dump capturing mechanism, update the crashkernel +parameter (Else, remove "crashkernel=" parameter too, using grubby):
- # grubby --update-kernel=/boot/vmlinuz-$kver --args="crashkernl=auto"
+Reboot the system for the settings to take effect.