Some cloud people complained that VM boot speed is slower than Ubuntu and other distributions, after some debugging, we found that one of the causes is kdump service starts too slow(7seconds according to the test result on the test VM), actually there is no "crashkernel=X" specified. Although kdump service is parallel, it affects the speed more or less especially on VMs with few cpus, which is unacceptable. It is even worse when kdump initramfs is built out in case of no reserved memory at first boot.
Commit afa4a35d3 ("kdumpctrl: kdump feasibility should fail if no crash memory") can actually solve this issue.
This patch is a supplement of above-mentioned commit, we bail out start() even earlier in case of no reserved memory.
Also made some cosmatic changes for check_crash_mem_reserved().
1) Before this patch $ time kdumpctl start No memory reserved for crash kernel. Starting kdump: [FAILED]
real 0m0.282s user 0m0.184s sys 0m0.146s
2) After this patch $ time kdumpctl start No memory reserved for crash kernel Starting kdump: [FAILED]
real 0m0.010s user 0m0.008s sys 0m0.001s
Signed-off-by: Xunlei Pang xlpang@redhat.com --- kdumpctl | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/kdumpctl b/kdumpctl index 6b5be09..129cb6a 100755 --- a/kdumpctl +++ b/kdumpctl @@ -1074,10 +1074,11 @@ is_secure_boot_enforced()
check_crash_mem_reserved() { - MEM_RESERVED=$(cat /sys/kernel/kexec_crash_size) - if [ $MEM_RESERVED -eq 0 ] - then - echo "No memory reserved for crash kernel." >&2 + local mem_reserved + + mem_reserved=$(cat /sys/kernel/kexec_crash_size) + if [ $mem_reserved -eq 0 ]; then + echo "No memory reserved for crash kernel" return 1 fi
@@ -1163,22 +1164,23 @@ check_default_config()
start() { - check_config + check_dump_feasibility if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1 fi
- if sestatus 2>/dev/null | grep -q "SELinux status.*enabled"; then - selinux_relabel - fi - save_raw + check_config if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1 fi
- check_dump_feasibility + if sestatus 2>/dev/null | grep -q "SELinux status.*enabled"; then + selinux_relabel + fi + + save_raw if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1
On 04/29/17 at 09:15am, Xunlei Pang wrote:
Some cloud people complained that VM boot speed is slower than Ubuntu and other distributions, after some debugging, we found that one of the causes is kdump service starts too slow(7seconds according to the test result on the test VM), actually there is no "crashkernel=X" specified. Although kdump service is parallel, it affects the speed more or less especially on VMs with few cpus, which is unacceptable. It is even worse when kdump initramfs is built out in case of no reserved memory at first boot.
Commit afa4a35d3 ("kdumpctrl: kdump feasibility should fail if no crash memory") can actually solve this issue.
This patch is a supplement of above-mentioned commit, we bail out start() even earlier in case of no reserved memory.
Also made some cosmatic changes for check_crash_mem_reserved().
- Before this patch
$ time kdumpctl start No memory reserved for crash kernel. Starting kdump: [FAILED]
real 0m0.282s user 0m0.184s sys 0m0.146s
- After this patch
$ time kdumpctl start No memory reserved for crash kernel Starting kdump: [FAILED]
real 0m0.010s user 0m0.008s sys 0m0.001s
Signed-off-by: Xunlei Pang xlpang@redhat.com
kdumpctl | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/kdumpctl b/kdumpctl index 6b5be09..129cb6a 100755 --- a/kdumpctl +++ b/kdumpctl @@ -1074,10 +1074,11 @@ is_secure_boot_enforced()
check_crash_mem_reserved() {
- MEM_RESERVED=$(cat /sys/kernel/kexec_crash_size)
- if [ $MEM_RESERVED -eq 0 ]
- then
echo "No memory reserved for crash kernel." >&2
- local mem_reserved
- mem_reserved=$(cat /sys/kernel/kexec_crash_size)
- if [ $mem_reserved -eq 0 ]; then
return 1 fiecho "No memory reserved for crash kernel"
@@ -1163,22 +1164,23 @@ check_default_config()
start() {
- check_config
- check_dump_feasibility if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1 fi
- if sestatus 2>/dev/null | grep -q "SELinux status.*enabled"; then
selinux_relabel
- fi
- save_raw
- check_config if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1 fi
- check_dump_feasibility
- if sestatus 2>/dev/null | grep -q "SELinux status.*enabled"; then
selinux_relabel
- fi
- save_raw if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1
-- 1.8.3.1
Acked-by: Dave Young dyoung@redhat.com
Thanks Dave
On Saturday 29 April 2017 06:45 AM, Xunlei Pang wrote:
Some cloud people complained that VM boot speed is slower than Ubuntu and other distributions, after some debugging, we found that one of the causes is kdump service starts too slow(7seconds according to the test result on the test VM), actually there is no "crashkernel=X" specified. Although kdump service is parallel, it affects the speed more or less especially on VMs with few cpus, which is unacceptable. It is even worse when kdump initramfs is built out in case of no reserved memory at first boot.
Commit afa4a35d3 ("kdumpctrl: kdump feasibility should fail if no crash memory") can actually solve this issue.
This patch is a supplement of above-mentioned commit, we bail out start() even earlier in case of no reserved memory.
Also made some cosmatic changes for check_crash_mem_reserved().
- Before this patch
$ time kdumpctl start No memory reserved for crash kernel. Starting kdump: [FAILED]
real 0m0.282s user 0m0.184s sys 0m0.146s
- After this patch
$ time kdumpctl start No memory reserved for crash kernel Starting kdump: [FAILED]
real 0m0.010s user 0m0.008s sys 0m0.001s
Signed-off-by: Xunlei Pang xlpang@redhat.com
Well, above is certainly an improvement and so Acked-by: Pratyush Anand panand@redhat.com
However, there is one more point..kdumpctl takes around 5 second on my 8 Core(TM) i7-4700MQ CPU @ 2.40GHz,8G machine for a positive case when crash mem is reserved and initrd did not need rebuilt.
# time kdumpctl restart kexec: unloaded kdump kernel Stopping kdump: [OK] TS before check_files_modified 27.244429329 TS before check_dump_fs_modified 27.252359273 TS before check_wdt_modified 29.068929664 TS After check_wdt_modified 31.316004020 kexec: loaded kdump kernel Starting kdump: [OK]
real 0m4.857s user 0m4.953s sys 0m0.768s
both check_dump_fs_modified() and check_wdt_modified() are taking approximately 2 sec to execute because of lsinitrd call.
So, I was just wondering if we can cache all necessary initrd info in some /etc/ file whenever a new rebuild happens.. Alternative could be to improve rebuild process and lsinitrd application so that such information can be stored in first file and can be searched quickly.
I think, 1st option is easier to do and independent of other packages.
~Pratyush
kdumpctl | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/kdumpctl b/kdumpctl index 6b5be09..129cb6a 100755 --- a/kdumpctl +++ b/kdumpctl @@ -1074,10 +1074,11 @@ is_secure_boot_enforced()
check_crash_mem_reserved() {
- MEM_RESERVED=$(cat /sys/kernel/kexec_crash_size)
- if [ $MEM_RESERVED -eq 0 ]
- then
echo "No memory reserved for crash kernel." >&2
- local mem_reserved
- mem_reserved=$(cat /sys/kernel/kexec_crash_size)
- if [ $mem_reserved -eq 0 ]; then
return 1 fiecho "No memory reserved for crash kernel"
@@ -1163,22 +1164,23 @@ check_default_config()
start() {
- check_config
- check_dump_feasibility if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1 fi
- if sestatus 2>/dev/null | grep -q "SELinux status.*enabled"; then
selinux_relabel
- fi
- save_raw
- check_config if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1 fi
- check_dump_feasibility
- if sestatus 2>/dev/null | grep -q "SELinux status.*enabled"; then
selinux_relabel
- fi
- save_raw if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1
On 04/30/2017 at 10:24 AM, Pratyush Anand wrote:
On Saturday 29 April 2017 06:45 AM, Xunlei Pang wrote:
Some cloud people complained that VM boot speed is slower than Ubuntu and other distributions, after some debugging, we found that one of the causes is kdump service starts too slow(7seconds according to the test result on the test VM), actually there is no "crashkernel=X" specified. Although kdump service is parallel, it affects the speed more or less especially on VMs with few cpus, which is unacceptable. It is even worse when kdump initramfs is built out in case of no reserved memory at first boot.
Commit afa4a35d3 ("kdumpctrl: kdump feasibility should fail if no crash memory") can actually solve this issue.
This patch is a supplement of above-mentioned commit, we bail out start() even earlier in case of no reserved memory.
Also made some cosmatic changes for check_crash_mem_reserved().
- Before this patch
$ time kdumpctl start No memory reserved for crash kernel. Starting kdump: [FAILED]
real 0m0.282s user 0m0.184s sys 0m0.146s
- After this patch
$ time kdumpctl start No memory reserved for crash kernel Starting kdump: [FAILED]
real 0m0.010s user 0m0.008s sys 0m0.001s
Signed-off-by: Xunlei Pang xlpang@redhat.com
Well, above is certainly an improvement and so Acked-by: Pratyush Anand panand@redhat.com
However, there is one more point..kdumpctl takes around 5 second on my 8 Core(TM) i7-4700MQ CPU @ 2.40GHz,8G machine for a positive case when crash mem is reserved and initrd did not need rebuilt.
# time kdumpctl restart kexec: unloaded kdump kernel Stopping kdump: [OK] TS before check_files_modified 27.244429329 TS before check_dump_fs_modified 27.252359273 TS before check_wdt_modified 29.068929664 TS After check_wdt_modified 31.316004020 kexec: loaded kdump kernel Starting kdump: [OK]
real 0m4.857s user 0m4.953s sys 0m0.768s
both check_dump_fs_modified() and check_wdt_modified() are taking approximately 2 sec to execute because of lsinitrd call.
Yes, this is another issue need to be solved, there was BZ1446698 tracking this.
So, I was just wondering if we can cache all necessary initrd info in some /etc/ file whenever a new rebuild happens.. Alternative could be to improve rebuild process and lsinitrd application so that such information can be stored in first file and can be searched quickly.
I think, 1st option is easier to do and independent of other packages.
Agree, one approach that I can think of now is that we can insert a special file into the initramfs, so we only need one "lsinitrd -f" operation during the whole start process.
Regards, Xunlei
~Pratyush
kdumpctl | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/kdumpctl b/kdumpctl index 6b5be09..129cb6a 100755 --- a/kdumpctl +++ b/kdumpctl @@ -1074,10 +1074,11 @@ is_secure_boot_enforced()
check_crash_mem_reserved() {
- MEM_RESERVED=$(cat /sys/kernel/kexec_crash_size)
- if [ $MEM_RESERVED -eq 0 ]
- then
echo "No memory reserved for crash kernel." >&2
- local mem_reserved
- mem_reserved=$(cat /sys/kernel/kexec_crash_size)
- if [ $mem_reserved -eq 0 ]; then
fiecho "No memory reserved for crash kernel" return 1
@@ -1163,22 +1164,23 @@ check_default_config()
start() {
- check_config
- check_dump_feasibility if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1 fi
- if sestatus 2>/dev/null | grep -q "SELinux status.*enabled"; then
selinux_relabel
- fi
- save_raw
- check_config if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1 fi
- check_dump_feasibility
- if sestatus 2>/dev/null | grep -q "SELinux status.*enabled"; then
selinux_relabel
- fi
- save_raw if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1
On Monday 01 May 2017 10:00 AM, Xunlei Pang wrote:
both check_dump_fs_modified() and check_wdt_modified() are taking approximately 2 sec to execute because of lsinitrd call.
Yes, this is another issue need to be solved, there was BZ1446698 tracking this.
Ah..ok..Good to know that we already have a tracker for this.
~Pratyush
On 04/29/17 at 09:15am, Xunlei Pang wrote:
Some cloud people complained that VM boot speed is slower than Ubuntu and other distributions, after some debugging, we found that one of the causes is kdump service starts too slow(7seconds according to the test result on the test VM), actually there is no "crashkernel=X" specified. Although kdump service is parallel, it affects the speed more or less especially on VMs with few cpus, which is unacceptable. It is even worse when kdump initramfs is built out in case of no reserved memory at first boot.
Commit afa4a35d3 ("kdumpctrl: kdump feasibility should fail if no crash memory") can actually solve this issue.
This patch is a supplement of above-mentioned commit, we bail out start() even earlier in case of no reserved memory.
Also made some cosmatic changes for check_crash_mem_reserved().
- Before this patch
$ time kdumpctl start No memory reserved for crash kernel. Starting kdump: [FAILED]
real 0m0.282s user 0m0.184s sys 0m0.146s
- After this patch
$ time kdumpctl start No memory reserved for crash kernel Starting kdump: [FAILED]
real 0m0.010s user 0m0.008s sys 0m0.001s
Signed-off-by: Xunlei Pang xlpang@redhat.com
kdumpctl | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/kdumpctl b/kdumpctl index 6b5be09..129cb6a 100755 --- a/kdumpctl +++ b/kdumpctl @@ -1074,10 +1074,11 @@ is_secure_boot_enforced()
check_crash_mem_reserved() {
- MEM_RESERVED=$(cat /sys/kernel/kexec_crash_size)
- if [ $MEM_RESERVED -eq 0 ]
- then
echo "No memory reserved for crash kernel." >&2
- local mem_reserved
- mem_reserved=$(cat /sys/kernel/kexec_crash_size)
- if [ $mem_reserved -eq 0 ]; then
return 1 fiecho "No memory reserved for crash kernel"
@@ -1163,22 +1164,23 @@ check_default_config()
start() {
- check_config
- check_dump_feasibility if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1 fi
- if sestatus 2>/dev/null | grep -q "SELinux status.*enabled"; then
selinux_relabel
- fi
- save_raw
- check_config if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1 fi
- check_dump_feasibility
- if sestatus 2>/dev/null | grep -q "SELinux status.*enabled"; then
selinux_relabel
- fi
- save_raw if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1
-- 1.8.3.1
Hi Xunlei
Applied this patch, but I did not bump the kexec-tools version, will increase after the other series get discussed.
Thanks Dave
On 05/02/2017 at 05:26 PM, Dave Young wrote:
On 04/29/17 at 09:15am, Xunlei Pang wrote:
Some cloud people complained that VM boot speed is slower than Ubuntu and other distributions, after some debugging, we found that one of the causes is kdump service starts too slow(7seconds according to the test result on the test VM), actually there is no "crashkernel=X" specified. Although kdump service is parallel, it affects the speed more or less especially on VMs with few cpus, which is unacceptable. It is even worse when kdump initramfs is built out in case of no reserved memory at first boot.
Commit afa4a35d3 ("kdumpctrl: kdump feasibility should fail if no crash memory") can actually solve this issue.
This patch is a supplement of above-mentioned commit, we bail out start() even earlier in case of no reserved memory.
Also made some cosmatic changes for check_crash_mem_reserved().
- Before this patch
$ time kdumpctl start No memory reserved for crash kernel. Starting kdump: [FAILED]
real 0m0.282s user 0m0.184s sys 0m0.146s
- After this patch
$ time kdumpctl start No memory reserved for crash kernel Starting kdump: [FAILED]
real 0m0.010s user 0m0.008s sys 0m0.001s
Signed-off-by: Xunlei Pang xlpang@redhat.com
kdumpctl | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/kdumpctl b/kdumpctl index 6b5be09..129cb6a 100755 --- a/kdumpctl +++ b/kdumpctl @@ -1074,10 +1074,11 @@ is_secure_boot_enforced()
check_crash_mem_reserved() {
- MEM_RESERVED=$(cat /sys/kernel/kexec_crash_size)
- if [ $MEM_RESERVED -eq 0 ]
- then
echo "No memory reserved for crash kernel." >&2
- local mem_reserved
- mem_reserved=$(cat /sys/kernel/kexec_crash_size)
- if [ $mem_reserved -eq 0 ]; then
return 1 fiecho "No memory reserved for crash kernel"
@@ -1163,22 +1164,23 @@ check_default_config()
start() {
- check_config
- check_dump_feasibility if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1 fi
- if sestatus 2>/dev/null | grep -q "SELinux status.*enabled"; then
selinux_relabel
- fi
- save_raw
- check_config if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1 fi
- check_dump_feasibility
- if sestatus 2>/dev/null | grep -q "SELinux status.*enabled"; then
selinux_relabel
- fi
- save_raw if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1
-- 1.8.3.1
Hi Xunlei
Applied this patch, but I did not bump the kexec-tools version, will increase after the other series get discussed.
OK, thanks!
Thanks Dave