If dump target is ipv6 address, a host should have ipv6 address ready before starting kdump service. Otherwise, kdump service fails to start due to the failure "ssh dump_server_ip mkdir -p $SAVE_PATH". And user can see message like: "Could not create root@2620:52:0:10da:46a8:42ff:fe23:3272/var/crash"
I observe a long period (about 30s) on some machine before they got ipv6 address dynamiclly, which is never seen on ipv4 host.
Hence kdump service has a dependency on ipv6 address. But there is no good way to resolve it. One way is asking user to run the cmd "nmcli connection modify eth0 ipv6.may-fail false". But this will block systemd until ipv6 address is ready. Despite doing so, kdump can try its best (wait 1 minutes after it starts up) before failure.
How to implement the wait is arguable. It will involve too many technique details if explicitly waiting on ipv6 address, instead, just lean on 'ssh' return value to see the availability of network.
Signed-off-by: Pingfan Liu piliu@redhat.com --- kdumpctl | 36 ++++++++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-)
diff --git a/kdumpctl b/kdumpctl index a1a6ee2..60df26b 100755 --- a/kdumpctl +++ b/kdumpctl @@ -730,11 +730,43 @@ check_ssh_config() return 0 }
+# ipv6 host address may takes a long time to be ready. +# Instead of checking against ipv6 address, we just check the network reachable +# by the return val of 'ssh' +check_and_wait_network_ready() +{ + local netready="false" + local start_time=$(date +%s) + + + while true; do + ssh -q -i $SSH_KEY_LOCATION -o BatchMode=yes $DUMP_TARGET mkdir -p $SAVE_PATH + # ssh exits with the exit status of the remote command or with 255 if an error occurred + if [ $? -ne 255 ]; then + netready="true" + break; + fi + cur=$(date +%s) + diff=$( $cur - $start_time ) + # 60s time out + if [ $diff -gt 60 ]; then + break; + fi + sleep 1 + done + + if [ "$netready" == "false" ]; then + return 1 + fi + + return 0 +} + check_ssh_target() { local _ret - ssh -q -i $SSH_KEY_LOCATION -o BatchMode=yes $DUMP_TARGET mkdir -p $SAVE_PATH - _ret=$? + + _ret=check_and_wait_network_ready() if [ $_ret -ne 0 ]; then echo "Could not create $DUMP_TARGET:$SAVE_PATH, you probably need to run "kdumpctl propagate"" >&2 return 1
On 08/12/2019 11:06 AM, Pingfan Liu wrote:
If dump target is ipv6 address, a host should have ipv6 address ready before starting kdump service. Otherwise, kdump service fails to start due to the failure "ssh dump_server_ip mkdir -p $SAVE_PATH". And user can see message like: "Could not create root@2620:52:0:10da:46a8:42ff:fe23:3272/var/crash"
I observe a long period (about 30s) on some machine before they got ipv6 address dynamiclly, which is never seen on ipv4 host.
Hence kdump service has a dependency on ipv6 address. But there is no good way to resolve it. One way is asking user to run the cmd "nmcli connection modify eth0 ipv6.may-fail false". But this will block systemd until ipv6 address is ready. Despite doing so, kdump can try its best (wait 1 minutes after it starts up) before failure.
How to implement the wait is arguable. It will involve too many technique details if explicitly waiting on ipv6 address, instead, just lean on 'ssh' return value to see the availability of network.
Signed-off-by: Pingfan Liu piliu@redhat.com
kdumpctl | 36 ++++++++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-)
diff --git a/kdumpctl b/kdumpctl index a1a6ee2..60df26b 100755 --- a/kdumpctl +++ b/kdumpctl @@ -730,11 +730,43 @@ check_ssh_config() return 0 }
+# ipv6 host address may takes a long time to be ready. +# Instead of checking against ipv6 address, we just check the network reachable +# by the return val of 'ssh' +check_and_wait_network_ready() +{
- local netready="false"
- local start_time=$(date +%s)
- while true; do
ssh -q -i $SSH_KEY_LOCATION -o BatchMode=yes $DUMP_TARGET mkdir -p $SAVE_PATH# ssh exits with the exit status of the remote command or with 255 if an error occurredif [ $? -ne 255 ]; then
NACK, Kairui pointed out that "mkdir" may fail under some special condition, e.g. selinux. [..]
On 08/12/19 at 04:07pm, piliu wrote:
On 08/12/2019 11:06 AM, Pingfan Liu wrote:
If dump target is ipv6 address, a host should have ipv6 address ready before starting kdump service. Otherwise, kdump service fails to start due to the failure "ssh dump_server_ip mkdir -p $SAVE_PATH". And user can see message like: "Could not create root@2620:52:0:10da:46a8:42ff:fe23:3272/var/crash"
I observe a long period (about 30s) on some machine before they got ipv6 address dynamiclly, which is never seen on ipv4 host.
Hence kdump service has a dependency on ipv6 address. But there is no good way to resolve it. One way is asking user to run the cmd "nmcli connection modify eth0 ipv6.may-fail false". But this will block systemd until ipv6 address is ready. Despite doing so, kdump can try its best (wait 1 minutes after it starts up) before failure.
How to implement the wait is arguable. It will involve too many technique details if explicitly waiting on ipv6 address, instead, just lean on 'ssh' return value to see the availability of network.
Signed-off-by: Pingfan Liu piliu@redhat.com
kdumpctl | 36 ++++++++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-)
diff --git a/kdumpctl b/kdumpctl index a1a6ee2..60df26b 100755 --- a/kdumpctl +++ b/kdumpctl @@ -730,11 +730,43 @@ check_ssh_config() return 0 }
+# ipv6 host address may takes a long time to be ready. +# Instead of checking against ipv6 address, we just check the network reachable +# by the return val of 'ssh' +check_and_wait_network_ready() +{
- local netready="false"
- local start_time=$(date +%s)
- while true; do
ssh -q -i $SSH_KEY_LOCATION -o BatchMode=yes $DUMP_TARGET mkdir -p $SAVE_PATH# ssh exits with the exit status of the remote command or with 255 if an error occurredif [ $? -ne 255 ]; thenNACK, Kairui pointed out that "mkdir" may fail under some special condition, e.g. selinux.
The patch has been merged, can you send a appending fix?
Maybe use below for the network ready checking: ssh -q -i $SSH_KEY_LOCATION -o BatchMode=yes $DUMP_TARGET true
And do the mkdir later
Thanks Dave
On 08/13/2019 09:56 AM, Dave Young wrote:
On 08/12/19 at 04:07pm, piliu wrote:
On 08/12/2019 11:06 AM, Pingfan Liu wrote:
If dump target is ipv6 address, a host should have ipv6 address ready before starting kdump service. Otherwise, kdump service fails to start due to the failure "ssh dump_server_ip mkdir -p $SAVE_PATH". And user can see message like: "Could not create root@2620:52:0:10da:46a8:42ff:fe23:3272/var/crash"
I observe a long period (about 30s) on some machine before they got ipv6 address dynamiclly, which is never seen on ipv4 host.
Hence kdump service has a dependency on ipv6 address. But there is no good way to resolve it. One way is asking user to run the cmd "nmcli connection modify eth0 ipv6.may-fail false". But this will block systemd until ipv6 address is ready. Despite doing so, kdump can try its best (wait 1 minutes after it starts up) before failure.
How to implement the wait is arguable. It will involve too many technique details if explicitly waiting on ipv6 address, instead, just lean on 'ssh' return value to see the availability of network.
Signed-off-by: Pingfan Liu piliu@redhat.com
kdumpctl | 36 ++++++++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-)
diff --git a/kdumpctl b/kdumpctl index a1a6ee2..60df26b 100755 --- a/kdumpctl +++ b/kdumpctl @@ -730,11 +730,43 @@ check_ssh_config() return 0 }
+# ipv6 host address may takes a long time to be ready. +# Instead of checking against ipv6 address, we just check the network reachable +# by the return val of 'ssh' +check_and_wait_network_ready() +{
- local netready="false"
- local start_time=$(date +%s)
- while true; do
ssh -q -i $SSH_KEY_LOCATION -o BatchMode=yes $DUMP_TARGET mkdir -p $SAVE_PATH# ssh exits with the exit status of the remote command or with 255 if an error occurredif [ $? -ne 255 ]; thenNACK, Kairui pointed out that "mkdir" may fail under some special condition, e.g. selinux.
The patch has been merged, can you send a appending fix?
No
Maybe use below for the network ready checking: ssh -q -i $SSH_KEY_LOCATION -o BatchMode=yes $DUMP_TARGET true
And do the mkdir later
The v3 patch has considered this situation, and return 1 for either connection failure or mkdir.
Thanks, Pingfan
-
Dave Young -
piliu -
Pingfan Liu