The kdump cannot dump the core in the correct directory in Atomic, if set the path "/var/crash" in the /etc/kdump.conf. We found that kdump cannot deal with the bind mounts correctly.
Fix this issue to make kdump dump core correctly.
v2: - fix the issue to dump directory /var/crash correctly in Atomic. - wrap the common function dump_common_fs to handle the dump core - Add a new function to determine the directory is bind mounts or not
v1: - fix the issue to dump directory /sysroot/var/crash instead of /sysroot/crash in Atomic
Minfei Huang (2): dracut-moduel-setup: Filte the bind mounted directory in the Atomic kdump-lib-initramfs: Dump the correct directory in the Atomic
kdump-lib-initramfs.sh | 91 +++++++++++++++++++++++++++++++++++++++++++------- kdump-lib.sh | 43 +++++++++++++++++++++++- 2 files changed, 121 insertions(+), 13 deletions(-)
The crash dump is saved in the /sysroot/crash directory, not the /sysroot/var/crash in the atomic system.
We found we have the uncorrect dump directory, when we use command "findmnt" to find the dump directory's mountpoint which is a bind mounted directory.
To solve this issue, we should determine the real mountpont, if the directory is a bind mounted directory.
Signed-off-by: Minfei Huang mhuang@redhat.com --- kdump-lib.sh | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-)
diff --git a/kdump-lib.sh b/kdump-lib.sh index f24f08d..e4df376 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -86,9 +86,41 @@ get_root_fs_device() return }
+# findmnt uses the option "-v, --nofsroot" to exclusive the [/dir] +# in the SOURCE column for bind-mounts, then if $_dev equals to +# $_dev_no_bind, the mountpoint is not bind mounted directory. +is_bind_mount() +{ + local _dev=$(findmnt -k -n -r -o SOURCE $1) + local _dev_no_bind=$(findmnt -k -n -r -v -o SOURCE $1) + + if [[ $_dev = $_dev_no_bind ]]; then + return 1 + else + return 0 + fi +} + +# We are looking to see if a part of dump path is mounted on some disk. +# But we don't want to look for bind mounted directories. So we should +# determine the real mountpoint, if the directory is a bind mounted +# directory. get_mntpoint_from_path() { - echo $(df $1 | tail -1 | awk '{print $NF}') + local _mnt=$(df $1 | tail -1 | awk '{print $NF}') + + if is_bind_mount $mnt; then + # bind mounted directory, we should find the real mount point + # where the device mounts. + for m in `findmnt -k -n -r -o TARGET $_dev_no_bind`; do + if ! is_bind_mount $m; then + echo $m + break + fi + done + else + echo $_mnt + fi }
get_target_from_path()
Karel Zak is maintaining util-linux, he as a blog about bind mount: http://karelzak.blogspot.sg/2011/04/bind-mounts-mtab-and-read-only.html
Ccing him. Karel, can you help reviewing this patch to differenciate bind mount?
On 02/10/15 at 06:46pm, Minfei Huang wrote:
The crash dump is saved in the /sysroot/crash directory, not the /sysroot/var/crash in the atomic system.
We found we have the uncorrect dump directory, when we use command "findmnt" to find the dump directory's mountpoint which is a bind mounted directory.
To solve this issue, we should determine the real mountpont, if the directory is a bind mounted directory.
Signed-off-by: Minfei Huang mhuang@redhat.com
kdump-lib.sh | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-)
diff --git a/kdump-lib.sh b/kdump-lib.sh index f24f08d..e4df376 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -86,9 +86,41 @@ get_root_fs_device() return }
+# findmnt uses the option "-v, --nofsroot" to exclusive the [/dir] +# in the SOURCE column for bind-mounts, then if $_dev equals to +# $_dev_no_bind, the mountpoint is not bind mounted directory. +is_bind_mount() +{
- local _dev=$(findmnt -k -n -r -o SOURCE $1)
- local _dev_no_bind=$(findmnt -k -n -r -v -o SOURCE $1)
- if [[ $_dev = $_dev_no_bind ]]; then
return 1
- else
return 0
- fi
+}
+# We are looking to see if a part of dump path is mounted on some disk. +# But we don't want to look for bind mounted directories. So we should +# determine the real mountpoint, if the directory is a bind mounted +# directory. get_mntpoint_from_path() {
- echo $(df $1 | tail -1 | awk '{print $NF}')
- local _mnt=$(df $1 | tail -1 | awk '{print $NF}')
- if is_bind_mount $mnt; then
# bind mounted directory, we should find the real mount point
# where the device mounts.
for m in `findmnt -k -n -r -o TARGET $_dev_no_bind`; do
if ! is_bind_mount $m; then
echo $m
break
fi
done
- else
echo $_mnt
- fi
}
get_target_from_path()
2.2.2
kexec mailing list kexec@lists.fedoraproject.org https://lists.fedoraproject.org/mailman/listinfo/kexec
On Wed, Feb 11, 2015 at 03:17:48PM +0800, Dave Young wrote:
Karel Zak is maintaining util-linux, he as a blog about bind mount: http://karelzak.blogspot.sg/2011/04/bind-mounts-mtab-and-read-only.html
Ccing him. Karel, can you help reviewing this patch to differenciate bind mount?
It seems I need new blog post to more explicitly explain that there is no "bind mounts" at all, the "bind" is operation, not status.
On 02/10/15 at 06:46pm, Minfei Huang wrote:
To solve this issue, we should determine the real mountpont, if the directory is a bind mounted directory.
There is no real and bind mountpoint, both is the same thing. The bind operation creates another independent connection to the filesystem.
+# findmnt uses the option "-v, --nofsroot" to exclusive the [/dir] +# in the SOURCE column for bind-mounts, then if $_dev equals to +# $_dev_no_bind, the mountpoint is not bind mounted directory.
The [/dir] means that you have mounted subdirectory rather than root of the filesystem.
mount /dev/sdc1 /mnt/A mkdir /mnt/A/subdir
mount --bind /mnt/A/subdir /mnt/B mount --bind /mnt/A /mnt/C
TARGET SOURCE FSTYPE OPTIONS /mnt/A /dev/sdc1 ext4 rw,relatime,seclabel,stripe=32,data=ordered /mnt/B /dev/sdc1[/subdir] ext4 rw,relatime,seclabel,stripe=32,data=ordered /mnt/C /dev/sdc1 ext4 rw,relatime,seclabel,stripe=32,data=ordered
... for /mnt/C there is no [/dir]
Note that btrfs uses the same if you mount btrfs subvolume.
+is_bind_mount() +{
- local _dev=$(findmnt -k -n -r -o SOURCE $1)
- local _dev_no_bind=$(findmnt -k -n -r -v -o SOURCE $1)
- if [[ $_dev = $_dev_no_bind ]]; then
return 1
- else
return 0
- fi
+}
+# We are looking to see if a part of dump path is mounted on some disk. +# But we don't want to look for bind mounted directories. So we should +# determine the real mountpoint, if the directory is a bind mounted +# directory. get_mntpoint_from_path() {
- echo $(df $1 | tail -1 | awk '{print $NF}')
- local _mnt=$(df $1 | tail -1 | awk '{print $NF}')
local _mnt=$(findmnt -n -o TARGET --target $1)
- if is_bind_mount $mnt; then
# bind mounted directory, we should find the real mount point
# where the device mounts.
for m in `findmnt -k -n -r -o TARGET $_dev_no_bind`; do
if ! is_bind_mount $m; then
echo $m
break
fi
done
The mountpoint created by bind operation is independent on previous mounts,
mount /mnt/sda1 /A mount --bind /A /B umount /A
is pretty valid situation and /B still points to the filesystem.
- else
echo $_mnt
- fi
}
I'm not sure if I good understand the original problem, but I'm sure that care about bind mounts is mistake ;-) If the problem is that dump is not in the filesystem root then you can try to check if the root of the filesystem is mounted somewhere else, but maybe the best would be avoid such fragile setup at all.
Karel
Karel
Thanks a lot for your comments, it is helpful to us.
On 02/11/15 at 12:39pm, Karel Zak wrote:
On Wed, Feb 11, 2015 at 03:17:48PM +0800, Dave Young wrote:
Karel Zak is maintaining util-linux, he as a blog about bind mount: http://karelzak.blogspot.sg/2011/04/bind-mounts-mtab-and-read-only.html
Ccing him. Karel, can you help reviewing this patch to differenciate bind mount?
It seems I need new blog post to more explicitly explain that there is no "bind mounts" at all, the "bind" is operation, not status.
Ok, so there's nothing we can do to differenciate them because they are just same..
On 02/10/15 at 06:46pm, Minfei Huang wrote:
To solve this issue, we should determine the real mountpont, if the directory is a bind mounted directory.
There is no real and bind mountpoint, both is the same thing. The bind operation creates another independent connection to the filesystem.
+# findmnt uses the option "-v, --nofsroot" to exclusive the [/dir] +# in the SOURCE column for bind-mounts, then if $_dev equals to +# $_dev_no_bind, the mountpoint is not bind mounted directory.
The [/dir] means that you have mounted subdirectory rather than root of the filesystem.
mount /dev/sdc1 /mnt/A mkdir /mnt/A/subdir
mount --bind /mnt/A/subdir /mnt/B mount --bind /mnt/A /mnt/C
TARGET SOURCE FSTYPE OPTIONS /mnt/A /dev/sdc1 ext4 rw,relatime,seclabel,stripe=32,data=ordered /mnt/B /dev/sdc1[/subdir] ext4 rw,relatime,seclabel,stripe=32,data=ordered /mnt/C /dev/sdc1 ext4 rw,relatime,seclabel,stripe=32,data=ordered
... for /mnt/C there is no [/dir]
Note that btrfs uses the same if you mount btrfs subvolume.
+is_bind_mount() +{
- local _dev=$(findmnt -k -n -r -o SOURCE $1)
- local _dev_no_bind=$(findmnt -k -n -r -v -o SOURCE $1)
- if [[ $_dev = $_dev_no_bind ]]; then
return 1
- else
return 0
- fi
+}
+# We are looking to see if a part of dump path is mounted on some disk. +# But we don't want to look for bind mounted directories. So we should +# determine the real mountpoint, if the directory is a bind mounted +# directory. get_mntpoint_from_path() {
- echo $(df $1 | tail -1 | awk '{print $NF}')
- local _mnt=$(df $1 | tail -1 | awk '{print $NF}')
local _mnt=$(findmnt -n -o TARGET --target $1)
- if is_bind_mount $mnt; then
# bind mounted directory, we should find the real mount point
# where the device mounts.
for m in `findmnt -k -n -r -o TARGET $_dev_no_bind`; do
if ! is_bind_mount $m; then
echo $m
break
fi
done
The mountpoint created by bind operation is independent on previous mounts,
mount /mnt/sda1 /A mount --bind /A /B umount /A
is pretty valid situation and /B still points to the filesystem.
Hmm, got it, so they are equal, but for subdirectory mount there will be dependency?
- else
echo $_mnt
- fi
}
I'm not sure if I good understand the original problem, but I'm sure that care about bind mounts is mistake ;-) If the problem is that dump is not in the filesystem root then you can try to check if the root of the filesystem is mounted somewhere else, but maybe the best would be avoid such fragile setup at all.
We need to know more about the setup of atomic systems, will add you to the rh bug cc
Thanks Dave
On Thu, Feb 12, 2015 at 01:50:32PM +0800, Dave Young wrote:
The mountpoint created by bind operation is independent on previous mounts,
mount /mnt/sda1 /A mount --bind /A /B umount /A
is pretty valid situation and /B still points to the filesystem.
Hmm, got it, so they are equal, but for subdirectory mount there will be dependency?
No, you can still umount, it's just kernel feature that mount can points to non-root filesystem directory.
# findmnt /dev/sdc1 TARGET SOURCE FSTYPE OPTIONS /mnt/B /dev/sdc1[/subdir] ext4 rw,relatime,seclabel,stripe=32,data=ordered /mnt/A /dev/sdc1 ext4 rw,relatime,seclabel,stripe=32,data=ordered
# umount /mnt/A
# findmnt /dev/sdc1 TARGET SOURCE FSTYPE OPTIONS /mnt/B /dev/sdc1[/subdir] ext4 rw,relatime,seclabel,stripe=32,data=ordered
Karel
On 02/12/15 at 01:03pm, Karel Zak wrote:
On Thu, Feb 12, 2015 at 01:50:32PM +0800, Dave Young wrote:
The mountpoint created by bind operation is independent on previous mounts,
mount /mnt/sda1 /A mount --bind /A /B umount /A
is pretty valid situation and /B still points to the filesystem.
Hmm, got it, so they are equal, but for subdirectory mount there will be dependency?
No, you can still umount, it's just kernel feature that mount can points to non-root filesystem directory.
# findmnt /dev/sdc1 TARGET SOURCE FSTYPE OPTIONS /mnt/B /dev/sdc1[/subdir] ext4 rw,relatime,seclabel,stripe=32,data=ordered /mnt/A /dev/sdc1 ext4 rw,relatime,seclabel,stripe=32,data=ordered
# umount /mnt/A
# findmnt /dev/sdc1 TARGET SOURCE FSTYPE OPTIONS /mnt/B /dev/sdc1[/subdir] ext4 rw,relatime,seclabel,stripe=32,data=ordered
Karel
Hi, All!
Thanks Karel for explaination.
Dave, I think we should not mount the root in the 2nd kernel. Since the root and "/var" are bind mount directory, we don't need to mount the root firstly. I think we can do whatever we want, if we mount the device where the root and "/var" are in.
Following is the tree in atomic. -bash-4.2# findmnt / TARGET SOURCE FSTYPE OPTIONS / /dev/mapper/atomicos-root[/ostree/deploy/rhel-atomic-host/deploy/4b513b5012db32d51b411f8e54e357a231f92b94ff6343bc0fb44cb53b9e0941.0] xfs rw,relat -bash-4.2# findmnt /var TARGET SOURCE FSTYPE OPTIONS /var /dev/mapper/atomicos-root[/ostree/deploy/rhel-atomic-host/var] xfs rw,relat
We can find the fact that the root directory is a bind mount directory which points to the /ostree/deploy/rhel-atomic-host/deploy/4b513b5012db32d51b411f8e54e357a231f92b94ff6343bc0fb44cb53b9e0941.0 directory, and "/var" points to the /ostree/deploy/rhel-atomic-host/var directory.
To summarize the point, we can mount the device (/dev/mapper/atomicos-root) to dump to the corresponding subdirecotry (/ostree/deploy/rhel-atomic-host/var).
Thanks Minfei
-- Karel Zak kzak@redhat.com http://karelzak.blogspot.com
On 02/11/15 at 12:39pm, Karel Zak wrote:
On Wed, Feb 11, 2015 at 03:17:48PM +0800, Dave Young wrote:
Karel Zak is maintaining util-linux, he as a blog about bind mount: http://karelzak.blogspot.sg/2011/04/bind-mounts-mtab-and-read-only.html
Ccing him. Karel, can you help reviewing this patch to differenciate bind mount?
It seems I need new blog post to more explicitly explain that there is no "bind mounts" at all, the "bind" is operation, not status.
On 02/10/15 at 06:46pm, Minfei Huang wrote:
To solve this issue, we should determine the real mountpont, if the directory is a bind mounted directory.
There is no real and bind mountpoint, both is the same thing. The bind operation creates another independent connection to the filesystem.
+# findmnt uses the option "-v, --nofsroot" to exclusive the [/dir] +# in the SOURCE column for bind-mounts, then if $_dev equals to +# $_dev_no_bind, the mountpoint is not bind mounted directory.
The [/dir] means that you have mounted subdirectory rather than root of the filesystem.
mount /dev/sdc1 /mnt/A mkdir /mnt/A/subdir
mount --bind /mnt/A/subdir /mnt/B mount --bind /mnt/A /mnt/C
TARGET SOURCE FSTYPE OPTIONS /mnt/A /dev/sdc1 ext4 rw,relatime,seclabel,stripe=32,data=ordered /mnt/B /dev/sdc1[/subdir] ext4 rw,relatime,seclabel,stripe=32,data=ordered /mnt/C /dev/sdc1 ext4 rw,relatime,seclabel,stripe=32,data=ordered
... for /mnt/C there is no [/dir]
Note that btrfs uses the same if you mount btrfs subvolume.
+is_bind_mount() +{
- local _dev=$(findmnt -k -n -r -o SOURCE $1)
- local _dev_no_bind=$(findmnt -k -n -r -v -o SOURCE $1)
- if [[ $_dev = $_dev_no_bind ]]; then
return 1
- else
return 0
- fi
+}
+# We are looking to see if a part of dump path is mounted on some disk. +# But we don't want to look for bind mounted directories. So we should +# determine the real mountpoint, if the directory is a bind mounted +# directory. get_mntpoint_from_path() {
- echo $(df $1 | tail -1 | awk '{print $NF}')
- local _mnt=$(df $1 | tail -1 | awk '{print $NF}')
local _mnt=$(findmnt -n -o TARGET --target $1)
- if is_bind_mount $mnt; then
# bind mounted directory, we should find the real mount point
# where the device mounts.
for m in `findmnt -k -n -r -o TARGET $_dev_no_bind`; do
if ! is_bind_mount $m; then
echo $m
break
fi
done
The mountpoint created by bind operation is independent on previous mounts,
mount /mnt/sda1 /A mount --bind /A /B umount /A
is pretty valid situation and /B still points to the filesystem.
- else
echo $_mnt
- fi
}
I'm not sure if I good understand the original problem, but I'm sure that care about bind mounts is mistake ;-) If the problem is that dump is not in the filesystem root then you can try to check if the root of the filesystem is mounted somewhere else, but maybe the best would be avoid such fragile setup at all.
It is complicate for kdump, if we deal with the bind mounted. So in order to simplify the process, how about add the bind mounted path to the dump target, if the target is bind mounted?
Following is an example:
-bash-4.2# cat /etc/kdump.conf |grep ^path path /var/crash
-bash-4.2# findmnt /var | tail -n 1 | awk '{print $2}' /dev/mapper/atomicos-root[/ostree/deploy/rhel-atomic-host/var]
-bash-4.2# findmnt -v /var | tail -n 1 | awk '{print $2}' /dev/mapper/atomicos-root
Then we can found it that the real path of dumping core is /ostree/deploy/rhel-atomic-host/var/crash.
Here is the scratch path to fix this issue.
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index 6a7da8c..5b45714 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -345,6 +345,16 @@ kdump_install_net() { fi }
+adjust_dump_target() +{ + local _fstype=$1 _target=$2 _path=$3 + echo "$_fstype $_target" >> /tmp/$$-kdump.conf + + #erase the old path line, then insert the parsed path + sed -i "/^path/d" /tmp/$$-kdump.conf + echo "path $_path" >> /tmp/$$-kdump.conf +} + default_dump_target_install_conf() { local _target _fstype @@ -359,23 +369,23 @@ default_dump_target_install_conf()
_mntpoint=$(get_mntpoint_from_path $_save_path) _target=$(get_target_from_path $_save_path) - if [ "$_mntpoint" != "/" ]; then - _fstype=$(get_fs_type_from_target $_target) + _fstype=$(get_fs_type_from_target $_target)
+ if is_bind_mount $_mntpoint; then + _path=${_save_path##"$_mntpoint"} + _path=$(get_bind_mount_directory $_mntpoint)/$_path + + adjust_dump_target $_fstype $_target $_path + elif [ "$_mntpoint" != "/" ]; then if $(is_fs_type_nfs $_fstype); then kdump_install_net "$_target" _fstype="nfs" else _target=$(kdump_to_udev_name $_target) fi - - echo "$_fstype $_target" >> /tmp/$$-kdump.conf - _path=${_save_path##"$_mntpoint"}
- #erase the old path line, then insert the parsed path - sed -i "/^path/d" /tmp/$$-kdump.conf - echo "path $_path" >> /tmp/$$-kdump.conf - _path=${_save_path##"$_mntpoint"}
- #erase the old path line, then insert the parsed path - sed -i "/^path/d" /tmp/$$-kdump.conf - echo "path $_path" >> /tmp/$$-kdump.conf + adjust_dump_target $_fstype $_target $_path fi
} diff --git a/kdump-lib.sh b/kdump-lib.sh index cb1bab7..02b4253 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -86,6 +86,31 @@ get_root_fs_device() return }
+is_bind_mount() +{ + local _mntpoint=$(findmnt $1 | tail -n 1 | awk '{print $2}') + local _mntpoint_nofsroot=$(findmnt -v $1 | tail -n 1 | awk '{print $2}') + + if [[ $_mntpoint = $_mntpoint_nofsroot ]]; then + return 1 + else + return 0 + fi +} + +get_bind_mount_directory() +{ + local _mntpoint=$(findmnt $1 | tail -n 1 | awk '{print $2}') + local _mntpoint_nofsroot=$(findmnt -v $1 | tail -n 1 | awk '{print $2}') + + _mntpoint=${_mntpoint##$_mntpoint_nofsroot} + + _mntpoint=${_mntpoint#[} + _mntpoint=${_mntpoint%]} + + echo $_mntpoint +} + get_mntpoint_from_path() { echo $(df $1 | tail -1 | awk '{print $NF}')
Thanks Minfei
Karel
-- Karel Zak kzak@redhat.com http://karelzak.blogspot.com
On Sun, Feb 15, 2015 at 09:15:43PM +0800, Minfei Huang wrote:
[..]
I'm not sure if I good understand the original problem, but I'm sure that care about bind mounts is mistake ;-) If the problem is that dump is not in the filesystem root then you can try to check if the root of the filesystem is mounted somewhere else, but maybe the best would be avoid such fragile setup at all.
It is complicate for kdump, if we deal with the bind mounted. So in order to simplify the process, how about add the bind mounted path to the dump target, if the target is bind mounted?
Following is an example:
-bash-4.2# cat /etc/kdump.conf |grep ^path path /var/crash
-bash-4.2# findmnt /var | tail -n 1 | awk '{print $2}' /dev/mapper/atomicos-root[/ostree/deploy/rhel-atomic-host/var]
-bash-4.2# findmnt -v /var | tail -n 1 | awk '{print $2}' /dev/mapper/atomicos-root
Then we can found it that the real path of dumping core is /ostree/deploy/rhel-atomic-host/var/crash.
Here is the scratch path to fix this issue.
Minfei,
So what you are suggesting is that let us figure out where is the real disk and path with-in disk which is mounted on "path" as specified in kdump.conf.
I think that will work. So basically we will say that we will not ignore bind mounts instead resolve the "path" to map to real disk and path with-in disk. We just need to make sure that logic is generic enough that it can detect and sort out even complicated cases of bind mounts like multiple bind mounts.
mount /dev/foo / mount /dev/foo1 /var mount /dev/foo2 /var/crash
Thanks Vivek
On 02/16/15 at 07:44pm, Vivek Goyal wrote:
On Sun, Feb 15, 2015 at 09:15:43PM +0800, Minfei Huang wrote:
[..]
I'm not sure if I good understand the original problem, but I'm sure that care about bind mounts is mistake ;-) If the problem is that dump is not in the filesystem root then you can try to check if the root of the filesystem is mounted somewhere else, but maybe the best would be avoid such fragile setup at all.
It is complicate for kdump, if we deal with the bind mounted. So in order to simplify the process, how about add the bind mounted path to the dump target, if the target is bind mounted?
Following is an example:
-bash-4.2# cat /etc/kdump.conf |grep ^path path /var/crash
-bash-4.2# findmnt /var | tail -n 1 | awk '{print $2}' /dev/mapper/atomicos-root[/ostree/deploy/rhel-atomic-host/var]
-bash-4.2# findmnt -v /var | tail -n 1 | awk '{print $2}' /dev/mapper/atomicos-root
Then we can found it that the real path of dumping core is /ostree/deploy/rhel-atomic-host/var/crash.
Here is the scratch path to fix this issue.
Minfei,
So what you are suggesting is that let us figure out where is the real disk and path with-in disk which is mounted on "path" as specified in kdump.conf.
I think that will work. So basically we will say that we will not ignore bind mounts instead resolve the "path" to map to real disk and path with-in disk. We just need to make sure that logic is generic enough that it can detect and sort out even complicated cases of bind mounts like multiple bind mounts.
mount /dev/foo / mount /dev/foo1 /var mount /dev/foo2 /var/crash
Thanks Vivek
Hi, Vivek!
Yes, I think it can simplify our working on the bind mounted issue.
Finding the real disk and path with-in disk is a simple way to solve the bind mounted issue. It is complicated to sort out the mount point in 2nd kernel, if the 1st kernel uses bind mounted.
Thanks Minfei
On Wed, Feb 11, 2015 at 12:39:48PM +0100, Karel Zak wrote:
[..]
I'm not sure if I good understand the original problem, but I'm sure that care about bind mounts is mistake ;-) If the problem is that dump is not in the filesystem root then you can try to check if the root of the filesystem is mounted somewhere else, but maybe the best would be avoid such fragile setup at all.
Hi Karel,
So problem in a nutshell is figuring out where to dump the core when second kernel boots. As we are in second kernel, we have lost all the context of first kernel (including bind mount relations) and that means we need to sort out all the information in first kernel and store it so that kdump can parse it in second kernel.
Now what's the information we are looking for. Primarily we are looking for what disk do we want to dump to and what path with-in disk we should dump to.
For example, by default kdump dumps to /var/crash/ directory. Now one could bind mount /var/crash to say /dev/foo[/foo-subdir]. Now the question is when second kernel boots, where should we put crash dump.
Should it go in /var/crash of root disk or should it do in /foo-subdir of /dev/foo disk?
In simple case we could say that we always put it in /var/crash of root disk but one could argue that it should put it on /dev/foo disk in foo-subdir.
To handle more complicated case, kdump script will have to have find a way where given a path (/var/crash/ by default), it can traverse through whole path, figure out what components are mounted/bind mounted where and finally zoom on to final disk and path with-in disk and then ask dracut to mount that disk.
Thanks Vivek
If set the dump target "/var/crash", the kdump will save the core in /sysroot/var/crash, instead of /var/crash.
The mount point "/sysroot" is a special directory in Atomic, that it will conflict in 2nd kernel where kdump will mount the "/" on the mount point "/sysroot/".
To solve it, handle the Atomic kdump specially. If is the Atomic and kdump path is the subdirectory in "/var/", save the core to the path which specified in the /etc/kdump.conf directly.
Signed-off-by: Minfei Huang mhuang@redhat.com --- kdump-lib-initramfs.sh | 91 +++++++++++++++++++++++++++++++++++++++++++------- kdump-lib.sh | 9 +++++ 2 files changed, 88 insertions(+), 12 deletions(-)
diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 57b8304..8ccd72c 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -81,10 +81,74 @@ get_kdump_confs() fi }
-# dump_fs <mount point| device> -dump_fs() +dump_common_fs() +{ + local _mp=$1 + + # Remove -F in makedumpfile case. We don't want a flat format dump here. + [[ $CORE_COLLECTOR = *makedumpfile* ]] && CORE_COLLECTOR=`echo $CORE_COLLECTOR | sed -e "s/-F//g"` + + mkdir -p $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR || return 1 + + save_vmcore_dmesg_fs ${DMESG_COLLECTOR} "$_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/" + + echo "kdump: saving vmcore" + $CORE_COLLECTOR /proc/vmcore $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/vmcore-incomplete || return 1 + mv $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/vmcore-incomplete $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/vmcore + sync + + echo "kdump: saving vmcore complete" +} + +is_ostree() { + local _ostree=$(getarg ostree=) + if test -n "${_ostree}"; then + return 0 + fi + return 1 +} + +# find out the specified target in Atomic, like /var +atomic_specified_path() +{ + local _path="" + + for p in "var/" "/var/"; do + if [[ $KDUMP_PATH =~ "$p"* ]]; then + _path="/$p" + break + fi + done + echo $_path +} + +is_atomic_specified_path() { + local _path=$(atomic_specified_path) + if [[ "x" = "x$_path" ]]; then + echo 1 + else + echo 0 + fi +} + +# handle the special case Atomic, if the target is "/var". Due to the +# bind mounted directory /var, we hard code the real path of /var +# where the /var is in. +dump_atomic_rootfs() +{ + local _mp=$NEWROOT + local _bind_mp="$_mp/ostree/deploy/rhel-atomic-host" + + echo "kdump: saving to $_bind_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/" + mount -o remount,rw $_mp || return 1
+ dump_common_fs $_bind_mp + return $? +} + +dump_generic_fs() +{ local _dev=$(findmnt -k -f -n -r -o SOURCE $1) local _mp=$(findmnt -k -f -n -r -o TARGET $1)
@@ -95,22 +159,25 @@ dump_fs() return 1 fi
- # Remove -F in makedumpfile case. We don't want a flat format dump here. - [[ $CORE_COLLECTOR = *makedumpfile* ]] && CORE_COLLECTOR=`echo $CORE_COLLECTOR | sed -e "s/-F//g"` - echo "kdump: saving to $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/"
mount -o remount,rw $_mp || return 1 - mkdir -p $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR || return 1
- save_vmcore_dmesg_fs ${DMESG_COLLECTOR} "$_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/" + dump_common_fs $_mp
- echo "kdump: saving vmcore" - $CORE_COLLECTOR /proc/vmcore $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/vmcore-incomplete || return 1 - mv $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/vmcore-incomplete $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/vmcore - sync + return $? +}
- echo "kdump: saving vmcore complete" +# dump_fs <mount point| device> +dump_fs() +{ + + if is_ostree && is_atomic_specified_path && is_rootfs; then + dump_atomic_rootfs + else + dump_generic_fs $1 + fi + return $? }
save_vmcore_dmesg_fs() { diff --git a/kdump-lib.sh b/kdump-lib.sh index e4df376..81f2969 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -17,6 +17,15 @@ is_nfs_dump_target() grep -q "^nfs" /etc/kdump.conf }
+is_rootfs() +{ + if is_ssh_dump_target || is_nfs_dump_target; then + return 1 + else + return 0 + fi +} + is_raw_dump_target() { grep -q "^raw" /etc/kdump.conf
On 02/10/15 at 06:46pm, Minfei Huang wrote:
If set the dump target "/var/crash", the kdump will save the core in /sysroot/var/crash, instead of /var/crash.
The mount point "/sysroot" is a special directory in Atomic, that it will conflict in 2nd kernel where kdump will mount the "/" on the mount point "/sysroot/".
To solve it, handle the Atomic kdump specially. If is the Atomic and kdump path is the subdirectory in "/var/", save the core to the path which specified in the /etc/kdump.conf directly.
Minfei, it does not explain why it does not works originally, I will assume it is because dracut use /sysroot as NEWROOT mount point but there's already /sysroot, thus the rootfs mounting fails, right? Please explain it in changelog so that we will know the problem better.
I did not look the patch detail but I feel it is bad to add another special case.
Here is a line in dracut to set NEWROOT: Have you tried to provide some parameters to dracut so that it can use a different NEWROOT? modules.d/98systemd/dracut-mount.service:Environment=NEWROOT=/sysroot
Rethink about it, since dracut is use /sysroot as mount point, really atomic should choose another dir name instead of /sysroot which conflicts with dracut.
Jeremy and Colin?
Signed-off-by: Minfei Huang mhuang@redhat.com
kdump-lib-initramfs.sh | 91 +++++++++++++++++++++++++++++++++++++++++++------- kdump-lib.sh | 9 +++++ 2 files changed, 88 insertions(+), 12 deletions(-)
diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 57b8304..8ccd72c 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -81,10 +81,74 @@ get_kdump_confs() fi }
-# dump_fs <mount point| device> -dump_fs() +dump_common_fs() +{
- local _mp=$1
- # Remove -F in makedumpfile case. We don't want a flat format dump here.
- [[ $CORE_COLLECTOR = *makedumpfile* ]] && CORE_COLLECTOR=`echo $CORE_COLLECTOR | sed -e "s/-F//g"`
- mkdir -p $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR || return 1
- save_vmcore_dmesg_fs ${DMESG_COLLECTOR} "$_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/"
- echo "kdump: saving vmcore"
- $CORE_COLLECTOR /proc/vmcore $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/vmcore-incomplete || return 1
- mv $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/vmcore-incomplete $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/vmcore
- sync
- echo "kdump: saving vmcore complete"
+}
+is_ostree() {
- local _ostree=$(getarg ostree=)
- if test -n "${_ostree}"; then
return 0
- fi
- return 1
+}
+# find out the specified target in Atomic, like /var +atomic_specified_path() +{
- local _path=""
- for p in "var/" "/var/"; do
if [[ $KDUMP_PATH =~ "$p"* ]]; then
_path="/$p"
break
fi
- done
- echo $_path
+}
+is_atomic_specified_path() {
- local _path=$(atomic_specified_path)
- if [[ "x" = "x$_path" ]]; then
echo 1
- else
echo 0
- fi
+}
+# handle the special case Atomic, if the target is "/var". Due to the +# bind mounted directory /var, we hard code the real path of /var +# where the /var is in. +dump_atomic_rootfs() +{
local _mp=$NEWROOT
local _bind_mp="$_mp/ostree/deploy/rhel-atomic-host"
echo "kdump: saving to $_bind_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/"
mount -o remount,rw $_mp || return 1
dump_common_fs $_bind_mp
return $?
+}
+dump_generic_fs() +{ local _dev=$(findmnt -k -f -n -r -o SOURCE $1) local _mp=$(findmnt -k -f -n -r -o TARGET $1)
@@ -95,22 +159,25 @@ dump_fs() return 1 fi
# Remove -F in makedumpfile case. We don't want a flat format dump here.
[[ $CORE_COLLECTOR = *makedumpfile* ]] && CORE_COLLECTOR=`echo $CORE_COLLECTOR | sed -e "s/-F//g"`
echo "kdump: saving to $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/"
mount -o remount,rw $_mp || return 1
mkdir -p $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR || return 1
save_vmcore_dmesg_fs ${DMESG_COLLECTOR} "$_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/"
- dump_common_fs $_mp
- echo "kdump: saving vmcore"
- $CORE_COLLECTOR /proc/vmcore $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/vmcore-incomplete || return 1
- mv $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/vmcore-incomplete $_mp/$KDUMP_PATH/$HOST_IP-$DATEDIR/vmcore
- sync
- return $?
+}
- echo "kdump: saving vmcore complete"
+# dump_fs <mount point| device> +dump_fs() +{
- if is_ostree && is_atomic_specified_path && is_rootfs; then
dump_atomic_rootfs
- else
dump_generic_fs $1
- fi
- return $?
}
save_vmcore_dmesg_fs() { diff --git a/kdump-lib.sh b/kdump-lib.sh index e4df376..81f2969 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -17,6 +17,15 @@ is_nfs_dump_target() grep -q "^nfs" /etc/kdump.conf }
+is_rootfs() +{
- if is_ssh_dump_target || is_nfs_dump_target; then
return 1
- else
return 0
- fi
+}
is_raw_dump_target() { grep -q "^raw" /etc/kdump.conf -- 2.2.2
kexec mailing list kexec@lists.fedoraproject.org https://lists.fedoraproject.org/mailman/listinfo/kexec
Rethink about it, since dracut is use /sysroot as mount point, really atomic should choose another dir name instead of /sysroot which conflicts with dracut.
Without investigating closely, there's no conflict here, it's just confusing. In the initramfs, there exists a /sysroot that points to the physical storage.
Thus, there is a:
/sysroot/ostree/deploy/rhel-atomic-host/deploy/$checksum.$serial/sysroot
When booted, there is also a /sysroot which points to the physical storage.
But I don't believe there is a point where there is an actual conflict.
On 02/11/15 at 04:19pm, Colin Walters wrote:
Rethink about it, since dracut is use /sysroot as mount point, really atomic should choose another dir name instead of /sysroot which conflicts with dracut.
Without investigating closely, there's no conflict here, it's just confusing. In the initramfs, there exists a /sysroot that points to the physical storage.
Dracut is using /sysroot as mounting point for real root since the very beginning Also see below systemd doc (Sectin "Bootup in the Initial RAM Disk): http://www.freedesktop.org/software/systemd/man/bootup.html
It says " If the root device can be mounted at /sysroot, the sysroot.mount unit becomes active and initrd-root-fs.target is reached.
So there's many assumptions about /sysroot...
Thus, there is a:
/sysroot/ostree/deploy/rhel-atomic-host/deploy/$checksum.$serial/sysroot
When booted, there is also a /sysroot which points to the physical storage.
But I don't believe there is a point where there is an actual conflict.
On 12.02.2015 07:00, Dave Young wrote:
On 02/11/15 at 04:19pm, Colin Walters wrote:
Rethink about it, since dracut is use /sysroot as mount point, really atomic should choose another dir name instead of /sysroot which conflicts with dracut.
Without investigating closely, there's no conflict here, it's just confusing. In the initramfs, there exists a /sysroot that points to the physical storage.
Dracut is using /sysroot as mounting point for real root since the very beginning Also see below systemd doc (Sectin "Bootup in the Initial RAM Disk): http://www.freedesktop.org/software/systemd/man/bootup.html
It says " If the root device can be mounted at /sysroot, the sysroot.mount unit becomes active and initrd-root-fs.target is reached.
So there's many assumptions about /sysroot...
Note: initrd-root-fs.target also has an additional condition
ConditionPathExists=/etc/initrd-release
So, that on normal systems, the target is not activated.
Thus, there is a:
/sysroot/ostree/deploy/rhel-atomic-host/deploy/$checksum.$serial/sysroot
When booted, there is also a /sysroot which points to the physical storage.
But I don't believe there is a point where there is an actual conflict.
On 02/12/15 at 12:57pm, Harald Hoyer wrote:
On 12.02.2015 07:00, Dave Young wrote:
On 02/11/15 at 04:19pm, Colin Walters wrote:
Rethink about it, since dracut is use /sysroot as mount point, really atomic should choose another dir name instead of /sysroot which conflicts with dracut.
Without investigating closely, there's no conflict here, it's just confusing. In the initramfs, there exists a /sysroot that points to the physical storage.
Dracut is using /sysroot as mounting point for real root since the very beginning Also see below systemd doc (Sectin "Bootup in the Initial RAM Disk): http://www.freedesktop.org/software/systemd/man/bootup.html
It says " If the root device can be mounted at /sysroot, the sysroot.mount unit becomes active and initrd-root-fs.target is reached.
So there's many assumptions about /sysroot...
Note: initrd-root-fs.target also has an additional condition
ConditionPathExists=/etc/initrd-release
So, that on normal systems, the target is not activated.
There's below logic for ostree prepare root, not sure if this is the one in RHEL though https://github.com/GNOME/ostree/blob/master/src/switchroot/ostree-prepare-ro...
It setup /sysroot and leave it for systemd initrd-switch-root.target to take over
Looks like it is running in initrd?
But from the code it is restruct /sysroot so there seems no conflict.
Thus, there is a:
/sysroot/ostree/deploy/rhel-atomic-host/deploy/$checksum.$serial/sysroot
When booted, there is also a /sysroot which points to the physical storage.
But I don't believe there is a point where there is an actual conflict.
kexec mailing list kexec@lists.fedoraproject.org https://lists.fedoraproject.org/mailman/listinfo/kexec
On 02/12/15 at 02:00pm, Dave Young wrote:
On 02/11/15 at 04:19pm, Colin Walters wrote:
Rethink about it, since dracut is use /sysroot as mount point, really atomic should choose another dir name instead of /sysroot which conflicts with dracut.
Without investigating closely, there's no conflict here, it's just confusing. In the initramfs, there exists a /sysroot that points to the physical storage.
Dracut is using /sysroot as mounting point for real root since the very beginning Also see below systemd doc (Sectin "Bootup in the Initial RAM Disk): http://www.freedesktop.org/software/systemd/man/bootup.html
It says " If the root device can be mounted at /sysroot, the sysroot.mount unit becomes active and initrd-root-fs.target is reached.
So there's many assumptions about /sysroot...
Hi, Dave!
I think /sysroot is ok in the 2nd kernel. It fails to mount the root in the 2nd kernel, if we just use command "mount /dev/mapper/atomicos-root /sysroot". Due to the fact that "/" is a bind mounted directory, so we should mount the subdirectory to the mountpoint "/", if we want to mount the root directory.
I think the kdump works, if we find the subdirectory correctly.
Thanks Minfei
Thus, there is a:
/sysroot/ostree/deploy/rhel-atomic-host/deploy/$checksum.$serial/sysroot
When booted, there is also a /sysroot which points to the physical storage.
But I don't believe there is a point where there is an actual conflict.
On 02/11/15 at 04:19pm, Colin Walters wrote:
Rethink about it, since dracut is use /sysroot as mount point, really atomic should choose another dir name instead of /sysroot which conflicts with dracut.
Without investigating closely, there's no conflict here, it's just confusing. In the initramfs, there exists a /sysroot that points to the physical storage.
Thus, there is a:
/sysroot/ostree/deploy/rhel-atomic-host/deploy/$checksum.$serial/sysroot
When booted, there is also a /sysroot which points to the physical storage.
But I don't believe there is a point where there is an actual conflict.
Ok, so ostree-prepare-root will move physical root to somewhere under deploy/ then it will move a cooked sysroot dir to /sysroot. So maybe the right way is to do same in kdump initrd.
Thanks Dave