Hi,
I use LVM on top of LUKS.
My entire root file system is on a logical volume. I have no seperate home or swap partitions.
I would like to know how can I clone this logical volume to my external hard disk.
I have seen commands like :
dd if=/dev/vg/root_lv of=/mnt/exthdd bs=64k conv=noerror,sync status=progress
I have a couple of questions about this:
1) Why use noerror and sync?
noerror might give me a corrupted image and sync is just going to increase the size of my image with padding.
Why use them?
2) What is rationale behind block size?
Will using 4M block size instead of 64K give me a different image? I need my image to be an EXACT COPY.
Need your help.
Thanks.
On 11/22/20 4:08 PM, Sreyan Chakravarty wrote:
I would like to know how can I clone this logical volume to my external hard disk.
Hi,
You really should use a proper tool for the job. Something like partclone. That way you don't waste time & storage since you'll be saving *only* the used blocks. I mostly use CloneZilla (that uses partclone) but you can install partclone [1] in Fedora.
Also, for your root filesystem I'd rather boot off externenal media (USB/CD) and then perform the image backup (with volume-group deactivated and so on).
[1] https://linuxconfig.org/how-to-use-partclone-to-create-a-smart-partition-bac...
On Mon, Nov 23, 2020 at 2:02 AM Jorge Fábregas jorge.fabregas@gmail.com wrote:
You really should use a proper tool for the job. Something like partclone. That way you don't waste time & storage since you'll be saving *only* the used blocks. I mostly use CloneZilla (that uses partclone) but you can install partclone [1] in Fedora.
Please note I will not be cloning my partition but just the contents of the root logical volume.
The partition will still exist.
The reason I am doing this is because I want to move to thin LVMs, rather than the old CoW LVM.
Is partclone the write option? Or am I better off with dd ?
On 11/22/20 11:12 PM, Sreyan Chakravarty wrote:
On Mon, Nov 23, 2020 at 2:02 AM Jorge Fábregas <jorge.fabregas@gmail.com mailto:jorge.fabregas@gmail.com> wrote:
You really should use a proper tool for the job. Something like partclone. That way you don't waste time & storage since you'll be saving *only* the used blocks. I mostly use CloneZilla (that uses partclone) but you can install partclone [1] in Fedora.
Please note I will not be cloning my partition but just the contents of the root logical volume.
"partition" is the usual container, but these programs will copy a filesystem from whatever block device you point them at. There is no significant difference in this case between a partition and a logical volume.
The partition will still exist.
The reason I am doing this is because I want to move to thin LVMs, rather than the old CoW LVM.
Is partclone the write option? Or am I better off with dd ?
Particularly since you want to move the data to a thin LVM, you need to use something that understands the filesystem. If you use dd, then all the unused blocks will still be copied and take up space in the new LVM partition. You can use partclone or e2image (just make sure you use the option to include data) or clonezilla. I would suggest also doing a full tar backup of the filesystem before trying anything else, just in case.
On Mon, Nov 23, 2020 at 1:38 PM Samuel Sieb samuel@sieb.net wrote:
You can use partclone or e2image (just make sure you use the option to include data) or clonezilla.
I thought partclone is a part of Clonezilla.
I would suggest also doing a full tar backup of the filesystem before trying anything else, just in case.
Can you please tell me how I can do a full tar backup ?
On 11/23/20 12:12 AM, Sreyan Chakravarty wrote:
On Mon, Nov 23, 2020 at 1:38 PM Samuel Sieb <samuel@sieb.net mailto:samuel@sieb.net> wrote:
You can use partclone or e2image (just make sure you use the option to include data) or clonezilla.
I thought partclone is a part of Clonezilla.
No, clonezilla uses partclone.
I would suggest also doing a full tar backup of the filesystem before trying anything else, just in case.
Can you please tell me how I can do a full tar backup ?
tar cvfj /external/drive/myroot.tbz --one-file-system /
On 11/22/20 12:08 PM, Sreyan Chakravarty wrote:
I would like to know how can I clone this logical volume to my external hard disk.
I have seen commands like :
dd if=/dev/vg/root_lv of=/mnt/exthdd bs=64k conv=noerror,sync status=progress
I have a couple of questions about this:
- Why use noerror and sync?
noerror might give me a corrupted image and sync is just going to increase the size of my image with padding.
Why use them?
I don't know where you got that command from, but "noerror" would generally be used if the source device is failing. And yes, in that case, you might get a corrupted image. I think "sync" is related to that. If there's an error, you might get a short read, so "sync" makes sure a full block is written. Otherwise, the copied filesystem would be completely messed up.
- What is rationale behind block size?
Will using 4M block size instead of 64K give me a different image? I need my image to be an EXACT COPY.
The block size won't change the resulting image. It's just that the default block size of 512 bytes can be quite inefficient. Using a larger block reduces the CPU time used in system calls.
But as Jorge said, this is probably not the right tool for what you want to do.
On Mon, Nov 23, 2020 at 5:37 AM Samuel Sieb samuel@sieb.net wrote:
I don't know where you got that command from, but "noerror" would generally be used if the source device is failing. And yes, in that case, you might get a corrupted image.
The command came from here:
https://www.cyberciti.biz/faq/unix-linux-dd-create-make-disk-image-commands/
So it's better to use just "sync" and not "noerror" right ?
The block size won't change the resulting image. It's just that the default block size of 512 bytes can be quite inefficient. Using a larger block reduces the CPU time used in system calls.
Ok so it makes no difference, and I will use whatever is most efficient on my hardware.
But as Jorge said, this is probably not the right tool for what you want to do.
But again, I just want to copy the contents of the root logical volume, and not the partition itself.
The reason I am doing this is because I want to move to the newer thin LVMs. The partition will remain just the root logical volume will be deleted and then recreated as a thin logical volume.
Is partclone still the best option in the above scenario ?
On Mon, Nov 23, 2020 at 01:38:34AM +0530, Sreyan Chakravarty wrote:
Hi,
I use LVM on top of LUKS.
My entire root file system is on a logical volume. I have no seperate home or swap partitions.
I would like to know how can I clone this logical volume to my external hard disk.
I have seen commands like :
dd if=/dev/vg/root_lv of=/mnt/exthdd bs=64k conv=noerror,sync status=progress
I have a couple of questions about this:
- Why use noerror and sync?
noerror might give me a corrupted image and sync is just going to increase the size of my image with padding.
Why use them?
- What is rationale behind block size?
Will using 4M block size instead of 64K give me a different image? I need my image to be an EXACT COPY.
Does dd write a complete final block to the destination. If so, the image would not be an exact copy unless the source size is an exact multiple of the blocksize. Gdisk reports my root filesystem is 240197632 1K blocks long.
$ factor 240197632 240197632: 2 2 2 2 2 2 2 2 2 2 2 2 2 109 269
So 2^13, or 8KB is the largest block size where input and output would match.
Again, this assumes dd writes a full final block. Possibly a bad assumption
Jon
I agree with the suggestions to use a tool appropriate for this task, and that dd isn't likely what you want. I like dd/ddrescue for the specific case of data recovery. But for data replication, it's better to use something that's intended for that purpose.
Btrfs, it's either the seed/sprout feature. Or you can get finer granularity with send/receive. Seed/sprout replication results in the volume UUID's being unique, and duplicates at the Btrfs block group level (a group of extents). It's quite fast, and skips over unused/unallocated areas. send/receive is a bit different, there is a stream format that isn't an exact copy of the on-disk representation but it will copy all files and their metadata including permissions, owner, date/time stamps and xattr. Both work on mounted file systems.
xfs has xfs_copy
ext4 I think it has some kind of dump facility but the man page for dumpe2fs doesn't look like it does that. So I'm not sure.
And of course file copy tools are OK as well, though they sometimes come with the burden/risk of many extra options.
--- Chris Murphy
On Mon, Nov 23, 2020 at 9:18 AM Chris Murphy lists@colorremedies.com wrote:
I agree with the suggestions to use a tool appropriate for this task, and that dd isn't likely what you want. I like dd/ddrescue for the specific case of data recovery. But for data replication, it's better to use something that's intended for that purpose.
Well again, is partclone truly the best option if I am not cloning a partition ?
I am cloning a logical volume and not a partition. Will partclone help ?
Btrfs, it's either the seed/sprout feature. Or you can get finer granularity with send/receive. Seed/sprout replication results in the volume UUID's being unique, and duplicates at the Btrfs block group level (a group of extents). It's quite fast, and skips over unused/unallocated areas. send/receive is a bit different, there is a stream format that isn't an exact copy of the on-disk representation but it will copy all files and their metadata including permissions, owner, date/time stamps and xattr. Both work on mounted file systems.
xfs has xfs_copy
ext4 I think it has some kind of dump facility but the man page for dumpe2fs doesn't look like it does that. So I'm not sure.
I don't use btrfs or xfs, and I have never heard of dumpe2fs, but I will look into it.
Again, just cloning a logical volume and not a full partition.
And of course file copy tools are OK as well, though they sometimes come with the burden/risk of many extra options.
I really don't want to use the copy options for something like cloning.
On Sun, Nov 22, 2020 at 08:47:33PM -0700, Chris Murphy wrote:
ext4 I think it has some kind of dump facility but the man page for dumpe2fs doesn't look like it does that. So I'm not sure.
dumpe2fs is for dumping the filesystem information for ext2/3/4. The package name for the filesystem dump/restore is appropriately named 'dump' and contains a 'dump' and 'restore' executable. The advantages of being there before the others. :)
'dump' is probably the most appropriate tool to capture an ext4 volume to be converted to a thin-provisioned volume. Tar could also work as long as you capture all the extended (selinux) attributes, which I don't believe were described in the other post. dump/restore will only copy the data (and not unused blocks) and it reads the filesystem directly rather than going through the VFS layer.
On Mon, Nov 23, 2020 at 9:27 AM Jonathan Billings billings@negate.org wrote:
'dump' is probably the most appropriate tool to capture an ext4 volume to be converted to a thin-provisioned volume. Tar could also work as long as you capture all the extended (selinux) attributes, which I don't believe were described in the other post. dump/restore will only copy the data (and not unused blocks) and it reads the filesystem directly rather than going through the VFS layer.
I have never used dumpe2fs.
How does it compare to partclone ?
On 11/22/20 9:31 PM, Jon LaBadie wrote:
On Mon, Nov 23, 2020 at 01:38:34AM +0530, Sreyan Chakravarty wrote:
Hi,
I use LVM on top of LUKS.
My entire root file system is on a logical volume. I have no seperate home or swap partitions.
I would like to know how can I clone this logical volume to my external hard disk.
I have seen commands like :
dd if=/dev/vg/root_lv of=/mnt/exthdd bs=64k conv=noerror,sync status=progress
I have a couple of questions about this:
- Why use noerror and sync?
noerror might give me a corrupted image and sync is just going to increase the size of my image with padding.
Why use them?
- What is rationale behind block size?
Will using 4M block size instead of 64K give me a different image? I need my image to be an EXACT COPY.
Does dd write a complete final block to the destination. If so, the image would not be an exact copy unless the source size is an exact multiple of the blocksize.
Unless "conv=sync" is used, it does not. Without "conv=sync", when the input reaches EOF, dd will write whatever partial block has been read. If you _do_ use "conv=sync", then that output block will be zero-padded to the obs size. If the output is to a device and not a file, that might cause a superfluous "out of space" message and a failure exit code. If output is to a file, then the resulting file could be larger than the source.
The status message from dd indicates the number of complete+partial blocks read and written, e.g.: 1201+1 records in 1202+0 records out showing an output block that was padded to the full block size.
On Mon, Nov 23, 2020 at 9:45 AM Robert Nichols rnicholsNOSPAM@comcast.net wrote:
Unless "conv=sync" is used, it does not. Without "conv=sync", when the input reaches EOF, dd will write whatever partial block has been read. If you _do_ use "conv=sync", then that output block will be zero-padded to the obs size. If the output is to a device and not a file, that might cause a superfluous "out of space" message and a failure exit code. If output is to a file, then the resulting file could be larger than the source.
The status message from dd indicates the number of complete+partial blocks read and written, e.g.: 1201+1 records in 1202+0 records out showing an output block that was padded to the full block size.
So it's better to use "sync" and not "noerror", right ?
On 11/22/20 11:27 PM, Sreyan Chakravarty wrote:
On Mon, Nov 23, 2020 at 9:45 AM Robert Nichols <rnicholsNOSPAM@comcast.net mailto:rnicholsNOSPAM@comcast.net> wrote:
Unless "conv=sync" is used, it does not. Without "conv=sync", when the input reaches EOF, dd will write whatever partial block has been read. If you _do_ use "conv=sync", then that output block will be zero-padded to the obs size. If the output is to a device and not a file, that might cause a superfluous "out of space" message and a failure exit code. If output is to a file, then the resulting file could be larger than the source. The status message from dd indicates the number of complete+partial blocks read and written, e.g.: 1201+1 records in 1202+0 records out showing an output block that was padded to the full block size.
So it's better to use "sync" and not "noerror", right ?
No. Unless you are trying to recover a failing drive, you would not want either of those.
On Mon, Nov 23, 2020 at 1:37 PM Samuel Sieb samuel@sieb.net wrote:
No. Unless you are trying to recover a failing drive, you would not want either of those.
So do I use partclone or dd ?
On 11/23/20 2:08 AM, Sreyan Chakravarty wrote:
On Mon, Nov 23, 2020 at 1:37 PM Samuel Sieb <samuel@sieb.net mailto:samuel@sieb.net> wrote:
No. Unless you are trying to recover a failing drive, you would not want either of those.
So do I use partclone or dd ?
partclone understands filesystems and will avoid copying the free space.
dd just copies the entire device (drive, partition, LVM logical volume, decrypted container, ...) bit-for-bit.
If you have a broken filesystem or are trying to examine deleted files or do other forensic analysis, then "dd" is the appropriate choice. Otherwise, "partclone" or other filesystem-aware tools are more efficient. That is particularly true if the destination is on an SSD, where you really do _not_ want to be storing the content of all the free space.
On Mon, Nov 23, 2020 at 9:02 AM Jon LaBadie jonfu@jgcomp.com wrote:
Does dd write a complete final block to the destination. If so, the image would not be an exact copy unless the source size is an exact multiple of the blocksize.
I don't know. I have not tried it yet.
How would I verify something like that?
Gdisk reports my root filesystem is 240197632 1K blocks long.
$ factor 240197632 240197632: 2 2 2 2 2 2 2 2 2 2 2 2 2 109 269
So 2^13, or 8KB is the largest block size where input and output would match.
I don't quite follow the logic.
Can you please elaborate ?
On 11/22/20 9:08 PM, Sreyan Chakravarty wrote:
dd if=/dev/vg/root_lv of=/mnt/exthdd bs=64k conv=noerror,sync status=progress
- Why use noerror and sync?
noerror might give me a corrupted image and sync is just going to increase the size of my image with padding.
Indeed, noerror and sync are both bad advice, do not use them.
- What is rationale behind block size?
Will using 4M block size instead of 64K give me a different image? I need my image to be an EXACT COPY.
Block size is for efficiency and 64k is better than 512, but since we are in 2020 go for 4M, it is the best option. The copy will be perfect even if the total size is not a perfect multiple of blocksize, dd is smart.
Regards.
Just for info. I've maintained the g4l disk imaging project since 2004, and the basic line it uses for doing a partition/disk image locally is.
(dd bs=1M if=$localback 2>/dev/null |jetcat-mod -f 5000 -p $readsize 2>$progout |lzop -c - >/mnt/local$localpath$localimagename) &
It boots from a cd or flash or from grub menu as an option and runs in ram, so partitions are not mounted. Uses a dialog text interface.
The above information variables are gotten from dialog script. This is the basic required parts to make image - The removed stuff is to have a running dialog process screen that displays why the copy happens in background. The /mnt/local points to a partition or disk other than the one being imaged. The $localback would be the /dev/sda1 or similar being backed up for partition 1.
dd bs=1M if=$localback |lzop -c - > /mnt/local$localpath$localimagename
I use lzop for compression of image, since it gives good compress and is considerable faster than gzip.
Note: Since it does copy all blocks, it is best to clear the currently unused blocks before hand. Long ago, did an image of a disk with clean install of Fedora 3 on 80GB disk. Image created was 12G in size compress. Cleared free space, and redid imaged, and size dropped to 2.5G. Compression does depend on what is on disk..
use bs=1M did some testing, and found using bigger sizes made little or no difference. Too small would make it slower..
Use to use it in my classroom all the time to back and reimage systems, but retired in 2017 with 36+ years teaching computer science at College..
Well, good luck and be Safe.
On 23 Nov 2020 at 9:12, Roberto Ragusa wrote:
Subject: Re: Clone logical volume using dd To: users@lists.fedoraproject.org From: Roberto Ragusa mail@robertoragusa.it Date sent: Mon, 23 Nov 2020 09:12:32 +0100 Send reply to: Community support for Fedora users users@lists.fedoraproject.org
On 11/22/20 9:08 PM, Sreyan Chakravarty wrote:
dd if=/dev/vg/root_lv of=/mnt/exthdd bs=64k conv=noerror,sync status=progress
- Why use noerror and sync?
noerror might give me a corrupted image and sync is just going to increase the size of my image with padding.
Indeed, noerror and sync are both bad advice, do not use them.
- What is rationale behind block size?
Will using 4M block size instead of 64K give me a different image? I need my image to be an EXACT COPY.
Block size is for efficiency and 64k is better than 512, but since we are in 2020 go for 4M, it is the best option. The copy will be perfect even if the total size is not a perfect multiple of blocksize, dd is smart.
Regards.
Roberto Ragusa mail at robertoragusa.it
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
On 11/23/20 1:02 AM, Michael D. Setzer II via users wrote:
dd bs=1M if=$localback |lzop -c - > /mnt/local$localpath$localimagename
I use lzop for compression of image, since it gives good compress and is considerable faster than gzip.
Note: Since it does copy all blocks, it is best to clear the currently unused blocks before hand. Long ago, did an image of a disk with clean install of Fedora 3 on 80GB disk. Image created was 12G in size compress. Cleared free space, and redid imaged, and size dropped to 2.5G. Compression does depend on what is on disk..
This is still not suitable for his purpose. It's not about the compressed size, it's about the number of blocks copied. Using dd, you are still copying *all* the unused blocks which will be really bad for a thin LV which is the intended destination.
On 23 Nov 2020 at 12:50, Samuel Sieb wrote:
Subject: Re: Clone logical volume using dd To: users@lists.fedoraproject.org From: Samuel Sieb samuel@sieb.net Date sent: Mon, 23 Nov 2020 12:50:46 -0800 Send reply to: Community support for Fedora users users@lists.fedoraproject.org
On 11/23/20 1:02 AM, Michael D. Setzer II via users wrote:
dd bs=1M if=$localback |lzop -c - > /mnt/local$localpath$localimagename
I use lzop for compression of image, since it gives good compress and is considerable faster than gzip.
Note: Since it does copy all blocks, it is best to clear the currently unused blocks before hand. Long ago, did an image of a disk with clean install of Fedora 3 on 80GB disk. Image created was 12G in size compress. Cleared free space, and redid imaged, and size dropped to 2.5G. Compression does depend on what is on disk..
This is still not suitable for his purpose. It's not about the compressed size, it's about the number of blocks copied. Using dd, you are still copying *all* the unused blocks which will be really bad for a thin LV which is the intended destination.
The difference between a bit level image and an image that has to understand all aspects of the partition info. If the unused data blocks are cleared by filling with nulls, it compress to nothing. The resulting images are very similar in size.
Agree, it takes longer to make the image, since it does have to read all blocks of the disk/partition, but it copies all data. Using a program that determines what needs to be backed up and what doesn't may or may not copy everything at some point.
With the disk bit level images, one could restore to a new disk, and not have to do any prep of creating the partition setup, since that was already included in the image. With partitions, that isn't an issue.
I had linux and windows on my classroom machines, and sometimes imaged one system to all the other machines using udpcast with bit level image. Also, had options to be able to restore the windows image in about 10 minutes to its partition from the local extra partition.
So, just another option.
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
+------------------------------------------------------------+ Michael D. Setzer II - Computer Science Instructor (Retired) mailto:mikes@guam.net mailto:msetzerii@gmail.com Guam - Where America's Day Begins G4L Disk Imaging Project maintainer http://sourceforge.net/projects/g4l/ +------------------------------------------------------------+
On 11/23/20 1:35 PM, Michael D. Setzer II wrote:
On 23 Nov 2020 at 12:50, Samuel Sieb wrote:
This is still not suitable for his purpose. It's not about the compressed size, it's about the number of blocks copied. Using dd, you are still copying *all* the unused blocks which will be really bad for a thin LV which is the intended destination.
The difference between a bit level image and an image that has to understand all aspects of the partition info. If the unused data blocks are cleared by filling with nulls, it compress to nothing. The resulting images are very similar in size.
Agree, it takes longer to make the image, since it does have to read all blocks of the disk/partition, but it copies all data. Using a program that determines what needs to be backed up and what doesn't may or may not copy everything at some point.
With the disk bit level images, one could restore to a new disk, and not have to do any prep of creating the partition setup, since that was already included in the image. With partitions, that isn't an issue.
I had linux and windows on my classroom machines, and sometimes imaged one system to all the other machines using udpcast with bit level image. Also, had options to be able to restore the windows image in about 10 minutes to its partition from the local extra partition.
For general imaging, dd is fine. But you need to read the context here. He wants to make an image of the current filesystem in order to write it back to a thin LV. For that purpose, you only want to be writing the used blocks. Otherwise, you will be wasting all the space and the thin LV becomes useless. I suppose if there is enough space available, you could run a trim afterwards to get the space back, but that's really inefficient.
On Mon, Nov 23, 2020 at 5:12 PM Samuel Sieb samuel@sieb.net wrote:
For general imaging, dd is fine. But you need to read the context here. He wants to make an image of the current filesystem in order to write it back to a thin LV. For that purpose, you only want to be writing the used blocks. Otherwise, you will be wasting all the space and the thin LV becomes useless. I suppose if there is enough space available, you could run a trim afterwards to get the space back, but that's really inefficient.
I ended up using partclone. Worked like a charm. I too believe dd would be too inefficient, also not sure if thin LVMs would have liked a dd image.
On Mon, Nov 23, 2020 at 4:02 AM Michael D. Setzer II via users < users@lists.fedoraproject.org> wrote:
dd bs=1M if=$localback |lzop -c - > /mnt/local$localpath$localimagename
I use lzop for compression of image, since it gives good compress and is considerable faster than gzip.
Note: Since it does copy all blocks, it is best to clear the currently unused blocks before hand. Long ago, did an image of a disk with clean install of Fedora 3 on 80GB disk. Image created was 12G in size compress. Cleared free space, and redid imaged, and size dropped to 2.5G. Compression does depend on what is on disk..
I did not know about lzop. Thanks.