I just did a clean install of Fedora 7 from the live CD onto my laptop, which previously had a Fedora install upgraded from one of the F7 test releases, partitioned as suggested by anaconda (LVM, one swap partition, everything else under '/')
When reinstalling, I kept the partition layout and specifically told Anaconda *not* to reformat / (having booted in rescue mode beforehand, and removing everything but /home). Anaconda gave a warning that the leftover files might interfere with the installed system, which gave the impression that those files won't actually be removed during installation.
As it turns out, however, the old contents are completely gone (I'm trying out different recovery tools now to see if I could rescue some of the data). It's as if the live CD simply used dd to transfer the install image to the hard drive (in which case, how does it actually handle different partition layouts, e.g. /usr, /var, /home on separate partitions -- does it just move the files afterwards?)
As it stands it seems that the Live CD is a very dangerous tool, at least as long as 1) the default behaviour of Anaconda is to put everything under / 2) it does not carry more warnings about what it will do to the / partition during installation
Could someone let us know how the live CD actually performs its work? It would aid tremendously in the data recovery part. Would 'dd'-ing the entire partition to an external drive, and mounting it on a Windows computer (most recovery tools are unfortunately for that platform) preserve all the data required? I'm assuming that the new installation overwrote the same parts of the disk that was used to hold the OS in the previous install.
Thanks,
Michel Salim wrote:
I just did a clean install of Fedora 7 from the live CD onto my laptop, which previously had a Fedora install upgraded from one of the F7 test releases, partitioned as suggested by anaconda (LVM, one swap partition, everything else under '/')
When reinstalling, I kept the partition layout and specifically told Anaconda *not* to reformat / (having booted in rescue mode beforehand, and removing everything but /home). Anaconda gave a warning that the leftover files might interfere with the installed system, which gave the impression that those files won't actually be removed during installation.
I don't remember the specific warning, but if it is not clearly indicating that unselecting the format option on '/' is not allowed, then that is a bug that should be fixed either with better user messages, or alternate installation mechanisms.
I didn't have anything to do with what's in F7, though I did know enough about it to catch the fact that the 'formatting / filesystem' during the process was meaningless and time consuming, so that will be gone from from F8.
As it turns out, however, the old contents are completely gone (I'm trying out different recovery tools now to see if I could rescue some of the data). It's as if the live CD simply used dd to transfer the install image to the hard drive (in which case, how does it actually handle different partition layouts, e.g. /usr, /var, /home on separate partitions -- does it just move the files afterwards?)
As it stands it seems that the Live CD is a very dangerous tool, at least as long as
- the default behaviour of Anaconda is to put everything under /
- it does not carry more warnings about what it will do to the / partition
during installation
Could someone let us know how the live CD actually performs its work?
It takes a 4.0G ext3 fsimage from the cd, effectively dd's it to the chosen root volume, then does a resize2fs.
It
would aid tremendously in the data recovery part. Would 'dd'-ing the entire partition to an external drive, and mounting it on a Windows computer (most recovery tools are unfortunately for that platform) preserve all the data required? I'm assuming that the new installation overwrote the same parts of the disk that was used to hold the OS in the previous install.
As long as you haven't done much, data that was sitting in the first 4.0G of the filesystem will be gone, but data in the rest should be largely untouched. I have no idea how to go about trying to recover it (haven't used any windows tools), except asking for expensive professional date recovery help. Or if all I cared about was a text file, using 'strings' and 'grep' on chunks of the disk data.
-dmc
Douglas McClendon wrote:
Michel Salim wrote:
I just did a clean install of Fedora 7 from the live CD onto my laptop, which previously had a Fedora install upgraded from one of the F7 test releases, partitioned as suggested by anaconda (LVM, one swap partition, everything else under '/')
When reinstalling, I kept the partition layout and specifically told Anaconda *not* to reformat / (having booted in rescue mode beforehand, and removing everything but /home). Anaconda gave a warning that the leftover files might interfere with the installed system, which gave the impression that those files won't actually be removed during installation.
I don't remember the specific warning, but if it is not clearly indicating that unselecting the format option on '/' is not allowed, then that is a bug that should be fixed either with better user messages, or alternate installation mechanisms.
WOW. I was wrong. There is NO such message. That is a horrible bug. I may try to check to see if it's still in the latest anaconda, and provide some sort of simple patch later today.
Also, to answer your question more thoroughly than my first reply- Yes, after the dd, if there is a seperate /usr or other partitions, files are then copied from / to there. This is all very related to my turboLiveInst patch which I recently posted to livecd-list and anaconda-devel.
I'm surprised I don't remember hearing about this bug before. I had personally run into the same warning you saw, but that is just a general warning that has nothing to do with the livecd installer case specifically, and the livecd installer will stupidly let you just march along with the / fs not scheduled for formatting, even though it is going to anyway.
-dmc
I didn't have anything to do with what's in F7, though I did know enough about it to catch the fact that the 'formatting / filesystem' during the process was meaningless and time consuming, so that will be gone from from F8.
As it turns out, however, the old contents are completely gone (I'm trying out different recovery tools now to see if I could rescue some of the data). It's as if the live CD simply used dd to transfer the install image to the hard drive (in which case, how does it actually handle different partition layouts, e.g. /usr, /var, /home on separate partitions -- does it just move the files afterwards?)
As it stands it seems that the Live CD is a very dangerous tool, at least as long as
- the default behaviour of Anaconda is to put everything under /
- it does not carry more warnings about what it will do to the /
partition during installation
Could someone let us know how the live CD actually performs its work?
It takes a 4.0G ext3 fsimage from the cd, effectively dd's it to the chosen root volume, then does a resize2fs.
It
would aid tremendously in the data recovery part. Would 'dd'-ing the entire partition to an external drive, and mounting it on a Windows computer (most recovery tools are unfortunately for that platform) preserve all the data required? I'm assuming that the new installation overwrote the same parts of the disk that was used to hold the OS in the previous install.
As long as you haven't done much, data that was sitting in the first 4.0G of the filesystem will be gone, but data in the rest should be largely untouched. I have no idea how to go about trying to recover it (haven't used any windows tools), except asking for expensive professional date recovery help. Or if all I cared about was a text file, using 'strings' and 'grep' on chunks of the disk data.
-dmc
On 01/08/07, Douglas McClendon dmc.fedora@filteredperception.org wrote:
Douglas McClendon wrote:
Michel Salim wrote:
I just did a clean install of Fedora 7 from the live CD onto my laptop, which previously had a Fedora install upgraded from one of the F7 test releases, partitioned as suggested by anaconda (LVM, one swap
partition,
everything else under '/')
When reinstalling, I kept the partition layout and specifically told Anaconda *not* to reformat / (having booted in rescue mode beforehand, and removing everything but /home). Anaconda gave a warning that the
leftover
files might interfere with the installed system, which gave the impression that those files won't actually be removed during installation.
I don't remember the specific warning, but if it is not clearly indicating that unselecting the format option on '/' is not allowed, then that is a bug that should be fixed either with better user messages, or alternate installation mechanisms.
WOW. I was wrong. There is NO such message. That is a horrible bug. I may try to check to see if it's still in the latest anaconda, and provide some sort of simple patch later today.
Also, to answer your question more thoroughly than my first reply- Yes, after the dd, if there is a seperate /usr or other partitions, files are then copied from / to there. This is all very related to my turboLiveInst patch which I recently posted to livecd-list and anaconda-devel.
Uh. Does it ensure that the root partition is at least 4.0 GB in size, or will it just try to dd the image regardless? I've had esoteric partitioning in the past, with small /, and large /usr and /opt partitions.
I'm surprised I don't remember hearing about this bug before. I had
personally run into the same warning you saw, but that is just a general warning that has nothing to do with the livecd installer case specifically, and the livecd installer will stupidly let you just march along with the / fs not scheduled for formatting, even though it is going to anyway.
I guess we really need to have different types of updates, where major and potentially data-loss-causing changes need to be more exhaustively tested. No one probably bothered testing a live CD install to a non-formatted partition before..
Still waiting for the dd process to finish backing up the partition to an external drive; will report if anything is salvageable. Wishing I did not blow away the Windows partition, would have made recovery much easier.
While we're on the subject of making Anaconda changes, how about putting /home on a separate partition? The BSDs traditionally do that, I think.
Thanks,
Michel Salim wrote:
On 01/08/07, Douglas McClendon dmc.fedora@filteredperception.org wrote:
Douglas McClendon wrote: Also, to answer your question more thoroughly than my first reply- Yes, after the dd, if there is a seperate /usr or other partitions, files are then copied from / to there. This is all very related to my turboLiveInst patch which I recently posted to livecd-list and anaconda-devel.
Uh. Does it ensure that the root partition is at least 4.0 GB in size, or will it just try to dd the image regardless? I've had esoteric partitioning in the past, with small /, and large /usr and /opt partitions.
That it does. Though the aforementioned turboLiveInst patch improves upon that, in that the rootfs only really needs to be 2.1G, which is the size of the uncompressed data that lives in the 4.0G filesystem image. But as mentioned with that patch, that is still technically deficient for the case of separate /usr, where / needn't really even be 2.1G. The solution to that is to have an alternate file level copy installation mechanism, rather than the (fast) block level fsimage copy installation mechanism. This also fixes the potential problem of reintroducing support for non-ext3 (e.g. xfs) target root filesystems.
I'm surprised I don't remember hearing about this bug before. I had
personally run into the same warning you saw, but that is just a general warning that has nothing to do with the livecd installer case specifically, and the livecd installer will stupidly let you just march along with the / fs not scheduled for formatting, even though it is going to anyway.
I guess we really need to have different types of updates, where major and potentially data-loss-causing changes need to be more exhaustively tested. No one probably bothered testing a live CD install to a non-formatted partition before.
F7 was the _first_ fedora release to have a livecd installer. And that generic warning about choosing to not format '/', probably steered most people away from doing that. But I agree, this situation should definitely be a part of some sort of test matrix.
Still waiting for the dd process to finish backing up the partition to an external drive; will report if anything is salvageable. Wishing I did not blow away the Windows partition, would have made recovery much easier.
While we're on the subject of making Anaconda changes, how about putting /home on a separate partition? The BSDs traditionally do that, I think.
No doubt you mean by default. Personally I've long been a fan of the single large partition (to the extreme of going out of my way to put boot, swap and suspend2 areas on /). Though these days, I tend to favor the 'upgrades are never worth it', and 'wipegrades are the future'. So /home makes more sense for 'wipegrades'. Although with all the version specific cruft that ends up in your homedir (~/.gnome*, blabla) I tend to have a subdirectory under home, and when I wipegrade, I only keep the subdir. If I have any ~/.* files that are important enough, I keep a copy in the subdir, and a script to easily replace them after wipegrade (and gconftool2 to reintroduce all my desktop prefs). Just my strategy, ymmv.
-dmc
On 01/08/07, Douglas McClendon dmc.fedora@filteredperception.org wrote:
Michel Salim wrote:
On 01/08/07, Douglas McClendon dmc.fedora@filteredperception.org
wrote:
Douglas McClendon wrote: Also, to answer your question more thoroughly than my first
reply- Yes,
after the dd, if there is a seperate /usr or other partitions, files are then copied from / to there. This is all very related to my turboLiveInst patch
which
I recently posted to livecd-list and anaconda-devel.
Uh. Does it ensure that the root partition is at least 4.0 GB in size,
or
will it just try to dd the image regardless? I've had esoteric
partitioning
in the past, with small /, and large /usr and /opt partitions.
That it does. Though the aforementioned turboLiveInst patch improves upon that, in that the rootfs only really needs to be 2.1G, which is the size of the uncompressed data that lives in the 4.0G filesystem image. But as mentioned with that patch, that is still technically deficient for the case of separate /usr, where / needn't really even be 2.1G. The solution to that is to have an alternate file level copy installation mechanism, rather than the (fast) block level fsimage copy installation mechanism. This also fixes the potential problem of reintroducing support for non-ext3 (e.g. xfs) target root filesystems.
I'm surprised I don't remember hearing about this bug before. I had
personally run into the same warning you saw, but that is just a general warning
that
has nothing to do with the livecd installer case specifically, and the
livecd
installer will stupidly let you just march along with the / fs not scheduled for formatting, even though it is going to anyway.
I guess we really need to have different types of updates, where major
and
potentially data-loss-causing changes need to be more exhaustively
tested.
No one probably bothered testing a live CD install to a non-formatted partition before.
F7 was the _first_ fedora release to have a livecd installer. And that generic warning about choosing to not format '/', probably steered most people away from doing that. But I agree, this situation should definitely be a part of some sort of test matrix.
Still waiting for the dd process to finish backing up the partition to
an
external drive; will report if anything is salvageable. Wishing I did
not
blow away the Windows partition, would have made recovery much easier.
While we're on the subject of making Anaconda changes, how about putting /home on a separate partition? The BSDs traditionally do that, I think.
No doubt you mean by default. Personally I've long been a fan of the single large partition (to the extreme of going out of my way to put boot, swap and suspend2 areas on /). Though these days, I tend to favor the 'upgrades are never worth it', and 'wipegrades are the future'. So /home makes more sense for 'wipegrades'. Although with all the version specific cruft that ends up in your homedir (~/.gnome*, blabla) I tend to have a subdirectory under home, and when I wipegrade, I only keep the subdir. If I have any ~/.* files that are important enough, I keep a copy in the subdir, and a script to easily replace them after wipegrade (and gconftool2 to reintroduce all my desktop prefs). Just my strategy, ymmv.
That's my strategy. Well, wipe the dots (apart from .gaim and a few others) and keep Documents, Music, Pictures, etc. It gets a bit messy because there are things in .gnome2 you definitely don't want to wipe too (epiphany puts its bookmarks there if I'm not mistaken, and there's also Rhythmbox). Writing a small script that chattr +i the important directories under the hidden dirs and then wipe the remainders would do the trick in a more systematic way, I suppose.
I guess the partition split does not even have to be / and /home, it just has to be something that logically maps to RPM-managed vs non-RPM-managed. So users that need persistent home directories (local users, basically) should have home dirs in /local/home/ or something.
Anyway, hard lesson learned. I wonder how Ubuntu does their live CD install. openSUSE is introducing one too, hopefully for their users the same bug does not occur.
Rahul Sundaram wrote:
Michel Salim wrote:
Anyway, hard lesson learned. I wonder how Ubuntu does their live CD install. openSUSE is introducing one too, hopefully for their users the same bug does not occur.
Pretty much every live cd does a dd afaik.
ubuntu 7.04 keeps their rootfs natively on squashfs (versus fedora which keeps it natively on an ext3 image which lives on a squashfs).
Therefore they can't be doing a dd for the install.
The reason for the difference is because ubuntu uses unionfs for copy-on-write, whereas fedora uses devicemapper-snapshot for copy-on-write.
-dmc
On 01/08/07, Douglas McClendon dmc.fedora@filteredperception.org wrote:
Rahul Sundaram wrote:
Michel Salim wrote:
Anyway, hard lesson learned. I wonder how Ubuntu does their live CD install. openSUSE is introducing one too, hopefully for their users the same bug does not occur.
Pretty much every live cd does a dd afaik.
ubuntu 7.04 keeps their rootfs natively on squashfs (versus fedora which keeps it natively on an ext3 image which lives on a squashfs).
Therefore they can't be doing a dd for the install.
Any reason why dd is preferred over, say, tar? The latter would be much safer. And files would not have to be moved to their final partitions after 'dd' too.
Michel Salim wrote:
On 01/08/07, Douglas McClendon dmc.fedora@filteredperception.org wrote:
Rahul Sundaram wrote:
Michel Salim wrote:
Anyway, hard lesson learned. I wonder how Ubuntu does their live CD install. openSUSE is introducing one too, hopefully for their users the same bug does not occur.
Pretty much every live cd does a dd afaik.
ubuntu 7.04 keeps their rootfs natively on squashfs (versus fedora which keeps it natively on an ext3 image which lives on a squashfs).
Therefore they can't be doing a dd for the install.
Any reason why dd is preferred over, say, tar? The latter would be much safer. And files would not have to be moved to their final partitions after 'dd' too.
seeking (cdrom and disk). Lots slower (5X?). I suspect that file level copy (tar) will be the long term answer for the flexible general case. Though I also suspect the much faster dd will be part of the long term answer as well for the typical case (i.e. formatting / as ext3, and no separate /usr).
For example, on my system, using the existing 4.0G dd, the copy takes 299s. Using my turboLiveInst patch, I can shave that down to 250s. Using tar however to copy the 85896 files, took 1247s. And you also need to throw in another 30s or so for the format that isn't required in the dd case. Maybe(??) there is some more efficient way to copy those 85K files than the ( tar cpsf - | tar xpsf - ) that I used for that trial.
Lastly safety has nothing to do with the tar vs dd issue. The safety issue had to do with the bug of the user interface not accurately reflecting what was happening. fyi, I did go ahead and file the bug, and even submit a patch which might fix it.
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=250301
That doesn't mean of course that you don't have every right and justification to be cursing the bits of the F7 livecd while watching it spark in a microwave.
-dmc
-
Douglas McClendon -
Michel Salim -
Rahul Sundaram