Folks,
Please consider the following two patches for inclusion in the anaconda and pykickstart sources. The patches provide three features, described below, related to hard drive installations. Together these features enable a kickstart and/or an exploded tree installation image to be placed on a portable storage device, such as a USB flash memory or hard drive device, and the device to be used to install one or more systems with unknown hard drive configuration. In effect, with this patch, installing from a portable storage device has the same ease of use characteristics as installing from a CD, with the added benefit that kickstart and install files can be modified directly on the device as needed, without the overhead of creating new iso images and burning CDs.
The anaconda patch was created using version 11.2.0.9 of the sources. The pykickstart patch was created using version 0.36. We can provide an updated patches against the most current versions, if helpful. Given the rate at which sources are changing, we decided to hold off until someone in anaconda has had a chance to review the proposed patches and suggest changes, if needed.
DOWNLOAD PATCHES ----------------
* The anaconda version 11.2.0.9 patch can be downloaded from http://www.abodiosoftware.com/patches/anaconda-11.2.0.9_abodio.patch
* The pykickstart version 0.36 patch can be downloaded from http://www.abodiosoftware.com/patches/pykickstart-0.36_abodio.patch
PATCH FEATURES --------------
1. Partition auto-detection. The patch allows the partition to be left unspecified when using the harddrive method for kickstart and install locations, and the item (kickstart or installtree) to be located by iterating through available partitions. Specifically, the kickstart and method options can now be specified as follows on the command line (note the empty partition section between the two colons):
ks=hd::/path/to/ks.cfg, and method=hd::/path/to/install/files
Within the kickstart file, a hard drive install can be specified as follows (note the lack of a --partition directive):
harddrive --dir=/path/to/install/files
2. Exploded tree support. The patch enables harddrive installation from an exploded tree, analogous to NFS installations. As a side effect, it also provides support for a RHupdates folder for an exploded tree installation.
3. Device modification protection. By default, the patch removes partitions on the harddrive containing the kickstart and/or install iso/tree from the default set to be formatted/partitioned. This can be overridden manually within the installer UI or by kickstart. Likewise, the install/kickstart harddrive is removed as a candidate for boot record modifications. This can similarly be overridden by kickstart.
IMPLEMENTATION DETAILS ---------------------- 1. Partition auto-detection
Previously, loader.c called the setMethodFromCmdLine()function as soon as the "method=" argument was seen. This can't be done anymore because now the partition might not be specified, and loader will (incorrectly) identify the /path/to/install/files as the partition. To get around this, we have added a new variable -- methodFromCmdLine -- to the loaderData struct. This variable is initialized while parsing command line arguments upon seeing the "method=" argument. After the SCSI, IDE and the DASD hardware have been initialize, the setMethodFromCmdLine() function is called. A race condition was seen here, by virtue of which the setMethodFromCmdLine() function was called before all the USB devices had been recognized. To get around this, the sleep() function was used to sleep for 5 seconds. If setMethodFromCmdLine() was called without the USB devices recognized, the getPartitionsList() (used by the findPartitionsWithFile() function, described later on) function would not return a complete list of devices.
If, in the case of an hd install, the partition isn't specified, the findPartitionWithFile() function is called with the supplied directory name as a parameter. This function retrieves the list of available partitions and walks through them looking for the full path to the directory on each of the partitions. It accepts two parameters: the first is the pointer to the device string (populated by this function) and the second is the file to look for on all attached devices. This function returns the first partition it finds that contains the requested directory. This implementation was chosen because it similar to the case when the "ks=" argument is encountered while parsing the command line arguments.
Similarly, with this patch, the user is no longer required to specify the partition of the ks file. In this case, like before, if the partition is not specified, the findPartitionWithFile() function is called with "/path/to/ks.cfg" as a parameter, and the partition containing the path is used from then on. The ks partition is passed along to the python stage of anaconda by the --kspartition option, which is added to the anaconda language. The ks partition is also saved in the loaderData struct.
Pykickstart raises an exception when neither or both of biospart and partition options are specified with the harddrive command. Since the --partition is no longer required, pykickstart was modified to not raise an exception anymore.
2. Exploded tree support
To support exploded tree harddrive installation, we mount the harddrive specified by the "method=" option to /tmp/hdimage and try to find the path to a valid ISO image (one that contains a stage2.img). If a valid ISO image is found, it is mounted to /mnt/source and the URL passed to anaconda is
hdiso://[dev]/[path]@/mnt/source
If no such path is found, we check to see if if /tmp/hdimage contains a valid exploded tree by looking for the stage2.img in the /tmp/hdimage/images folder. In this case, the tree itself is mounted to /mnt/source and the URL passed along to the next stage is
hd://[dev]/[path]@/mnt/source.
This way, we know which partition is the source partition as well as the path to the relevant install files. The above approach is modeled closely after the behavior of anaconda for NFS installations, where /mnt/source is used as the "base" installation directory.
In the python stage of anaconda, the HardDriveInstallMethod class is replaced by two classes HdInstallMethod and HdIsoInstallMethod, analogous to the NfsInstallMethod and NfsIsoInstallMethod classes. HdInstallMethod is identical to NfsInstallMethod except for the URL parsing logic. HdIsoInstallMethod, on the other hand, extends both HdInstallMethod and NfsIsoInstallMethod so that there is minimal code redundancy.
3. Device modification protection
The anaconda.id.diskset contains a list of blacklisted devices that should not be used by default by the anaconda partitioner; this list includes both the ks and source devices. Methods returning the ks and source devices are also added to the DiskSet class. A compareDrives() function is added to DiskSet that extends the functionality of isys.compareDrives() by adding in a check against the blacklisted devices. If one of the two devices appears on the blacklist, then this device is considered 'smaller' in comparison to the other. If, on the other hand, the devices are jointly present or absent from the blacklist, then the standard isys.compareDrives() function is called to compute the return value. All references to isys.compareDrives() have thus been replaced with anaconda.id.diskset.compareDrives() in order to comply with the necessary changes in this patch.
In the bootloaderSetupChoices() function, when the updateDriveList() method is called, a sorted list (using the DiskSet.compareDrives() method) is passed along as well. This ensures that none of the blacklisted devices are set as the primary boot device.
CHANGES NOT AFFECTING BEHAVIOR ------------------------------ The mountHardDrive() function is now a state machine similar to all the other install methods' "mounting functions."
TEST CASES ----------
The following tests were performed with the aforementioned changes. Each install method was tested with and without specifying the partition of the kickstart file and source files independently. Tests were performed by installing Fedora 6 on an i386 machine. The version of booty used for testing was 0.82 and the version of pykickstart used was 0.36. All tests passed (both text and graphical installations):
--------------------------------------------------------- |Install Type | KS | METHOD | | --------------------------------------------- | |w/ dev|w/o dev|remote|w/ dev|w/o dev|remote| |---------------------------------------------------------- | HDISO | Y | Y | Y | Y | Y | Y | | HD | Y | Y | Y | Y | Y | Y | | NFSISO | Y | Y | Y | N/A | N/A | N/A | | NFS | Y | Y | Y | N/A | N/A | N/A | | HTTP | Y | Y | Y | N/A | N/A | N/A | ---------------------------------------------------------
Fedora Core 6 comes with Anaconda Version 11.1.1.3-1, so I put the files modified between versions 11.1.1.3-1 and 11.2.0.9 in the RHupdates directory, and placed it at the same level as the Fedora directory in the exploded tree. This way I also verified that the RHupdates directory is used during the installation.
For ISO installations, I created an updates.img and placed it one level up from the ISO files.
BUGS ---- We are not aware of any bugs at this time. However, any bugs should be reported to me - Uday Prakash - by email to uprakash@abodiosoftware.com.
On 1/25/07, Uday Prakash uprakash@abodiosoftware.com wrote:
Folks,
Please consider the following two patches for inclusion in the anaconda and pykickstart sources. The patches provide three features, described below, related to hard drive installations. Together these features enable a kickstart and/or an exploded tree installation image to be placed on a portable storage device, such as a USB flash memory or hard drive device, and the device to be used to install one or more systems with unknown hard drive configuration. In effect, with this patch, installing from a portable storage device has the same ease of use characteristics as installing from a CD, with the added benefit that kickstart and install files can be modified directly on the device as needed, without the overhead of creating new iso images and burning CDs.
Thank You Uday, I think between your patch and the current FC6 behavior with CD's that fixes my problem. I'll get back to you with patches that work against RHEL4 anaconda.
Many Thanks...james
Uday Prakash wrote:
The anaconda patch was created using version 11.2.0.9 of the sources. The pykickstart patch was created using version 0.36. We can provide an updated patches against the most current versions, if helpful. Given the rate at which sources are changing, we decided to hold off until someone in anaconda has had a chance to review the proposed patches and suggest changes, if needed.
While things change pretty quickly, patches are generally able to be applied with some ease across versions, so as long as it's relatively recent (which this is), then I'm less concerned about the specific version the patch is against.
These patches, though, seem to have a fair bit of whitespace changes making finding the real changes more difficult. A diff for the relevant changes is far easier to review. Also, having a separate diff for each logical change helps as well.
PATCH FEATURES
- Partition auto-detection. The patch allows the partition to be left
unspecified when using the harddrive method for kickstart and install locations, and the item (kickstart or installtree) to be located by iterating through available partitions. Specifically, the kickstart and method options can now be specified as follows on the command line (note the empty partition section between the two colons):
ks=hd::/path/to/ks.cfg, and method=hd::/path/to/install/files
Within the kickstart file, a hard drive install can be specified as follows (note the lack of a --partition directive):
harddrive --dir=/path/to/install/files
What happens then if there's more than one device that has a ks.cfg or an install source? Is the problem you're trying to solve the fact that device naming isn't all that reproducible from one machine to another? If so, is there anything about the device that you're having people install from that could be used instead as the device key (much like the biosdisk/part stuff).
- Exploded tree support. The patch enables harddrive installation from an
exploded tree, analogous to NFS installations. As a side effect, it also provides support for a RHupdates folder for an exploded tree installation.
This is something that used to be supported, but invariably, people would delete files they didn't think they needed. This then led to tracebacks which weren't clear or otherwise failing installations. Since people are going to be downloading the ISOs anyway, just using the ISOs for the hard drive install is far better. It also has the plus of the ISOs can be trivially verified. RHupdates could be supported pretty trivially with hard drive installs if it was desired, but it seems a lot less important with the ease of creating an updates.img now that they can be a cpio.gz.
- Device modification protection. By default, the patch removes partitions
on the harddrive containing the kickstart and/or install iso/tree from the default set to be formatted/partitioned. This can be overridden manually within the installer UI or by kickstart. Likewise, the install/kickstart harddrive is removed as a candidate for boot record modifications. This can similarly be overridden by kickstart.
How is this different than the existing protected partition support? Why not just continue to use it instead of implementing new methods?
Jeremy
While things change pretty quickly, patches are generally able to be applied with some ease across versions, so as long as it's relatively recent (which this is), then I'm less concerned about the specific version the patch is against.
These patches, though, seem to have a fair bit of whitespace changes making finding the real changes more difficult. A diff for the relevant changes is far easier to review. Also, having a separate diff for each logical change helps as well.
Thanks. We'll provide separate patches and keep them clean after we've addressed the questions you raise below.
What happens then if there's more than one device that has a ks.cfg or an install source? Is the problem you're trying to solve the fact that device naming isn't all that reproducible from one machine to another? If so, is there anything about the device that you're having people install from that could be used instead as the device key (much like the biosdisk/part stuff).
The patch we initially provided selected the first partition encountered with the kickstart/install source. A solution that gives the user more control, however, is to identify all partitions with the specified kickstart/install source and: * if the number of devices is 0, do nothing, or * if the number of devices is 1, use that one, or * else pop up the dialog box and ask the user.
Something similar to this is done in the loadUpdates() function in the loader section, where the user is asked for input if the number of potential update disks is greater than 1.
This solution is general and doesn't rely on device specific information.
Our initial thinking (and that of others on the mailing list as well) had been to idenfity the device by determining the device used to boot the system, but we don't have a way to find this.
This is something that used to be supported, but invariably, people would delete files they didn't think they needed. This then led to tracebacks which weren't clear or otherwise failing installations. Since people are going to be downloading the ISOs anyway, just using the ISOs for the hard drive install is far better. It also has the plus of the ISOs can be trivially verified. RHupdates could be supported pretty trivially with hard drive installs if it was desired, but it seems a lot less important with the ease of creating an updates.img now that they can be a cpio.gz.
In our scenario, the OS is preinstalled on client systems and a complete install image is provided on a secondary storage device. The secondary device is used for client data backup as well as for OS recovery in the event of primary hard drive corruption or failure. The install image on these secondary devices is kept up to date via synchronization. Since the systems can be located across slow network connections, we minimize data download by syncing at the file rather than iso/updates.img level. We could sync individual files and have client side code create an updates.img for anaconda to read. But for us it is cleaner over the long run to modify anaconda to deal with reading exploded trees. Since others on this mailing list have requested this functionality, we hoped to provide a general solution.
As for exploded tree verification, one general approach could be to provide a file with a list of file-md5 pairs that sits alongside the .discinfo file (perhaps named .filemd). This file would have a function similar to the repomd.xml file in the repodata folder, but presumably would be in plaintext format rather than xml (since it would be read by loader). This file could be used when the mediacheck flag is present to check md5 values for files in the distribution. We can imagine several issues with this approach, but will do some experimenting to see what we can come up with.
How is this different than the existing protected partition support? Why not just continue to use it instead of implementing new methods?
The bootloaderSetupChoices function in bootloader.py, the __call__ method in PartitionTypeWindow (for text-based install) and the createAllowedDrives function in partition_ui_helpers_gui.py (for graphical install) do not filter out the kickstart and the source devices. We can simplify our implementation to avoid new methods. Note that something similar to what we are trying to do is done with the updates device (the device containing the updates img - anaconda.updateSrc) to uncheck it by default in the partitions list displayed.
Thanks much, Uday.
<snip>
As for exploded tree verification, one general approach could be to provide a file with a list of file-md5 pairs that sits alongside the .discinfo file (perhaps named .filemd). This file would have a function similar to the repomd.xml file in the repodata folder, but presumably would be in plaintext format rather than xml (since it would be read by loader). This file could be used when the mediacheck flag is present to check md5 values for files in the distribution. We can imagine several issues with this approach, but will do some experimenting to see what we can come up with.
The biggest issue with this approach is filesystem metadata is not part of the checksums, so you can actually have a corrupt filesystem (perms and such) but have files that seem perfectly fine. This is what the embedded iso md5 provides, in that it checksums even the iso/filesystem metadata.
That said, it would possible I am assuming to do something similar with other filesystems, but I can give no advice in regards to this. Furthermore if the tree in questions is only a subset of the files on the filesystem, then this would not be usefull either (i.e. the embedded checksum approach), whereas your file list would e the best that could be done. Which ultimately shows an advantage to some sorto of image based approach, but I can definately see cases where the tree approach is desirable.
Cheers...james
anaconda-devel@lists.stg.fedoraproject.org