https://fedoraproject.org/wiki/DNF_system_upgrade quote: "This will reboot your machine immediately. The system should boot again into Fedora using the same kernel, but this time, the upgrade process appears on the boot screen. "
NAICT, that last part did not take place, but the term "the boot screen" is ambiguous. No bootloader is installed on the subject/target installation, one of many on a multiboot machine on which everything boots from a master bootloader under exclusive control of myself.
Tail of .bash_history on freshly updated F24 installation cloned to a new partition with boot stanza and fstab appropriately adjusted: dnf remove kernel-PAE-core-4.1.0 kernel-PAE-core-4.2.0 kernel-PAE-core-4.3.0 df / dnf repolist all dnf install dnf-plugin-system-upgrade df / dnf system-upgrade download --refresh --releasever=25 dnf config-manager --set-disabled updates dnf system-upgrade download --refresh --releasever=25 dnf system-upgrade download --refresh --releasever=25 --allowerasing dnf system-upgrade reboot
I had tried several times just to use DNF to upgrade, but always got error messages about updates not refreshing, which is why I tried the officially recommended upgrade route, and why in history you can see I disabled updates. The system-upgrade reported 700+ packages successfully downloaded. I saw nothing that looked like any error message.
All I saw after booting was a normal login screen on vtty1. On login, I don't see anything happening in ps -A that looks like any kind of updating, nor do I hear any HD activity. dnf repolist all shows only Fedora 24 is enabled. What now?
On Sun, Jul 31, 2016 at 3:11 AM, Felix Miata mrmazda@earthlink.net wrote:
https://fedoraproject.org/wiki/DNF_system_upgrade quote: "This will reboot your machine immediately. The system should boot again into Fedora using the same kernel, but this time, the upgrade process appears on the boot screen. "
NAICT, that last part did not take place, but the term "the boot screen" is ambiguous. No bootloader is installed on the subject/target installation, one of many on a multiboot machine on which everything boots from a master bootloader under exclusive control of myself.
dnf system-upgrade leverages systemd offline update mechanism. There's no bootloader changes. What should be try is 'dnf system-upgrade reboot' sets a symlink /system-update and as long as that exists, systemd picks it up and starts the offline update target.
https://www.freedesktop.org/software/systemd/man/systemd.offline-updates.htm...
Tail of .bash_history on freshly updated F24 installation cloned to a new partition with boot stanza and fstab appropriately adjusted: dnf remove kernel-PAE-core-4.1.0 kernel-PAE-core-4.2.0 kernel-PAE-core-4.3.0 df / dnf repolist all dnf install dnf-plugin-system-upgrade df / dnf system-upgrade download --refresh --releasever=25 dnf config-manager --set-disabled updates dnf system-upgrade download --refresh --releasever=25 dnf system-upgrade download --refresh --releasever=25 --allowerasing dnf system-upgrade reboot
I had tried several times just to use DNF to upgrade, but always got error messages about updates not refreshing, which is why I tried the officially recommended upgrade route, and why in history you can see I disabled updates. The system-upgrade reported 700+ packages successfully downloaded. I saw nothing that looked like any error message.
All I saw after booting was a normal login screen on vtty1. On login, I don't see anything happening in ps -A that looks like any kind of updating, nor do I hear any HD activity. dnf repolist all shows only Fedora 24 is enabled. What now?
You'll need to figure out wher dnf system-upgrade download put all of the packages it downloaded. Make sure the /system-update symlink points to that location - since I think that gets set with the 'dnf system-upgrade reboot' command and then immediately reboots, you'd have to figure out a way to intercept the update process - e.g. boot another OS, mount the Fedora 24 root, and see if the symlink is there and points to the correct location. If it's not there or points to the wrong location, it's a bug. If it's there and points to the correct location, reboot and edit the Fedora 24 boot entry and remove 'rhgb quiet' so you can see the boot process. It's really obvious either way when offline updates are happening. With graphical boot, there's a status in percentage on the upper left of the display, and with text boot you'll see the systemd-offline-updates process installing packages, listing each package, along with a percent complete. It's not possible to miss it.
So if that's not happen, check again if the symlink is still there and pointed to the right location. Either way, you've got a bug it's just a matter of how it gets written up, but you'll probably need more verbose systemd logging to attach to the bug. So redo 'dnf system-upgrade reboot' and edit the Fedora 24 boot entry to add systemd.log_level=debug, boot that modified entry, and once you're in Fedora 24 do something like 'journalctl -b -o short-monotonic > journal_dnfupgradesystemdebug.log' and attach that log to the bug report. Hopefully it contains a hint. I'm pretty sure this is all done with just systemd, and doesn't involve any dracut stuff so you probably don't need to to also do rd.debug.
Chris Murphy composed on 2016-07-31 12:43 (UTC-0400):
Felix Miata wrote:
https://fedoraproject.org/wiki/DNF_system_upgrade quote: "This will reboot your machine immediately. The system should boot again into Fedora using the same kernel, but this time, the upgrade process appears on the boot screen. "
NAICT, that last part did not take place, but the term "the boot screen" is ambiguous. No bootloader is installed on the subject/target installation, one of many on a multiboot machine on which everything boots from a master bootloader under exclusive control of myself.
dnf system-upgrade leverages systemd offline update mechanism. There's no bootloader changes. What should be try is 'dnf system-upgrade reboot' sets a symlink /system-update and as long as that exists, systemd picks it up and starts the offline update target.
https://www.freedesktop.org/software/systemd/man/systemd.offline-updates.htm...
Quoting from it: "Very early in the new boot systemd-update-generator(8) checks whether /system-update exists. If so, it (temporarily and for this boot only) redirects (i.e. symlinks) default.target to system-update.target, a special target that is pulls in the base system (i.e. sysinit.target, so that all file systems are mounted but little else) and the system update units."
NAICT, the 3 in the default boot stanza hijacked the redirect back to multi-user.target. I restarted with the 3 removed after finding 700+ rpms in the symlink /system-update/. Now vtty1 has stopped with these at screen bottom:
... [OK] Started System Upgrade. [OK] Reached target System Update. [OK] Listening on D-Bus System Message Bus Socket.
The machine is very quiet, but I do see a flicker from the HD LED.
Later in that URL, again quoting: "If applicable and possible, it should create a file system snapshot, then install all packages."
What kind of snapshot does it want to create? / filesystem is EXT3, and freespace prior to update rpms download was about 40%, 26% after. Only /home and /usr/local are separate filesystems "required" by the F24 installation.
Tail of .bash_history on freshly updated F24 installation cloned to a new partition with boot stanza and fstab appropriately adjusted: dnf remove kernel-PAE-core-4.1.0 kernel-PAE-core-4.2.0 kernel-PAE-core-4.3.0 df / dnf repolist all dnf install dnf-plugin-system-upgrade df / dnf system-upgrade download --refresh --releasever=25 dnf config-manager --set-disabled updates dnf system-upgrade download --refresh --releasever=25 dnf system-upgrade download --refresh --releasever=25 --allowerasing dnf system-upgrade reboot
I had tried several times just to use DNF to upgrade, but always got error messages about updates not refreshing, which is why I tried the officially recommended upgrade route, and why in history you can see I disabled updates. The system-upgrade reported 700+ packages successfully downloaded. I saw nothing that looked like any error message.
All I saw after booting was a normal login screen on vtty1. On login, I don't see anything happening in ps -A that looks like any kind of updating, nor do I hear any HD activity. dnf repolist all shows only Fedora 24 is enabled. What now?
You'll need to figure out wher dnf system-upgrade download put all of the packages it downloaded. Make sure the /system-update symlink points to that location
It did/does.
- since I think that gets set with the 'dnf system-upgrade
reboot' command and then immediately reboots, you'd have to figure out a way to intercept the update process - e.g. boot another OS, mount the Fedora 24 root, and see if the symlink is there and points to the correct location.
I didn't shut it down, left it alone overnight.
If it's not there or points to the wrong location, it's a bug. If it's there and points to the correct location, reboot and edit the Fedora 24 boot entry and remove 'rhgb quiet' so you can see the boot
I don't ever use any boot stanzas with quiet or rhgb in them.
process. It's really obvious either way when offline updates are happening. With graphical boot, there's a status in percentage on the upper left of the display,
Plymouth is not installed. There is no graphical boot. GUI only appears after boot and me doing something to start it, or booting with 5 on cmdline instead of 3.
and with text boot you'll see the systemd-offline-updates process installing packages, listing each package, along with a percent complete. It's not possible to miss it.
Not possible to _see_ any such thing here apparently.
So if that's not happen, check again if the symlink is still there and pointed to the right location.
There are no vttys available to login on to check anything.
Either way, you've got a bug it's just a matter of how it gets written up, but you'll probably need more verbose systemd logging to attach to the bug. So redo 'dnf system-upgrade reboot' and edit the Fedora 24 boot entry to add systemd.log_level=debug, boot that modified entry, and once you're in Fedora 24 do something like 'journalctl -b -o short-monotonic > journal_dnfupgradesystemdebug.log' and attach that log to the bug report. Hopefully it contains a hint. I'm pretty sure this is all done with just systemd, and doesn't involve any dracut stuff so you probably don't need to to also do rd.debug.
I left it alone a while. When I came back into the room, memtest was running, indicated it rebooted itself. Logging into F25 reports "Fedora 25 (Rawhide)", but on 4.6.4 fc24 kernel. 'rpm -qa | grep fc24 | wc -l' reports 297. Same except s/fc24/fc25/g reports 776. Freespace is 35%. Oh, right, boot stanza points specifically to the prior current kernel, as the upgrade process presented me no opportunity to update kernel & initrd symlinks. 4.8.0...fc25 is installed. Reboot to 4.8 succeeds. :-)
'dnf repolist all' fails with "Error: failed to synchronize cache for repo 'updates-testing'".
Journals: http://fm.no-ip.com/Tmp/Linux/F/ -b: http://fm.no-ip.com/Tmp/Linux/F/gx27c-jrnl-0.txt -b -1: http://fm.no-ip.com/Tmp/Linux/F/gx27c-jrnl.txt -b -2: http://fm.no-ip.com/Tmp/Linux/F/gx27c-jrnl-1.txt -b -3: http://fm.no-ip.com/Tmp/Linux/F/gx27c-jrnl-2.txt -b -4: http://fm.no-ip.com/Tmp/Linux/F/gx27c-jrnl-3.txt
Any suggestions where to go/what to do from here? Does 27% of total installed packages =f24 sound about right for now?
On Sun, Jul 31, 2016, 2:57 PM Felix Miata mrmazda@earthlink.net wrote:
Chris Murphy composed on 2016-07-31 12:43 (UTC-0400):
Felix Miata wrote:
https://fedoraproject.org/wiki/DNF_system_upgrade quote: "This will reboot your machine immediately. The system should boot again into Fedora using the same kernel, but this time, the upgrade process appears on the boot screen. "
NAICT, that last part did not take place, but the term "the boot screen" is ambiguous. No bootloader is installed on the subject/target installation, one of many on a multiboot machine on which everything boots from a master bootloader under exclusive control of myself.
dnf system-upgrade leverages systemd offline update mechanism. There's no bootloader changes. What should be try is 'dnf system-upgrade reboot'
sets
a symlink /system-update and as long as that exists, systemd picks it up and starts the offline update target.
https://www.freedesktop.org/software/systemd/man/systemd.offline-updates.htm...
Quoting from it: "Very early in the new boot systemd-update-generator(8) checks whether /system-update exists. If so, it (temporarily and for this boot only) redirects (i.e. symlinks) default.target to system-update.target, a special target that is pulls in the base system (i.e. sysinit.target, so that all file systems are mounted but little else) and the system update units."
NAICT, the 3 in the default boot stanza hijacked the redirect back to multi-user.target. I restarted with the 3 removed after finding 700+ rpms in the symlink /system-update/. Now vtty1 has stopped with these at screen bottom:
... [OK] Started System Upgrade. [OK] Reached target System Update. [OK] Listening on D-Bus System Message Bus Socket.
The machine is very quiet, but I do see a flicker from the HD LED.
Later in that URL, again quoting: "If applicable and possible, it should create a file system snapshot, then install all packages."
What kind of snapshot does it want to create? / filesystem is EXT3, and freespace prior to update rpms download was about 40%, 26% after. Only /home and /usr/local are separate filesystems "required" by the F24 installation.
I'm not sure what kind of snapshot this refers to. I have done two dnf system upgrades from 23 to 24 on Btrfs, I don't recall seeing any Btrfs snapshots, but that's a weak statement.
Tail of .bash_history on freshly updated F24 installation cloned to a new partition with boot stanza and fstab appropriately adjusted: dnf remove kernel-PAE-core-4.1.0 kernel-PAE-core-4.2.0 kernel-PAE-core-4.3.0 df / dnf repolist all dnf install dnf-plugin-system-upgrade df / dnf system-upgrade download --refresh --releasever=25 dnf config-manager --set-disabled updates dnf system-upgrade download --refresh --releasever=25 dnf system-upgrade download --refresh --releasever=25 --allowerasing dnf system-upgrade reboot
I had tried several times just to use DNF to upgrade, but always got error messages about updates not refreshing, which is why I tried the officially recommended upgrade route, and why in history you can see I disabled updates. The system-upgrade reported 700+ packages successfully downloaded. I saw nothing that looked like any error message.
All I saw after booting was a normal login screen on vtty1. On login, I don't see anything happening in ps -A that looks like any kind of updating, nor do I hear any HD activity. dnf repolist all shows only Fedora 24 is enabled. What now?
You'll need to figure out wher dnf system-upgrade download put all of the packages it downloaded. Make sure the /system-update symlink points to that location
It did/does.
- since I think that gets set with the 'dnf system-upgrade
reboot' command and then immediately reboots, you'd have to figure out a way to intercept the update process - e.g. boot another OS, mount the Fedora 24 root, and see if the symlink is there and points to the correct location.
I didn't shut it down, left it alone overnight.
If it's not there or points to the wrong location, it's a bug. If it's there and points to the correct location, reboot and edit the Fedora 24 boot entry and remove 'rhgb quiet' so you can see the boot
I don't ever use any boot stanzas with quiet or rhgb in them.
process. It's really obvious either way when offline updates are happening. With graphical boot, there's a status in percentage on the upper left of the display,
Plymouth is not installed. There is no graphical boot. GUI only appears after boot and me doing something to start it, or booting with 5 on cmdline instead of 3.
and with text boot you'll see the systemd-offline-updates process installing packages, listing each
package,
along with a percent complete. It's not possible to miss it.
Not possible to _see_ any such thing here apparently.
So if that's not happen, check again if the symlink is still there and pointed to the right location.
There are no vttys available to login on to check anything.
Either way, you've got a bug it's just a matter of how it gets written up, but you'll probably need more verbose systemd logging to attach to the bug. So redo 'dnf system-upgrade reboot' and edit the Fedora 24 boot entry to add systemd.log_level=debug, boot that modified entry, and once you're in Fedora 24 do something like 'journalctl -b -o short-monotonic > journal_dnfupgradesystemdebug.log'
and
attach that log to the bug report. Hopefully it contains a hint. I'm pretty sure this is all done with just systemd, and doesn't involve any dracut stuff so you probably don't need to to also do rd.debug.
I left it alone a while. When I came back into the room, memtest was running, indicated it rebooted itself. Logging into F25 reports "Fedora 25 (Rawhide)", but on 4.6.4 fc24 kernel. 'rpm -qa | grep fc24 | wc -l' reports 297. Same except s/fc24/fc25/g reports 776. Freespace is 35%. Oh, right, boot stanza points specifically to the prior current kernel, as the upgrade process presented me no opportunity to update kernel & initrd symlinks. 4.8.0...fc25 is installed. Reboot to 4.8 succeeds. :-)
'dnf repolist all' fails with "Error: failed to synchronize cache for repo 'updates-testing'".
Apparently normal at the current state of branch in guessing. I get the same thing on clean installs with an 20160730 compose, as well as a pre branch rawhide to F25 distro-sync.
Journals: http://fm.no-ip.com/Tmp/Linux/F/ -b: http://fm.no-ip.com/Tmp/Linux/F/gx27c-jrnl-0.txt -b -1: http://fm.no-ip.com/Tmp/Linux/F/gx27c-jrnl.txt -b -2: http://fm.no-ip.com/Tmp/Linux/F/gx27c-jrnl-1.txt -b -3: http://fm.no-ip.com/Tmp/Linux/F/gx27c-jrnl-2.txt -b -4: http://fm.no-ip.com/Tmp/Linux/F/gx27c-jrnl-3.txt
Any suggestions where to go/what to do from here? Does 27% of total installed packages =f24 sound about right for now?
Rawhide to F25 distrosync download was about 1/3 the size of a live ISO, which may be a completely different way of comparing this.
Chris Murphy
On Sun, Jul 31, 2016 at 10:09 PM, Chris Murphy lists@colorremedies.com wrote:
I'm not sure what kind of snapshot this refers to. I have done two dnf system upgrades from 23 to 24 on Btrfs, I don't recall seeing any Btrfs snapshots, but that's a weak statement.
The earlier version of this man page refers to Btrfs snapshots. https://freedesktop.org/wiki/Software/systemd/SystemUpdates/
The current man page refers to "update script" as being the thing that should do the snapshot. If the update fails, it's supposed to make sure the /system-update symlink isn't in the snapshot, that way upon reboot the upgrade isn't re-attempted (in a possibly endless loop). And also it should presumably update the bootloader configuration so it boots the unmodified snapshot rather than the failed updated subvolume.
Thing is, it's not necessary to do offline updates if you can leverage snapshots. Instead they can be done out of band -> make snapshot, have a chroot or nspawn container do the update work on that snapshot, if it fails delete the snapshot, if it completes update bootloader config. It's failsafe. The user can keep working during the update. And there's only one reboot needed, and only if the update completes, if there was a failure, no reboot is needed at all. Seems pretty useful for servers and workstations.
-
Chris Murphy -
Felix Miata