On Tue, Aug 30, 2016 at 3:19 PM, Felix Miata mrmazda@earthlink.net wrote:

Kamil Paral composed on 2016-08-30 06:24 (UTC-0400):

...

...

Following the instructions on https://fedoraproject.org/wiki/DNF_system_upgrade eventually I reached dnf system-upgrade download --refresh --releasever=25 which apparently completed without error, ending by directing to run dnf system-upgrade reboot. So, I did, and it booted normally. Nothing is running that isn't supposed to be running in a normal boot. dnf system-upgrade log reports no logs found. The only dnf string in journalctl -b is starting the makecache timer. This is not the first time I've tried those wiki page instructions exactly like this with same result. What's supposed to be happening that isn't?

...

It's supposed to boot into an offline updates mode, and do the upgrade. Please file a bug and attach logs ("journalctl -b -1" after you boot into standard desktop instead of offline updates mode). Thanks.

What I was looking for is the process that triggers "offline updates mode". If it involves the bootloader, I know whatever it may be simply doesn't exist. This is a multiboot installation, so no Fedora bootloader is involved in its boot process. If not bootloader related, I'll go ahead and file a bug, or even if it is, but you indicate you want one anyway.

https://www.freedesktop.org/software/systemd/man/systemd.offline-updates.htm...

What triggers it is the creation of /system-update symlink and a reboot. Then systemd picks that up and for that boot considers system-update.target the default target.

On Tue, Aug 30, 2016 at 4:08 PM, Felix Miata mrmazda@earthlink.net wrote:

Chris Murphy composed on 2016-08-30 15:31 (UTC-0600):

...

Felix Miata wrote:

...

...

What I was looking for is the process that triggers "offline updates mode". If it involves the bootloader, I know whatever it may be simply doesn't exist. This is a multiboot installation, so no Fedora bootloader is involved in its boot process. If not bootloader related, I'll go ahead and file a bug, or even if it is, but you indicate you want one anyway.

...

https://www.freedesktop.org/software/systemd/man/systemd.offline-updates.htm...

...

What triggers it is the creation of /system-update symlink and a reboot. Then systemd picks that up and for that boot considers system-update.target the default target.

So apparently the kernel cmdline can be involved, and the only bug appears to be that the wiki page has no boot stanza admonition that the cmdline must not include anything to avoid booting to the temporary default target (in this case "3", which is a parameter I include on most of my kernel cmdlines).

Anything that overrides default.target is supposed to be temporary. To make it the default, and therefore maintain compatibility with system upgrades:

systemctl set-default multi-user.target

In effect the upgrade fails because there are two conflicting temporary default overrides: /system-updates and the 3.

Felix Miata composed on 2016-08-30 18:44 (UTC-0400):

I'm still waiting to see evidence that the upgrade process is actually proceeding, at least 35 minutes after having "Reached target System Update" show up on the screen.

Success. E7500 Core2Duo 2.93GHz CPU. About 54 minutes long boot according to journal timestamps.

On Thu, Sep 1, 2016 at 8:32 AM, Richard Ryniker ryniker@alum.mit.edu wrote:

I am unhappy with the suggestion that the semantics of kernel parameters may depend on some notion about how "temporary" they are. If a kernel parameter is specified, it is present; if not specified, it is absent.

3 isn't even a kernel parameter, most things now on the kernel parameter line isn't even meant for the kernel, there is no way to infer what will parse it or what it does. It's a simple hack, and therefore it means the user has to, you know, know what the fuck they're doing when they use boot params.

What you're basically saying is, you're unhappy that you have to know what the fuck you're doing to get it to do what you want it to do. Sorry.

It is proper to design parameter syntax and default values to favor common usage. There also have to be mechanisms to handle different specifications for the same parameter (first or last wins...) and use of incompatible parameters (pick one, ignore both... but always log the choice!)

The last one wins. The last one is the 3 because that's what you set at boot time. By doing that, you're asking for multi-user.target which is exactly what you get. You do not ever get default.target. and since /system-update symlink causes default.target to mean system-update.target, you don't get that.

In this case, "dnf system-upgrade reboot" is an explicit command to perform an offline system update.

Followed 10 seconds later by your 3. Last one wins.

That is what should happen, regardless of the (default) target. Any target specification should apply after the offline update.

You overrode that. That's what you're asking for when you set 3 in the bootloader configuration. If you don't like that, stop doing things wrong, and start doing them correctly.

Are you concerned about the possibility "dnf system-upgrade reboot" was ordered, then immediately decided it was wrong? How many times do you want to ask "Do you really mean this?" Why should specification of boot target 3 as a kernel parameter pretermit the offline update - but possibly leave it waiting to occur unexpectedly during a subsequent boot?

I don't have answers to your questions because I refuse the premise by which they even arise. I only use 3 when I'm editing a particular boot menu entry, non-persistently. If I wanted it persistent I'd change what default.target points to. That's the correct way to make persistent changes.

Boot parameters make the OS less portable also. And I foresee the day when Fedora cannot be booted from another OS's bootloader because of this.

Your explanation of how systemd implements offline update using a symlink offers an escape: boot from a different root file system (a live image, if necessary) and remove the offline update symlink. If you think that is too esoteric or awkward, implement a new kernel parameter such as "cancel-offline-update" that does this.

Request denied.

What is the relation between run level 1 and offline update? Does run level 1 stop the init process before offline update? If yes, then that is an easy escape: boot to run level 1, remove the offline update symlink, then "systemctl default".

My expectation is that any runlevel specified as a boot parameter will inhibit use of default.target and inhibit the offline update. Explicitly stating the runlevel at boot time means don't use default.target.

On Sat, Sep 3, 2016 at 1:47 PM, Richard Ryniker ryniker@alum.mit.edu wrote:

I seem to have touched a sore spot on Mr. Murphy, and apologize if I have unintentionally irritated him.

What annoys me is that people want to have things two ways. Basically they want what they want because they want it, which simply isn't good enough reasoning because it's circular logic. SysVInit systems set persistent runlevels in /etc/inittab, and systemd systems do it with the /etc/systemd/system/default.target symlink. That is how it works. If you want a certain runlevel or target to be persistent, there is a mechanism for that and putting it on the kernel parameter line is simply the wrong way to do it.

Apology is not required, my annoyance is not your responsibility.

If he is the designer of the offline update mechanism, and I correctly perceive the implications of his explanation, then I do scold him for failure to mitigate the damage that might occur to a system in the situation reported by the original poster.

I'm not the designer. But I'll defend the design because there's X^n ways for a user to sabotage updates, and there must be a limit on how much testing anything does to check whether or not the user has sabotaged the system. Adding a 3 permanently to grub.cfg is sabotage. The user doesn't know what they're doing, because if they did, they wouldn't modify /etc/default/grub in this manner.

And what damage? In this case dnf fails safe. It does nothing. That's a completely reasonable outcome as a result of prior user sabotage.

The smaller problem is that there is no explanation why the offline upgrade is not performed after "dnf system-upgrade reboot". I expect a person with the authority to execute the dnf command knows how to examine the system journal to seek an explanation for why it did not happen. Mr. Miata presumably looked and could find nothing, which is why he created the original post.

The problem here is either:

a. dnf is using a really simplistic message hand off to systemd via the /system-update symlink while not first parsing the existing bootloader configuration for a conflicting parameter and removing it. b. systemd is not resolving the ambiguity of finding a "3" boot parameter, and /system-update symlink. But how can it resolve that ambiguity? Is it supposed to check the date stamp of grub.cfg and /system-update and use the one with the most recent modification time?

So exactly what and where is the failure even determined? It's really a systemd issue as near as I can tell. So I suppose systemd could not do the offline update and just say something like "command line target conflicts with system-update.target" while always honoring the command line argument.

The larger problem is the offline update remains pending, indefinitely, until the old-style "run-level" number is removed from the "kernel parameters" (or whatever name you prefer to describe this hodgepodge collection of data.) This might happen days, even weeks, after the "dnf system-upgrade reboot" operation. Two bad things then happen:

There are always consequences to user sabotage. It doesn't matter if that sabotage is innocent ignorance or willful ignorance, the computer doesn't distinguish between them.

The boot operation, now executing the offline system update, takes a long time: typically one hour with a rotating drive, less with a solid state drive, longer if many extra packages are installed. If this is expected and planned, fine. If it is unexpected, an important service may go missing - an unplanned outage with expensive consequences for users.

It's expected.

There are a couple of ways to do out of band/out of tree updates that don't have this kind of impact and also with far reduced risk. rpm-ostree is one possible way to do that and there's an active plan to get Workstation to be rpm-ostree based installations.

Implement a service to check for a pending offline update and add it to the dependencies for the multi-user and graphical targets. At the least, this service will emit a message to say there is a pending offline update that was not executed. Even better, in my opinion, would be to remove the symlink that triggers an offline update and emit a message that says the pending offline update has been canceled and can be rescheduled with "dnf system-upgrade reboot"

Document the interaction between offline update and old style run level numbers. Resources such as

https://fedoraproject.org/wiki/DNF_system_upgrade

should provide enough information about this operation to avoid the need to tell a user "You're unhappy that you have to know what the fuck you're doing to get it to do what you want it to do."

Right, and thank you for calling me out on my bullshit. But the fact does remain that sometimes, and this is one of those cases, the user is expected to know what they're doing and because they don't, they unwittingly caused the update process to be thwarted in a way that the simplistic update mechanism doesn't itself test for. While it fails gracefully, it doesn't have a means to recognize the desired upgrade didn't happen and then notify the user.

What you're describing isn't simple. Maybe the simplest one would be just systemd logging the ambiguity of conflicting command line parameter and the existence of a /system-update symlink. But note that by necessity that log entry would not be in the current (re)boot, it'll be in the failed offline update boot, i.e. -b -1, which also isn't exactly obvious for the user to look for. But better than nothing I suppose.

So sure, try filing an RFE bug against system... might be interesting to see what they say.