On 12 September 2014 16:16, Nathanael d. Noblet nathanael@gnat.ca wrote:

Yeah, I almost never use the reboot & install method. 90% of the packages being installed/updated seem foolish to need a reboot to update.

I've been called worse that foolish I guess...

I typically do a yum update manually and then if I notice glibc/kernel/systemd or other big packages do a reboot.

That's just not safe. Have you ever had firefox open and done a firefox update? Widgets start disappearing, redraws start having weird artifects and then after a little while it just crashes. Other applications like LibreOffice behave the same. Anyone that says things like "the old version of the library stays in memory" obviously hasn't actually done much real-world GUI programming in the last decade, or runs any kind of secure desktop system. The *only* way to do this securely and safely in the system we have now is in a clean pre-boot environment, which is sad and crap UX, but still nevertheless true. When we have application sandboxing and a stable OS platform to use, we can certainly do something more sane, but until then we're just hacking around the problem.

What we could do is do updates on shutdown by basically killing everything except PID 1, and then restart everything, but even then that relies on no systemd or kernel updates being present. Solving the real problem is much harder (OS + Apps split) but we also need to slow the firehose of needless updates.

Richard

Am 12.09.2014 um 18:33 schrieb Nathanael d. Noblet:

So I don't use Firefox anymore but I do know back in the day if we had FF open when we updated it would do a double request for each page/form. However when updating we just restarted FF and it would work fine after that. I've never noticed any other issues than FF but like I said I don't use it anymore.

Granted that doesn't matter obviously we don't want that kind of behaviour.

I am curious though. Everyone says the only way to do it securely and safely is with nothing running. Why can't updates be applied with stuff running prior to a reboot?

1 out of a million cases needs offline updates

really - the only good at it is that you can stick at using YUM and decide what you have to do at your own - rarely updates really require a reboot

* lsof | grep DEL | grep /usr and restart services on servers * logout/login and the above on workstations

Am 12.09.2014 um 18:58 schrieb Till Maas:

On Fri, Sep 12, 2014 at 06:37:07PM +0200, Reindl Harald wrote:

...

• lsof | grep DEL | grep /usr and restart services on servers

There is a convenient script called "needs-restarting" to do this. Checkrestart on debian even suggest potential init scripts that can be used to restart the services

never worked relieable here on multiple machines

it often showed nothing where i knew the thing which should be restarted without looking and "lsof" proved it

On Mon, Sep 15, 2014 at 09:50:36AM +0200, Miroslav Suchý wrote:

On 09/12/2014 07:09 PM, Reindl Harald wrote:

...

never worked relieable here on multiple machines

it often showed nothing where i knew the thing which should be restarted without looking and "lsof" proved it

I am one of those guys who refuse to reboot after each upgrade (and it works for me) and needs-restarting is ugly and insufficient to me.

Therefore I initiated this project: https://github.com/FrostyX/tracer http://copr.fedoraproject.org/coprs/frostyx/tracer/

It is still not finished and ready for announcement, but if you are looking for some other way than offline-upgrade, this might be worth of participating.

It wasn't clear to me how tracer works for non-C programs.

However there was some Red Hat only discussion recently about how to do this for Python programs, with minimal overhead. Below I'm just reproducing a technique (untested) that I think will work for Python.

It requires a small patch to the Python interpreter, and a similar patch to any other language interpreters (eg. Perl, Ruby).

Rich.

------- For each module (*.py or *.pyc) that it imports, have it mmap the first page of that file into its memory.

The mmap would be PROT_NONE because it's not actually used, and the associated file descriptor should be closed.

This will appear in /proc/PID/maps, with a "(deleted)" flag if the underlying file gets deleted (and hence the process needs restarting).

The cost should be almost nothing:

- 4K of virtual memory, no real memory - an extra mmap syscall on import - an extra segment in the kernel's VM AVL -------

Rich.

Every of the script is based on assumption that you already read some library/unit whatever. But that is not enough. I wonder how you want to detect that you need restart in case that I have something like this:

$ ls foo.rb bar.rb

$ cat foo.rb

def some_function require 'bar' end

And now

1) I run some application, which loads my foo.rb file. 2) I later update the package which removes bar.rb file. 3) And I call some_function which fails due to missing bar.rb

There is no universal and reliable way how to detect this scenario IMO.

Vít

Dne 15.9.2014 10:06, Richard W.M. Jones napsal(a):

On Mon, Sep 15, 2014 at 09:50:36AM +0200, Miroslav Suchý wrote:

...

On 09/12/2014 07:09 PM, Reindl Harald wrote:

...

never worked relieable here on multiple machines

it often showed nothing where i knew the thing which should be restarted without looking and "lsof" proved it

I am one of those guys who refuse to reboot after each upgrade (and it works for me) and needs-restarting is ugly and insufficient to me.

Therefore I initiated this project: https://github.com/FrostyX/tracer http://copr.fedoraproject.org/coprs/frostyx/tracer/

It is still not finished and ready for announcement, but if you are looking for some other way than offline-upgrade, this might be worth of participating.

It wasn't clear to me how tracer works for non-C programs.

However there was some Red Hat only discussion recently about how to do this for Python programs, with minimal overhead. Below I'm just reproducing a technique (untested) that I think will work for Python.

It requires a small patch to the Python interpreter, and a similar patch to any other language interpreters (eg. Perl, Ruby).

Rich.

---

For each module (*.py or *.pyc) that it imports, have it mmap the first page of that file into its memory.

The mmap would be PROT_NONE because it's not actually used, and the associated file descriptor should be closed.

This will appear in /proc/PID/maps, with a "(deleted)" flag if the underlying file gets deleted (and hence the process needs restarting).

The cost should be almost nothing:

• 4K of virtual memory, no real memory
• an extra mmap syscall on import
• an extra segment in the kernel's VM AVL

---

Rich.

On 15 September 2014 13:06, Miroslav Suchý msuchy@redhat.com wrote:

But *I* do not want to reboot after each upgrade. Those crashes will be 0.00001% of all crashes on my workstation

I think you might change your mind when it's you're the one that has to triage those ABRT-reported bugs. Also, consoling users with lost data gets tiring really quick.

Richard

On Mon, Sep 15, 2014 at 10:57:13AM +0200, Vít Ondruch wrote:

Every of the script is based on assumption that you already read some library/unit whatever. But that is not enough. I wonder how you want to detect that you need restart in case that I have something like this:

$ ls foo.rb bar.rb

$ cat foo.rb

def some_function require 'bar' end

And now

1. I run some application, which loads my foo.rb file.
2. I later update the package which removes bar.rb file.
3. And I call some_function which fails due to missing bar.rb

How is this not 'foo' simply being broken? ie. Not expressing its needs properly in its RPM dependencies?

It would still have been broken even with a reboot.

Rich.

On Mon, Sep 15, 2014 at 02:40:34PM +0200, Reindl Harald wrote:

Am 15.09.2014 um 14:28 schrieb Richard W.M. Jones:

...

On Mon, Sep 15, 2014 at 10:57:13AM +0200, Vít Ondruch wrote:

...

1. I run some application, which loads my foo.rb file.
2. I later update the package which removes bar.rb file.
3. And I call some_function which fails due to missing bar.rb

How is this not 'foo' simply being broken? ie. Not expressing its needs properly in its RPM dependencies?

It would still have been broken even with a reboot

no - why should it?

'foo' is loaded in memory, updated and now has different dependencies no longer require 'bar.rb' but your running version still do

Please read closely. 'foo' has *not* been updated.

If 'foo' had been updated, we would have spotted it and restarted that process using my technique outlined in the previous email.

Rich.

On Mon, Sep 15, 2014 at 02:49:34PM +0200, Reindl Harald wrote:

Am 15.09.2014 um 14:44 schrieb Richard W.M. Jones:

...

On Mon, Sep 15, 2014 at 02:40:34PM +0200, Reindl Harald wrote:

...

Am 15.09.2014 um 14:28 schrieb Richard W.M. Jones:

...

On Mon, Sep 15, 2014 at 10:57:13AM +0200, Vít Ondruch wrote:

...

1. I run some application, which loads my foo.rb file.
2. I later update the package which removes bar.rb file.
3. And I call some_function which fails due to missing bar.rb

How is this not 'foo' simply being broken? ie. Not expressing its needs properly in its RPM dependencies?

It would still have been broken even with a reboot

no - why should it?

'foo' is loaded in memory, updated and now has different dependencies no longer require 'bar.rb' but your running version still do

Please read closely. 'foo' has *not* been updated.

If 'foo' had been updated, we would have spotted it and restarted that process using my technique outlined in the previous email

cross deps coming in my mind foo -> library -> library -> library

• the first maybe already loaded
• also loaded the second one in a previous call
• that version relies on teh third one for some operations
• in a update the deps have changed

so you may have a mix with different dep-chains in memory and some parts used the first time from disk with unexpected results

I have absolutely no idea what you are talking about.

Rich.

On Mon, Sep 15, 2014 at 04:07:39PM +0200, Vít Ondruch wrote:

Dne 15.9.2014 14:28, Richard W.M. Jones napsal(a):

...

On Mon, Sep 15, 2014 at 10:57:13AM +0200, Vít Ondruch wrote:

...

Every of the script is based on assumption that you already read some library/unit whatever. But that is not enough. I wonder how you want to detect that you need restart in case that I have something like this:

$ ls foo.rb bar.rb

$ cat foo.rb

def some_function require 'bar' end

And now

1. I run some application, which loads my foo.rb file.
2. I later update the package which removes bar.rb file.
3. And I call some_function which fails due to missing bar.rb

How is this not 'foo' simply being broken?

They might come from different packages.

OK, in which case the package that needs bar was broken because it didn't express that need in its dependencies, nor guard against the possibility that bar was not installed.

Or there might be also another level of requires, where bar.rb requires by baz.rb. In case that bar.rb stays and baz.rb is removed, you still cannot predict that this will fail in the future, since neither of these files was loaded before.

In which case bar.rb was similarly broken, for the same reason as above.

Or there might be another example of code with similar issues:

$ cat foo.rb

def some_function $files.each {|f| require f} end

$files = Dir.glob('*.rb')

I.e. during initialization, you list available files and you want to load them later, but at that moment, they are not there already.

This code is still by any measure broken. There are lots of ways that such code could fail to work.

Plainly what I'm trying to say is: If the potentially insecure code has been loaded into a Python process, and the python interpreter has the small modification that I suggested, then we will be able to detect that the insecure code is loaded into memory and flag the process/service as needing to be restarted.

That's all.

Rich.

Am 21.10.2014 um 22:08 schrieb Lennart Poettering:

On Fri, 12.09.14 18:37, Reindl Harald (h.reindl@thelounge.net) wrote:

...

1 out of a million cases needs offline updates

really - the only good at it is that you can stick at using YUM and decide what you have to do at your own - rarely updates really require a reboot

• lsof | grep DEL | grep /usr and restart services on servers

Well, some deps are not visible like that, because they do not involve continuous mappings or open fds.

may be true but in practice no problem over many years

Moreover, it won't help you much anyway, as some daemons are not restarble right now, most prominently dbus-daemon

you repeat that again and again while i restart dbus over years on headless machines for web/file/db-servers and frankly before F15 even messagebus was completly disabled on all that machines

And strictly speaking as you cannot restart all daemons at the very same instant, or even at the same instant as you install the new files and remove the old ones you will always have races where daemons might make use of resources or interfaces that are either newer than what they expect or older.

interesting is that not so long ago there where just not much such dependencies - mandatory presence of dbus is very recent

other services like some webapp talking to a db-server? frankly i wrote 10 years ago db-layers to wait and retry so you can restart the db server after an update

offline updates are really about make updates fully reliable. Yes, in most cases a "yum update" during runtime works well enough, and yes, I usually do my updates that way too. But I am actually able to help myself if something goes wrong. And so are you.

true

Offline updates are more for the cases where things need to be reliable, because no well educated admin is available to instantly fix things. Possibly because the machine is used by noobs only, or because the machine is buried somewhere under the see, or where so many instances of the machine are running that a human admins don't scale

Hope that makes some sense

yes, but keep in mind not introduce more and more dependencies to make them mandatory somewhere in the future

On Wed, Oct 22, 2014 at 08:48:59AM +0200, Miroslav Suchý wrote:

...

Offline updates are more for the cases where things need to be reliable, because no well educated admin is available to instantly fix things.

I will print it an pin up on my notice board.

And the implication is that offline updates are not for readers of devel@lists.fedoraproject.org

I don't think that's very fair, without the context. First, of course, we're developing for more than just ourselves. And second, this isn't a reversible statement: just because offline updates primarily target one user type doesn't mean that other user types can't or shouldn't use it.

Why do I care about this? The non-techie user wants less rebooting too. I'd love to see the updates toolchain get more smarts about recognizing when an update is "safe" (or at least "safer") and not reboot in those cases. That's possible but would be an investment of work. In the meantime, "offline updates if you want simple and guaranteed", "use yum or dnf online if you're able to handle unlikely but possible consequences" seems workable enough.

Am 12.09.2014 um 23:36 schrieb Richard W.M. Jones:

On Fri, Sep 12, 2014 at 10:33:13AM -0600, Nathanael d. Noblet wrote:

...

I am curious though. Everyone says the only way to do it securely and safely is with nothing running. Why can't updates be applied with stuff running prior to a reboot?

There's no reason, apart from the kernel. You're doing it right already

well, some libraries like glibc leads also in better reboot since they are used by nearly any component, but for 99% of all updates a reboot is a joke since you easily can restart servcices with nearly zero downtime

systemctl condrestart service1 service2 service3

condrestart since i distribute such commands over the whole infrsastructure after tests and deployment and not every installed service is enabled on all hosts

On Sat, Sep 13, 2014 at 3:32 AM, Chris Murphy lists@colorremedies.com wrote:

On Sep 12, 2014, at 9:47 AM, Richard Hughes hughsient@gmail.com wrote:

...

The *only* way to do this securely and safely in the system we have now is in a clean pre-boot environment,

Mostly clean post-boot environment, with the system we have now?

...

What we could do is do updates on shutdown by basically killing everything except PID 1, and then restart everything, but even then that relies on no systemd or kernel updates being present.

Even if the system is fully rebooted once, after the update is done in a post-boot environment, it's half the reboots needed now.

One step up from this would be a way for packages to contain metadata indicating they need a reboot after an update; if none require it, then isolate graphical.target rather than a reboot.

One step up from this would be something like a kpatch process in rpm combined with packaged metadata that replaces in-memory modules so that reboots wouldn't be necessary. Yeh, probably impossible.

Fred

the keys are in kernelspace IIRC and thus updated / passed on initrd /initramfs updates and kernel updates

Corey W Sheldon Freelance IT Consultant, Multi-Discipline Tutor 310.909.7672 www.facebook.com/1stclassmobileshine

On Sat, Sep 13, 2014 at 7:01 PM, Ian Pilcher arequipeno@gmail.com wrote:

On 09/13/2014 03:59 AM, Fred New wrote:

...

One step up from this would be something like a kpatch process in rpm combined with packaged metadata that replaces in-memory modules so that reboots wouldn't be necessary. Yeh, probably impossible.

This has almost certainly already been considered by people smarter than me, but it occurs to me that there's no reason that kexec couldn't some- how pass LUKS/dm-crypt keys to the new kernel.

--

Ian Pilcher arequipeno@gmail.com

-------- "I grew up before Mark Zuckerberg invented friendship" --------

-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

On 16 September 2014 10:36, Roberto Ragusa mail@robertoragusa.it wrote:

If applications would just use libraries correctly, the kernel would be able to let parts of deleted files be available for lazy loading.

Sure, as long as all[1] the resources were either open()d when the user started the program, or linked in as resources into the library itself then it works fine. You can certainly design an application that does this correctly, it's just that very few do. A lot of GNOME apps compile in resources into the binary itself, using GResource, and this very much helps live updating.

Richard

[1] Including help files, all clipart, all icons, all .ui files, and all the .css for all the dialogs.

On 16 September 2014 10:55, Zdenek Kabelac zkabelac@redhat.com wrote:

Just a thought - but wouldn't be better spend time to enlighten Gnome/Firefox developers how to write applications in a way the could be upgraded runtime

So, it's not just the application, it's every application and D-Bus service the application uses. Even glibc opens files-as-resources at runtime. The much bigger issues is if you're using a D-Bus service like most applications seem to do (and most use quite a few system and session, directly and indirectly) then you've also got to co-ordinate and handle changing D-Bus API (which typically isn't versioned). This means you've got to restart the system DBus service at the right time in the on-line update, which means co-ordinating with any other open sessions on the system. Most desktop apps don't just sit in islands on their own. It's probably okay to live-update powertop on a single-user system when it's not being run, but anything much more integrated than that just isn't going to work very well.

Richard

On 16 September 2014 11:26, Reindl Harald h.reindl@thelounge.net wrote:

and the other side of the story are some hundret dist-upgrades i made in the past years with YUM (in a screen session for safety) while just continue to browse the web and read / answer email

So you're saying my technical analysis is incorrect because it happens to work for you? I've triaged many bugs to do with online and offline update failures, and if we're going to say that we actually care about the users data, it becomes increasingly hard to defend the "old" way of doing it. I'm sure I could find numerous bugs numbers where doing an online update made the session/terminal crash which of course leaves you with duplicate packages on your system which may or may not be fixable.

Richard

On Tue, Sep 16, 2014 at 8:29 AM, Przemek Klosowski przemek.klosowski@nist.gov wrote:

On 09/16/2014 06:33 AM, Richard Hughes wrote:

I've triaged many bugs to do with online and offline update failures, and if we're going to say that we actually care about the users data, it becomes increasingly hard to defend the "old" way of doing it. I'm sure I could find numerous bugs numbers where doing an online update made the session/terminal crash which of course leaves you with duplicate packages on your system which may or may not be fixable. Richard

OK, but this is means that we painted ourselves in the corner---something is wrong if my Android phone, which I don't have to reboot for updates, has higher uptime than my computer.

You don't have to restart your phone for app updates, but the app itself is restarted. For Android updates, it does require you to reboot the phone. Android updates happen rather infrequently. Your analogy is slightly flawed.

We are in a bind: on one hand, the best security practice is to upgrade daily to avoid emerging vulnerabilities; on the other hand daily reboots aren't really a nutritious alternative. Something has to give---which one do we do:

• create a separate daily security upgrade stream, curated to not require

reboots if at all possible

This has been suggested before and it isn't a terrible idea. It requires people to work on it though.

• follow Microsoft and do a fixed 'patch Tuesday' schedule instead of ASAP

updates

There have been suggestions around having "levels" of updates. A firehose level, which is the equivalent of today, and then a batched level, etc. Also needs people to work on it.

josh

On 2014-09-16, Richard Hughes hughsient@gmail.com wrote:

The much bigger issues is if you're using a D-Bus service like most applications seem to do (and most use quite a few system and session, directly and indirectly) then you've also got to co-ordinate and handle changing D-Bus API (which typically isn't versioned).

Maybe it's time to version the API.

Look at microkernel based systems which utilize messaging heavily and the API consumers (applications or another subsystems) have to be prepared for spurious API provider (server) restarts.

A server can refuse a message any time (especially if it does not understand the request). Simple operations are usualy implemented as a sequence of requests and responses where initial request obtains a descriptor (a session, a handle) and subsequent requests passe it as context (a session) identifier. If the server is restarted in between, the context identifier becomes unrecognized and it's up to the application to deal with it.

Just the fact that somebody calls functions over dbus instead of jumping into a shared library do not free them from maintaing API.

-- Petr

Dne 16.9.2014 v 12:21 Richard Hughes napsal(a):

On 16 September 2014 10:55, Zdenek Kabelac zkabelac@redhat.com wrote:

...

Just a thought - but wouldn't be better spend time to enlighten Gnome/Firefox developers how to write applications in a way the could be upgraded runtime

So, it's not just the application, it's every application and D-Bus service the application uses. Even glibc opens files-as-resources at

So it's time to fix D-Bus then as well!

If something is broken - it will not get fixed by hiding broken design behind reboot&upgrade.

Guys it's 21st. century - it's so much broken idea to reboot 'machine' because I want to upgrade i.e. Firefox - that I simply don't have words for it.

If you can't fix the Firefox (or some other broken tool) and you want to allow install of new version - then you need to allow i.e. parallel installs of such packages - it's that simple - I've been doing this more then 15 years ago at University - so really nothing new here...

While the old-firefox is in use - new one could be installed in parallel and the old one is removed when the last user closes app - of course this has another problem with dependency hell - again solvable thing - look at i.e. NixOS.

Just fix things in their core instead of hiding problem in pointless reboots.

Have any of those 'inventors' of reboot&update ever managed multiuser system?

This 'reboot' idea is just 'somehow' usable maybe for a single seat single user desktop - but not anywhere else.

Has Fedora given up Unix ??

Zdenek

Dne 16.9.2014 v 16:39 Miloslav Trmač napsal(a):

----- Original Message -----

...

Dne 16.9.2014 v 12:21 Richard Hughes napsal(a):

...

On 16 September 2014 10:55, Zdenek Kabelac zkabelac@redhat.com wrote:

...

Just a thought - but wouldn't be better spend time to enlighten Gnome/Firefox developers how to write applications in a way the could be upgraded runtime

So, it's not just the application, it's every application and D-Bus service the application uses. Even glibc opens files-as-resources at

So it's time to fix D-Bus then as well!

If something is broken - it will not get fixed by hiding broken design behind reboot&upgrade.

Quite true.

...

If you can't fix the Firefox (or some other broken tool) and you want to allow install of new version - then you need to allow i.e. parallel installs of such packages - it's that simple - I've been doing this more then 15 years ago at University - so really nothing new here...

Well, what we would need is:

1. Ability to keep multiple versions (both ABI-compatible and ABI-incompatible) of a single application or library or service installed and running at the same time.

Other distributions allow to install multiple version of same libraries - I've never understood the Fedora policy to recompile whole Fedora when a new version of library is released.

This policy design is quite 'show-stopper' for anything like this...

2. Ability to detect which processes depend on which versions of which components.

We already managed to brought in systemd....

3. Ability to automatically restart such processes without loosing state (either completely transparently or with some user notification for GUIs).

I'm not quite sure why we would need restart - simply delayed lazy release of unused packages would do the trick here - doing here state-full design is much more complex thing....

The primary use-case to target should be to allow reinstall user packages while they are in use.

This has all been done before, and can be done again. (And it would make at least half of the userbase clamoring for containers what they need, without playing ugly complex nontransparent namespace games.) But let’s be clear about it, 1. means completely changing our filesystem layout, and 3. means changing our process model to go way beyond int main(...).

It is technically possible, it is the right thing to do. Do we want to do it and can we do it?

Fedora made not much useful /usr move thing - so maybe it's time to think and design something really useful.

As mentioned there are OSes which do handle things much better...

And surprisingly even Systemd guru realized there is something broken with current filesystem layout - except solving it with Btrfs is not really a fix...

...

Has Fedora given up Unix ??

The Unix history is actually closer to “edit header files to match your hardware, then rebuild the kernel and userpace with (make world), and reboot“. Packaged applications with an ISV source and an update stream separate from the OS have certainly been built on top of Unix but have never been a major design focus. Arguably the whole point of a “Linux distribution” has been to get closer to the BSD-style single-kernel-and-userspace distribution updated always as a whole.

My view rather is - Fedora is taking feature-by-feature away from my box.

Zdenek

Dne 16.9.2014 v 17:08 Miloslav Trmač napsal(a):

----- Original Message -----

...

...

Well, what we would need is:

1. Ability to keep multiple versions (both ABI-compatible and

ABI-incompatible) of a single application or library or service installed and running at the same time.

Other distributions allow to install multiple version of same libraries

AFAIK only when the ABI (soname) is different. We also need to allow for the case when the ABI stays the same but the internal implementation changes (e.g. changing a format of a resource file that is used by the library, making old processes incompatible with the newly installed resource).

That's what I mean - when something changes in the library - soname should change. It's quite broken to release same version of library with the same name when there is different internal implementation. It's something library package maintainer should take care of...

There is also support for versioned symbols.

...

...

2. Ability to detect which processes depend on which versions of which

components.

We already managed to brought in systemd....

I can’t see how systemd helps. See the other discussions about Python/Ruby modules that leave no obvious trace of their dependency after being loaded into memory.

It has similar complexity and maybe packages could be described just like services - so similar thing could be possibly reused here ?

Anyway - I've not been thinking too deeply about this...

...

...

3. Ability to automatically restart such processes without loosing state

(either completely transparently or with some user notification for GUIs).

I'm not quite sure why we would need restart - simply delayed lazy release of unused packages would do the trick here - doing here state-full design is much more complex thing....

Because otherwise you end up with an old version of Firefox running for 60 days (or however long a laptop can run with suspends and no restarts; most people about never quit their browser), and that version of Firefox keeping an old version of a system-wide daemon running for 60 days as well..

Sure if user is able to run Firefox for 60 days (though my usually goes out of RAM in just a week or so, if it's not crashing earlier...) - then he is happy user.

I guess something may show to user some nice gui dialog warning like - 'Hey there is a new version installed - so 'restart' your browser to get new cool feature' (FF normally already detects upgrades and suggest restart)

I don't see nothing wrong with this case - it's per user and doesn't require upgrade - and if I've something important is running in my browser I could delay restart to the moment I don't care.

...

And surprisingly even Systemd guru realized there is something broken with current filesystem layout - except solving it with Btrfs is not really a fix...

The sad thing is that adding more workarounds like namespaces and the like really might be easier than agreeing on making a real change and getting it done :( But we will live with the costs forever.

Look at NixOS - there are no namespace or btrfs requirements....

Zdenek

Dne 16.9.2014 v 17:36 Miloslav Trmač napsal(a):

----- Original Message -----

...

...

...

...

2. Ability to detect which processes depend on which versions of which

components.

We already managed to brought in systemd....

I can’t see how systemd helps. See the other discussions about Python/Ruby modules that leave no obvious trace of their dependency after being loaded into memory.

It has similar complexity and maybe packages could be described just like services - so similar thing could be possibly reused here ?

Extra manual dependency information that would get obsolete?

Runtime removal of unused packages might be fun. It could be something like 'fstrim' tool to run through cron...

...

...

...

...

3. Ability to automatically restart such processes without loosing state

(either completely transparently or with some user notification for GUIs).

I'm not quite sure why we would need restart - simply delayed lazy release of unused packages would do the trick here - doing here state-full design is much more complex thing....

Because otherwise you end up with an old version of Firefox running for 60 days (or however long a laptop can run with suspends and no restarts; most people about never quit their browser), and that version of Firefox keeping an old version of a system-wide daemon running for 60 days as well..

Sure if user is able to run Firefox for 60 days (though my usually goes out of RAM in just a week or so, if it's not crashing earlier...) - then he is happy user.

Until an exploit _that they have already installed an update for_ steals their data.

...

I guess something may show to user some nice gui dialog warning like - 'Hey there is a new version installed - so 'restart' your browser to get new cool feature' (FF normally already detects upgrades and suggest restart)

No, that’s not good enough. The OS should restart it for the user (other OSes already can do this). (In the case of user-invisible components there is no other option, and for user-visible GUIs it is also the right thing to do.)

As long as Firefox can't restart 'unnoticeable' while playing youtube video - this is not an option - when admins updates machine - it just can't kill every users' running firefox.

It's fine to place a warning somewhere and require restart with some 'many hours' timeout - but for almost all "Firefox" updates - it's good enough to restart it just sometimes latest - and for those 'have to restart' - it still upon admins policy - not upon Fedora package maintainer....

Zdenek

Dne 16.9.2014 v 17:29 Matthias Clasen napsal(a):

On Tue, 2014-09-16 at 16:21 +0200, Zdenek Kabelac wrote:

...

Has Fedora given up Unix ??

This thread has gone quite far out into the weeds. It started with a fairly concrete question: can we improve the offline update experience by requiring only a single reboot, instead of two ?

That's the core of the problem, while other systems have been trying for years to get rid of this nonsense (updates requiring reboots), we are moving the opposite direction. Technical reasons aside(*), this is NOT right for our users.

(*) I agree offline updates are the safest way technically but that should be only regarded as a workaround for a more fundamental problem, lying deep below the desktop level.

On 09/17/2014 11:54 AM, Bastien Nocera wrote:

All those OSes require reboots when updating the OS.

Define OS.

Firefox is definitely not OS. While systemd is OS. I am fine with reboot after systemd upgrade, but not after upgrading Firefox.

And there we have lots of packages in between those two. Where you would draw a line? What is OS? (And this brought us to core of Fedora.next discussion :) )

On 22 October 2014 04:31, Lennart Poettering mzerqung@0pointer.de wrote:

On Wed, 17.09.14 13:58, Miroslav Suchý (msuchy@redhat.com) wrote:

...

On 09/17/2014 11:54 AM, Bastien Nocera wrote:

...

All those OSes require reboots when updating the OS.

Define OS.

Firefox is definitely not OS. While systemd is OS. I am fine with reboot after systemd upgrade, but not after upgrading Firefox.

Well, on Fedora and Unixes the apps are just part of the OS, they are all dropped into /usr, and not isolated out.

It would be great if we could nicely isolate the apps from the OS so that we can restart the apps independently from the OS, but this requires isolating things first.

We are working on app sandboxes, and they will make this available, but it's the traditional Linux model cannot really deliver this.

Well it depends on the traditional model that you are going to refer to. In the model of the Unix systems I had to set up in the late 1980's and the 1990's... there was a separation in that items for the OS were in the root directories (say /bin and /sbin), and stuff that was not OS critical but user critical were in /usr/{bin,sbin}. And then local critical were in /opt or /usr/local depending on the OS.

We (Linux distributions) lost that distinction sometime in the past decade and then undid it completely with the various /usr/ merges. [This isn't meant to open that can of worms.. the distinction was broken before the merge.. the merge just made it clear.]

I am not sure how to best move from here. A complete reinvent of hierarchies is always tempting.. but it has always been the deathknell of every OS that has done it in the past due to chasm crossing issues (too much old stuff needing old things causing any benefits from new stuff to be actual detriments). Doing a more thorough job of packaging items so that system only items were in /bin,/lib, etc has never worked because too many things sit between the two. [its a user component AND a system component!] At best I can say it comes down to operating systems are too damn complicated and I need to go back to potato farming :)

Lennart

-- Lennart Poettering, Red Hat -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

On Wed, Oct 22, 2014 at 4:45 PM, Tom Rivers tom@impact-crater.com wrote:

On 10/22/2014 06:31, Lennart Poettering wrote:

...

It would be great if we could nicely isolate the apps from the OS so that we can restart the apps independently from the OS, but this requires isolating things first.

Isn't the differentiation between kernel space and user space sufficient for identifying what is the OS and what is an application? I know Windows blurred those lines during the Browser Wars with Netscape, but to my knowledge that separation still exists in Linux.

No the OS is more than just a kernel.

On 17.09.2014 13:58, Miroslav Suchý wrote:

On 09/17/2014 11:54 AM, Bastien Nocera wrote:

...

All those OSes require reboots when updating the OS.

Define OS.

Firefox is definitely not OS. While systemd is OS. I am fine with reboot after systemd upgrade, but not after upgrading Firefox.

the important point in that case is not reboot after upgrading Firefox but *before* upgrading Firefox, which means that at the time of the upgrade no Firefox will be running and potentially crashing because one of the 100s of DSOs it loads on-demand has changed in an incompatible way.

there used to be quite a few ABRT crashes reported against desktop applications with impossible looking stack traces (although with the automatic micro-reports it's less of a problem nowadays as they are easier to ignore), and sometimes the reporter gives feedback that the application was running across a yum update...

On Wed, Oct 22, 2014 at 11:35:00PM +0100, Ian Malone wrote:

On 22 October 2014 20:07, Michael Stahl mstahl@redhat.com wrote:

...

On 17.09.2014 13:58, Miroslav Suchý wrote:

...

On 09/17/2014 11:54 AM, Bastien Nocera wrote:

...

All those OSes require reboots when updating the OS.

Define OS.

Firefox is definitely not OS. While systemd is OS. I am fine with reboot after systemd upgrade, but not after upgrading Firefox.

the important point in that case is not reboot after upgrading Firefox but *before* upgrading Firefox, which means that at the time of the upgrade no Firefox will be running and potentially crashing because one of the 100s of DSOs it loads on-demand has changed in an incompatible way.

there used to be quite a few ABRT crashes reported against desktop applications with impossible looking stack traces (although with the automatic micro-reports it's less of a problem nowadays as they are easier to ignore), and sometimes the reporter gives feedback that the application was running across a yum update...

While it can be a bit confusing the first time it happens to you, the solution is just to start and stop firefox again in that case. If it the goal is just to tidy ABRT crash reports (and I'm not sure it is) then forcing reboots on users wouldn't be very kind.

Beyond cleaning crash reports and reducing them, there is also the perception of users.

If we want to have the same reputation as some older windows with "this software crashed and I do not know why" from our users, then having regular random crashes after a random delay after upgrade is the way to go. If we still aim for "the software is solid and stable", then we should try to reduce the random crash.

On 23 October 2014 09:56, Miroslav Suchý msuchy@redhat.com wrote:

The upgrade can last one hour (more or less).

An hour?! Most of my offline updates take a few tens of seconds with F21. Is this on fairly up-to-date SSD hardware? Are any specific packages taking longer than the others?

Richard

Am 23.10.2014 um 12:52 schrieb Daniel P. Berrange:

On Thu, Oct 23, 2014 at 11:43:29AM +0100, Richard Hughes wrote:

...

On 23 October 2014 09:56, Miroslav Suchý msuchy@redhat.com wrote:

...

The upgrade can last one hour (more or less).

An hour?! Most of my offline updates take a few tens of seconds with F21. Is this on fairly up-to-date SSD hardware? Are any specific packages taking longer than the others?

Don't assume everyone is using SSDs with Fedora....

that assumption is sadly a common mistake on many places and leads to implement things less efficient because "it's fast enough on 'fairly' recent hardware)

frankly it will take many years before i replace my 4x2 TB RAID10 disks by 4x2 TB SSD just because the price and no i won't re-install from scratch to just have the OS only on SSD and my virtual machines (which are the reason for that large disks) stay on slower disks

On 10/23/2014 12:57 PM, Reindl Harald wrote:

...

...

The upgrade can last one hour (more or less).

An hour?! Most of my offline updates take a few tens of seconds with F21. Is this on fairly up-to-date SSD hardware? Are any specific packages taking longer than the others?

an hour was surely exaggerated

Not at all. I'm not updating daily. I upgraded my machine IIRC 2-3 weeks ago. So lets benchmark it and provide you real data.

My machine have classic magnetic disk, however in SW RAID1. Timing cached reads: 12236 MB in 2.00 seconds = 6124.59 MB/sec Timing buffered disk reads: 412 MB in 3.00 seconds = 137.12 MB/sec Fedora 21, 16 GB RAM (2GB free), 8 CPU cores, swap available but none used

I run "dnf upgrade" and I have been offered 853 packages and 1.3 GB to download. Download lasted 3mins 20secs. Then installation started and since beginning "transaction started" till the end lasted exactly 53 minutes. No specific package is blocking the process, dnf was chewing packages one by one in steady pace. Veryfing phase lasted ~4 minutes, so it means approximately 3 second per package, which is what I am seeing on screen.

And this is nothing exceptional. I see similar times across all machines I maintain. When I'm updating box of my mother (old EeeBox, updating aprox every 3 months) then the time is usually 3 hours (however ~1 hour is just download phase).

Am 24.10.2014 um 12:02 schrieb Mathieu Bridon:

On Fri, 2014-10-24 at 12:00 +0200, Miroslav Suchý wrote:

...

I'm not updating daily. I upgraded my machine IIRC 2-3 weeks ago. So lets benchmark it and provide you real data.

My machine have classic magnetic disk, however in SW RAID1. Timing cached reads: 12236 MB in 2.00 seconds = 6124.59 MB/sec Timing buffered disk reads: 412 MB in 3.00 seconds = 137.12 MB/sec Fedora 21, 16 GB RAM (2GB free), 8 CPU cores, swap available but none used

I run "dnf upgrade" and I have been offered 853 packages and 1.3 GB to download. Download lasted 3mins 20secs. Then installation started and since beginning "transaction started" till the end lasted exactly 53 minutes. No specific package is blocking the process, dnf was chewing packages one by one in steady pace. Veryfing phase lasted ~4 minutes, so it means approximately 3 second per package, which is what I am seeing on screen.

And this is nothing exceptional. I see similar times across all machines I maintain. When I'm updating box of my mother (old EeeBox, updating aprox every 3 months) then the time is usually 3 hours (however ~1 hour is just download phase).

More than anything, doesn't this just shows that we simply push way too many updates in Fedora?

no

* first: the above is Rawhide/Alpha * second: the reason i run Fedora and not Debian/RHEL is fast updates * third: nobody should apply updates every 3 weeks but that above is Alpha and so no "production" machine

On 10/24/2014 12:59 PM, Richard Hughes wrote:

853 must be close to every package on your box...

Nope. $ rpm -qa |wc -l 4337

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 09/16/2014 04:02 PM, Lukáš Tinkl wrote:

Dne 16.9.2014 v 17:29 Matthias Clasen napsal(a):

...

On Tue, 2014-09-16 at 16:21 +0200, Zdenek Kabelac wrote:

...

Has Fedora given up Unix ??

This thread has gone quite far out into the weeds. It started with a fairly concrete question: can we improve the offline update experience by requiring only a single reboot, instead of two ?

That's the core of the problem, while other systems have been trying for years to get rid of this nonsense (updates requiring reboots), we are moving the opposite direction. Technical reasons aside(*), this is NOT right for our users.

While that may be a wonderfully ideal state to reach (and certainly worth discussion), I agree with Matthias that this has gone far afield of my original question.

I'd like to try to tie this thread back to answering a single question: Without a fundamental redesign of the way things work, which would be a VERY large and long-term effort, can we improve the user experience for people *today* with fairly little effort?

If the answer to that question is "yes", is there anything fundamentally wrong with adding that workaround while we solve the situation in a more complete manner?

I think this is a great example of the adage "perfect is the enemy of the good". We have a situation that everyone seems to agree is less than ideal and we have two proposed solutions: one that probably requires only a few weeks of work and testing and one that probably requires a fundamental change to the way the system works and therefore could take years to accomplish.

Ask yourself this question: are you as a user patient enough to wait for a perfect solution or are you happy with incremental improvement?

(*) I agree offline updates are the safest way technically but that should be only regarded as a workaround for a more fundamental problem, lying deep below the desktop level.

On Thu, 2014-10-02 at 07:53 -0400, Stephen Gallagher wrote:

On Fri, 2014-09-12 at 10:46 -0400, Stephen Gallagher wrote:

...

== The Problem ==

It is very common for users to have systems with encrypted root partitions (or even just /var and /etc). This may be due to a personal concern for their data or a corporate policy mandating full-disk encryption. Disk encryption requires a password (or other more complicated credentials) be be presented just after the kernel is booted and before the drives can be mounted and their data read.

With the current implementation of the offline updates in Fedora, this leads to a very unpleasant user experience when updating. We offer two ways to perform offline updates in the default user environment of Fedora Workstation: "Install updates and reboot" or "Install updates and shut down".

With "Install updates and reboot", the behavior is as follows:

1. The system shuts down and initiates an ACPI reboot.
2. The system presents the kernel boot menu and then starts the

updater kernel. 3) The system presents the user with a password prompt for the disk encryption (possibly more than one, if the system is configured with different passwords for different partitions). 4) The offline updates occur. 5) The system shuts down and initiates an ACPI reboot. 6) The system presents the kernel boot menu and then starts the standard (possibly updated) kernel. 7) The system presents the user with a password prompt for the disk encryption (possibly more than one, if the system is configured with different passwords for different partitions). 8) The system completes booting.

During this experience, the user has been required to enter their disk encryption password *twice*. The same is true for the "Install and shut down" case, except that the two passwords are separated by some actual wallclock time.

== Proposed Improvements ==

We could significantly improve this situation by allowing the system to drop directly from the interactive system into the updater environment without doing a full reboot or relaunching the kernel.

Lennart, would it be possible to set up a special systemd target for performing updates that would essentially stop all processes except for systemd and then apply the updates?

In an ideal world, it would then also be possible after update is completed to restore operation to the standard boot targets of systemd so that the system comes back up without having to perform a total reboot. The exceptional case would of course be that in which either the kernel, libc or systemd[1] needed to be updated, in which case a reboot could be performed.

In this scenario, we can reduce the number of encrypted disk challenges to at most a single one, and that only if absolutely minimal plumbing packages saw an update.

I'd very much like to hear from the plumbers on this matter.

[1] I'm told that this might not be necessary; that systemd can re-exec itself to pick up its own updates. That would reduce the scope presumably to "only the kernel" forcing reboots.

I'm bumping this thread to get comments from Lennart (CCed). There's a lot of chatter in the conversation, but I'd very much like to hear an answer to the specific questions I posed in this first email.

Trying one more time to get Lennart to chime in here.

On Wed, 22.10.14 14:11, Roberto Ragusa (mail@robertoragusa.it) wrote:

On 10/21/2014 10:02 PM, Lennart Poettering wrote:

...

Maybe that's actually a strategy to adopt here: upload the encryption keys into the firmware as efi vars, and then pull them out on next boots or so (assuming that efi vars can be marked to survive soft reboots without making them fully persistent...)

Hmmm, surrendering your encryption keys to the only software part which you do not have control on?

The firmware runs at the highest priviliges anyway, it can do whatever it wants... You don't have to "surrender" you keys to it. If it wants them it can take them anyway...

Lennart