While waiting on/working on getting Flatpaks building in the Fedora infrastructure, I took some time to write a tool to collect information about applications we might want to package as Flatpaks and how they correlate with the runtime. You can find the result at:
https://fishsoup.net/misc/flatpak-runtime-reports/applications.html
(more reports can be clicked through at the top.)
The data source for this are:
A) The Fedora appstream B) The Fedora package set C) The Flathub appstream D) ODRS reviews (odrs.gnome.org)
[ Scripts are in https://src.fedoraproject.org/modules/flatpak-runtime/ - but currently depend on only-on-my-computer versions of fedmod. Once the F28 hybrid compose is sorted out, I can start working on pushing those changes out. ]
Some initial observations:
* The most popularly reviewed applications on ODRS are either a) packaged in Fedora or b) can't be packaged in Fedora. This is somewhat circular because a large fraction of ODRS reviews come from Fedora.
* There are a lot of popularly reviewed applications in Fedora that are *not* on Flathub. Some are don't make sense or are very hard to Flatpak (GParted, GNOME Boxes), but plenty of others (qBittorent, Stellarium, etc.) should be reasonably straightforward.
* Many applications have a *lot* of dependencies that would need to be bundled given the current runtime contents. But often these are stray - e.g. gedit pulls 41 dependencies, but if it would only require 4 dependencies if it was fixed to depend on gvfs-client rather than gvfs. Or as another example, applications that depend on the KDE Frameworks pull in a pile of perl packages because of /usr/bin/preparetips5 in kf5-kconfigwidgets which seems marginally useful at runtime.
* A few packages would get bundled into many, many applications - and should definitely be added to the runtime (qt-settings, glx-utils). But on the other hand there are a huge set of packages that get bundled into exactly 1 out of the 800 packages examined here (maybe 1700/3700.)
* I'm not sure yet whether a "flatpak-shared-deps" module with dependencies built into /app is useful. The pros of it are: - Things built in it will be shared on disk between different apps because of ostree deduplication. (But not when downloading via OCI.) - Things that are hard to build (think perl, texlive) only need to be worked out once.
* The bulk of the runtime packages get used very broadly, but there is a short tail of stuff in there that *no* app requires. Some of this I've already identified as stray and to be removed (krb5-server, say.)
Once we get the building infrastructure going, we can start picking off some of the low-hanging fruit and looking at the harder cases.
Regards, Owen
On Wed, Feb 14, 2018 at 03:52:06PM -0500, Owen Taylor wrote:
While waiting on/working on getting Flatpaks building in the Fedora infrastructure, I took some time to write a tool to collect information about applications we might want to package as Flatpaks and how they correlate with the runtime. You can find the result at: https://fishsoup.net/misc/flatpak-runtime-reports/applications.html
Thanks Owen. This is super-interesting!
- I'm not sure yet whether a "flatpak-shared-deps" module with
dependencies built into /app is useful. The pros of it are:
- Things built in it will be shared on disk between different apps because
of ostree deduplication. (But not when downloading via OCI.)
- Things that are hard to build (think perl, texlive) only need to be
worked out once.
What are the cons?
- The bulk of the runtime packages get used very broadly, but there is a
short tail of stuff in there that *no* app requires. Some of this I've already identified as stray and to be removed (krb5-server, say.)
Once we get the building infrastructure going, we can start picking off some of the low-hanging fruit and looking at the harder cases.
A lot of these things -- like removing extraneous deps -- will improve the distro in general, too.
On Wed, Feb 14, 2018 at 4:00 PM, Matthew Miller mattdm@fedoraproject.org wrote:
On Wed, Feb 14, 2018 at 03:52:06PM -0500, Owen Taylor wrote:
- I'm not sure yet whether a "flatpak-shared-deps" module with
dependencies built into /app is useful. The pros of it are:
- Things built in it will be shared on disk between different apps
because
of ostree deduplication. (But not when downloading via OCI.)
- Things that are hard to build (think perl, texlive) only need to be
worked out once.
What are the cons?
The main con from my perspective is that it takes agency out of the hands of the person creating the flatpak. Instead of the simple proposition:
"If it's not in the runtime, it should be in your application module"
the flatpaker might instead be in a position where they get blocked because they decide some dep belongs in flatpak-shared-deps and have to wait for that to be resolved. It doesn't actually make things worse for the flatpak builder, but it could make it more confusing.
Owen
On Wed, 2018-02-14 at 15:52 -0500, Owen Taylor wrote:
- There are a lot of popularly reviewed applications in Fedora that
are *not* on Flathub.
Hi, just out of interest, what with the applications which have their flatpak scripts, but which do not fit Flathub?
Maybe this had been answered already somewhere, I only missed it, thus if so, I apologize for repeating the question, but just in case: will Fedora provide its own Flatpak repository, thus one could rely on versions of the host dependencies? If not, and the plan is to use Flathub exclusively, will Fedora push the applications there instead of the upstream/package maintainer in case where the upstream/package maintainer is not in favor of github, the place where Flathub is hosted? But I believe Fedora cannot use Flathub this way, because it cannot guarantee host dependencies there and also because Flathub is meant to be unrelated to any distribution (I understand it that way at least). Thanks and bye, Milan
----- Original Message -----
On Wed, 2018-02-14 at 15:52 -0500, Owen Taylor wrote:
- There are a lot of popularly reviewed applications in Fedora that
are *not* on Flathub.
Hi, just out of interest, what with the applications which have their flatpak scripts, but which do not fit Flathub?
Maybe this had been answered already somewhere, I only missed it, thus if so, I apologize for repeating the question, but just in case: will Fedora provide its own Flatpak repository, thus one could rely on versions of the host dependencies?
No, the Flatpak and host are separate. I'm guessing that you're worried about mismatches between evolution-data-server and its clients. There's really no other way to make this work than to work towards a stable, and versioned API.
If not, and the plan is to use Flathub exclusively, will Fedora push the applications there instead of the upstream/package maintainer in case where the upstream/package maintainer is not in favor of github, the place where Flathub is hosted?
The upstream maintainer doesn't have to use Flathub, and GitHub, but they'd have to replicate the setup. Which is one of the reasons why people use Flathub to build and distribute those rather than doing it themselves separately.
But I believe Fedora cannot use Flathub this way, because it cannot guarantee host dependencies there and also because Flathub is meant to be unrelated to any distribution (I understand it that way at least).
Flatpak cannot guarantee host dependencies either. That's both a feature and a bug, making it harder to build applications that rely on specific versions of host services, outside the sandbox, and easier to upgrade applications without pulling new dependencies at the host level.
Hi,
On Fri, 2018-02-16 at 04:50 -0500, Bastien Nocera wrote:
No, the Flatpak and host are separate. I'm guessing that you're worried about mismatches between evolution-data-server and its clients.
Heh, you just say 'no' and then change it to 'yes'. Some applications do depend on system services, regardless what they are built against.
There's really no other way to make this work than to work towards a stable, and versioned API.
It's not about API by any means. The (D-Bus and code) API can be stable, but the underlying code can change. I know that software like GNOME Calendar and GNOME Contacts and others use mentioned evolution- data-server (eds) and they compile against some version, but then use the host version of the D-Bus services. It's fine as long as you do not care of functionality. It happens often that some bug is noticed on the client side (being it GNOME Calendar, GNOME Contacts, Evolution,...), but the actual fix for it lands in the eds, like into the builtin backend. This fix doesn't change API, no soname version bump needed, no D-Bus service version number increased). That means that the Flatpak version can be built against eds which has the bug fixed, but whether it's truly fixed depends on the host system. That's odd. Adding eds into Flatpak is not correct from the point of sharing the same data between various applications and the system (GNOME Shell) itself (unless some portals of Flatpak or what that idea to catch on this was named).
If not, and the plan is to use Flathub exclusively, will Fedora push the applications there instead of the upstream/package maintainer in case where the upstream/package maintainer is not in favor of github, the place where Flathub is hosted?
The upstream maintainer doesn't have to use Flathub, and GitHub, but they'd have to replicate the setup. Which is one of the reasons why people use Flathub to build and distribute those rather than doing it themselves separately.
Yes, not talking that the upstream might tell the users how to add they Flatpak repository to their system. It didn't answer my original question though.
Flatpak cannot guarantee host dependencies either. That's both a feature and a bug, making it harder to build applications that rely on specific versions of host services, outside the sandbox, and easier to upgrade applications without pulling new dependencies at the host level.
I agree. Flatpak cannot be an answer to everything, not for such complicated tasks like shared data between the host system and various either Flatpak or native applications. Simple standalone applications work fine with Flatpak, just not those complicated. At least not right now.
My main concern is about Fedora. As I said, maybe it had been already answered elsewhere, I only didn't catch the answer. Fedora should not use some 3rd-party "repository" to deliver applications to users, it should have control on the applications being delivered in the distribution (reproducible builds, anyone?), thus provide its own Flatpak repository, in case it's really going to use Flatpak to deliver content to their users. Whether such Flatpak repository should be usable by any Fedora version onward or only being used by single Fedora version is another question.
Just my opinion. Bye, Milan
----- Original Message -----
Hi,
On Fri, 2018-02-16 at 04:50 -0500, Bastien Nocera wrote:
No, the Flatpak and host are separate. I'm guessing that you're worried about mismatches between evolution-data-server and its clients.
Heh, you just say 'no' and then change it to 'yes'. Some applications do depend on system services, regardless what they are built against.
Sorry, it was clearer in my head. The "no" applies to library APIs, the "yes" applies to D-Bus "remote" APIs.
There's really no other way to make this work than to work towards a stable, and versioned API.
It's not about API by any means. The (D-Bus and code) API can be stable, but the underlying code can change. I know that software like GNOME Calendar and GNOME Contacts and others use mentioned evolution- data-server (eds) and they compile against some version, but then use the host version of the D-Bus services. It's fine as long as you do not care of functionality. It happens often that some bug is noticed on the client side (being it GNOME Calendar, GNOME Contacts, Evolution,...), but the actual fix for it lands in the eds, like into the builtin backend. This fix doesn't change API, no soname version bump needed, no D-Bus service version number increased). That means that the Flatpak version can be built against eds which has the bug fixed, but whether it's truly fixed depends on the host system.
That will be the case for any host-side feature, especially the ones that rely on online services (gnome-online-accounts is probably another), or the ones that rely on hardware-related bug fixes. That's par for the course, and that will probably mean more backporting and stable releases.
That's odd. Adding eds into Flatpak is not correct from the point of sharing the same data between various applications and the system (GNOME Shell) itself (unless some portals of Flatpak or what that idea to catch on this was named).
On mobile, this would be split up more, so the calendar backend would live in the Calendar app, the Notes backend in the Notes app, etc. This leaves the sign-on on the host side. But they can do that because they have blessed front-ends which we don't in GNOME or Fedora.
On Fri, 2018-02-16 at 07:46 -0500, Bastien Nocera wrote:
On mobile, this would be split up more, so the calendar backend would live in the Calendar app, the Notes backend in the Notes app, etc
Hi, only to clarify the situation on the eds side, the calendar (events), notes (memos) and to do (tasks) are one backend only, serving all three kinds, with exception of Google Tasks backend, which does only Google Tasks and nothing else. After all, the only thing which changes is the component kind, not the storage itself (the code is reused as much as possible). Other similar services can have this done differently.
Note there are claims that the evolution-addressbook|calendar-factory- subprocess processes are too much for architectures with low memory, thus there's kind of requirement to limit the subprocesses as much as possible. It's against the intention of "if one backend crashes, only that kind of sources is lost, not everything", but I agree that some compromises are needed here (and the code should not crash). I mean, more splitting means more things to deliver and eventually to have running in the background, which is bad in certain situations. Would Flatpak-ed Fedora be usable in such environments at all? Bye, Milan
On Fri, Feb 16, 2018 at 3:56 AM, Milan Crha mcrha@redhat.com wrote:
On Wed, 2018-02-14 at 15:52 -0500, Owen Taylor wrote:
- There are a lot of popularly reviewed applications in Fedora that
are *not* on Flathub.
Hi,just out of interest, what with the applications which have their flatpak scripts, but which do not fit Flathub?
Maybe this had been answered already somewhere, I only missed it, thus if so, I apologize for repeating the question, but just in case: will Fedora provide its own Flatpak repository, thus one could rely on versions of the host dependencies?
Fedora will provide it's own Flatpak repository - with the intent that it will be enabled out-of-the-box on Fedora systems. But anybody running on another operating system could also access the Fedora repository - and given the stats I came up with, some people will likely want to do so.
If not, and the plan is to use Flathub exclusively, will Fedora push the applications there instead of the upstream/package maintainer in case where the upstream/package maintainer is not in favor of github, the place where Flathub is hosted? But I believe Fedora cannot use Flathub this way, because it cannot guarantee host dependencies there and also because Flathub is meant to be unrelated to any distribution (I understand it that way at least).
My initial take on this is that it's OK if a Flatpak in the Fedora repository doesn't work if a host service is missing - imagine a GNOME Boxes Flatpak that only works if qemu/kvm/etc are installed - but it should not work *gracefully*. It shouldn't just segfault and not show a window.
Owen
desktop@lists.fedoraproject.org
-
Bastien Nocera -
Matthew Miller -
Milan Crha -
Owen Taylor