devel October 2019

devel@lists.fedoraproject.org

201 participants
346 discussions

libdav1d SONAME bump
by Robert-André Mauchin 19 Nov '19

19 Nov '19

Hello, Dav1d 0.5.0 was published today and brings a SONAME bump from libdav1d.so. 2.0.0 to libdav1d.so.3.0.0. I will be updating it next week on F31/32, consumers of these libraries (ffmpeg, xine-lib, vlc) will need to rebuild their packages. Best regards, Robert-André

4 6

Backend storage clean-up
by Jakub Kadlcik 15 Nov '19

15 Nov '19

Hello, we have decided that it is a time to go through Copr backend data and clean it up a little. Due to several bugs forgetting to remove backend data after removing their records from the database, _temporarily_ backing up some folder here and there, etc, we ended up with a lot of unknown data, that should have been already deleted. So, we wrote a script [1] to find data that should be deleted, and we ... haven't deleted it yet. Instead, we put it aside in a case there were any false-positives by the script. If you want to be extra safe, I am attaching a list of all of those files, you may check it. If you find out, that we accidentally want to remove any of your data that is still being used, please contact us immediately! Otherwise, we are going to delete it once we are running out of disk space (which may not take a lot of time) [1] https://pagure.io/copr/copr/blob/master/f/backend/run/copr_print_results_to… Thank you for understanding, Jakub

2 1

Fedora 32 System-Wide Change proposal: Modules in Non-Modular Buildroot
by Ben Cotton 14 Nov '19

14 Nov '19

https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot Enable module default streams in the buildroot repository for modular and non-modular RPMs. == Summary == This Change (colloquially referred to as "Ursa Prime") enables the Koji build-system to include the RPM artifacts provided by module default streams in the buildroot when building non-modular (or "traditional") RPMs. == Owner == * Name: [[User:Sgallagh| Stephen Gallagher]] * Email: sgallagh(a)redhat.com * Responsible WG: Modularity WG == Detailed Description == As a major part of the Modularity design, we have a concept of default module streams. These streams are built as modules, but the RPM artifacts they deliver are intended to be used just like non-modular RPMS. The aspirational goal is that a user of the system who never executes a module-specific command (such as `dnf module install nodejs:8`) should experience no meaningful changes in behavior from how they would interact with a completely non-modular system. In practice, this may mean that the informational output of package managers may indicate that modules are being enabled and used, but a user that does not have a specific reason to interact with the selection of a module stream should have that managed on their behalf. Similarly, the experience for package maintainers of non-modular packages should be unaffected by an RPM dependency moving from the non-modular repository into a default module stream. Up to the present, this has not been the case; no module stream content has been available in the non-modular buildroot for other packages to consume. Koji builds of non-modular RPMs have had only the other non-modular RPMs from that release available to their buildroots. In contrast, building on local systems has access to both the non-modular RPMs and the RPMs from any of the default module streams. With this Change, Koji builds will have the same behavior and be able to depend on content provided by default module streams. It also enables the same behavior for Modular builds: the `platform` stream will now include the contents of the default module streams for each release and do not need to be explicitly specified in the modulemd `buildrequires`. Note: This Change does not address the other major Modularity issue we are facing around distribution upgrades with differing default streams. When discussing this Change, please keep that topic separate. == Benefit to Fedora == This will simplify the lives of package maintainers in Fedora in two primary ways. I'll use a hypothetical example of the Node.js interpreter and a JSApp package which is capable of running on Node.js 10 or 12 (but requires newer features than are provided by Node.js 8). Additionally, the JSApp package requires the same versions of Node.js at build-time. * Fedora 29 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module streams. The `nodejs:10` stream is set as the default stream. * Fedora 30 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module streams. The `nodejs:10` stream is set as the default stream. * Fedora 31 ships `nodejs:10` and `nodejs:12` module streams. The `nodejs:12` stream is set as the default stream. The `nodejs:14` stream will likely become available during the F31 lifetime. * Fedora 32 ships `nodejs:10` and `nodejs:12` module streams. The `nodejs:12` stream is set as the default stream. The `nodejs:14` stream will likely become available during the F32 lifetime. On Fedora 29 through 31, the Node.js package maintainer needs to build the `nodejs:10` package both as a module and as a non-modular RPM in the distribution so that the JSApp package can be built. With this Change, the Node.js package maintainer in Fedora 32+ will only need to build the various Node.js streams and make one of them the default stream. The packages from it will then be added to the buildroot for non-modular packages. This will also make the packaging process somewhat more efficient, as the maintainer needs only to manage the module stream and the MBS will build it for all configured platforms. Similarly, from the perspective of dependent maintainers, there will no longer be anxiety about needing to move their package to a module if one or more of their dependencies drops their non-modular version in favor of a default stream. Their builds will continue to work as they do today. == Scope == * Proposal owners: # Update Packaging Guidelines with [https://pagure.io/modularity/issue/146#comment-600328 requirements] for module default streams # Create a Pungi configuration to generate the buildroot from the default module streams. # Include `default_modules_scm_url` in the platform virtual module specification # Configure Koji tags for inheriting the new modular-defaults buildroot into the standard buildroot * Other developers: Packagers of default module streams will be required to conform to the [https://pagure.io/modularity/issue/146#comment-600328 policy] regarding visibility of stream artifacts. Any default module stream that is not in compliance by one week before Beta Freeze will cease to be a default stream. * Release engineering: # https://pagure.io/releng/issue/8879 - Create pungi config for Rawhide/F32 ursa prime buildroot # https://pagure.io/releng/issue/8880 - Include `default_modules_scm_url` in platform 31 virtual module # https://pagure.io/releng/issue/8881 - Configure Koji tags for inheriting f32-modular-buildroot * Policies and guidelines: The Modularity Packaging Guidelines will need to be updated to indicate the strict requirements on default streams. * Trademark approval: N/A (not needed for this Change) == Upgrade/compatibility impact == This change is on the build-system side of things and should not impact the upgrade process directly. == How To Test == # Build a modular stream # Make that stream a default stream (or a buildroot override) # Build a non-modular RPM that requires an artifact RPM from the modular stream. == User Experience == This should not change the end-user experience. == Dependencies == Nothing known that isn't listed in the scope. == Contingency Plan == * Contingency mechanism: Disable the buildroot inheritance in Koji to revert to the current behavior. * Blocks release? Ambiguous: lack of complete implementation may indirectly cause blocking issues. * Blocks product? No == Documentation == == Release Notes == None needed, the Change is not user-facing. -- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis

29 109

Add a rule to have a compose when Fedora branched
by jkonecny＠redhat.com 14 Nov '19

14 Nov '19

Hello everyone, I'm Anaconda developer and I'm also taking care about our infrastructure and this Fedora release brought me a plenty of "unnecessary" work thanks to the fact that compose for Fedora 31 was not available until a week before beta freeze. That is too late. I wasn't the only one who had these problems, copr had issues for Fedora 31 and couldn't enable chroot so they had to do changes to correct these broken things. And I'm not talking about Fedora QA team which couldn't test almost anything before beta freeze. The problem is that when we don't have a compose we don't have packages for testing and then more and more changes are getting in but we are not able to check if they are working. If we don't have packages the mock can't properly work and you are not able to do a system upgrade. The only test point is compose but that is just a small portion. Not being able to test Fedora for a few weeks is situation which should not happen. To make things even worse there was a switch to python 3.8 on Rawhide which wasn't really prepared (pylint did not worked). So for a few days we were with broken Fedora 31 and Rawhide too, so most of our tests were not working. I would really said that we were programming in the dark. No tests, no check that changes are working. It took me almost a week to make everything working again not talking about time spend waiting for the compose to be available. I want to ask for an improvement here. Ideal solution for me would be to add rule that there have to be compose to do the branching and if the compose fails then the branching won't happen. Not sure if this is doable or how hard it would be to implement a similar rule, however, it would be an ultimate solution. Then, the compose blocker bugs had to be solved on Rawhide where they should be solved. Please tell me what should I do next. Should I file a FESCO ticket to add this rule? Best Regards, Jirka

10 58

Fedora 31 Elections: Nomination period open
by Ben Cotton 13 Nov '19

13 Nov '19

Today we are starting the Nomination & Campaign period during which we accept nominations to the "steering bodies" of the following teams: * FESCo (Engineering) (5 seats) [1] * Fedora Council (1 seat) [2] * Mindshare (1 seat) [3] This period is open until 2019-11-13 at 23:59:59 UTC. Candidates may self-nominate. If you nominate someone else, please check with them to ensure that they are willing to be nominated before submitting their name. The steering bodies are currently selecting interview questions for the candidates. Nominees submit their questionnaire answers via a private Pagure issue. The Election Wrangler or their backup will publish the interviews to the Community Blog before the start of the voting period. Please note that the interview is mandatory for all nominees. Nominees not having their interview ready by end of the Interview period (2019-11-20) will be disqualified and removed from the election. As part of the campaign people may also ask questions to specific candidates on the appropriate mailing list. The full schedule of the elections is available on the Elections schedule[4]. For more information about the elections process, see the program management docs[5]. [1] https://fedoraproject.org/wiki/Development/SteeringCommittee/Nominations [2] https://fedoraproject.org/wiki/Council/Nominations [3] https://fedoraproject.org/wiki/Mindshare/Nominations [4] https://fedorapeople.org/groups/schedule/f-31/f-31-elections.html [5] https://docs.fedoraproject.org/en-US/program_management/elections/ -- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis

1 2

Fedora 32 System-Wide Change proposal: On-demand Side Tags
by Ben Cotton 11 Nov '19

11 Nov '19

https://fedoraproject.org/wiki/Changes/OnDemandSideTags see also: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org… = On-demand Side Tags = == Summary == Allow on-demand side tags, and allow packagers to a) tag whatever rpms as build dependencies into those side tags, b) build packages they want in those side tags, and then c) tag all or some of those new packages back into appropriate “main” tag (e.g. fNN-updates-candidate). This new process will allow packagers to build rpms using older or newer versions of build dependencies than the buildroot in a given Fedora release, providing a subset of Modularity features in an additive way (no influence on the way dependent packages are built or installed). It may also coexist with Modularity in its current form. As a policy change, default modules are disallowed and modules are only opt-in. == Owners == * Name: [[User:Zbyszek| Zbigniew Jędrzejewski-Szmek]] * Email: zbyszek at in.waw.pl * Name: [[User:Churchyard| Miro Hrončok]] * Email: == Detailed Description == === Motivation === Modularity in its design and current implementation provides a number of features that can be divided into two: for people ''building rpms in modules'', and for people ''installing rpms from modules''. There have been a number of issues, RFEs, and discussions about modularity, but they are mostly about the second, user-facing part. Rpms installed from modules interact with non-modular rpms, and of course every aspect of this interaction is visible to users. To make Modularity a success, we would not only need to make this interaction well-behaved, we would also need to explain it to users in a satisfactory way. The first part, module build internals, is not visible to users. Some packagers don’t like the syntax, which is opaque and best-suited for programmatic generation and consumption. Constant rebuilding of modules also creates noticeable load on the infrastructure, making everything else slower. If modularity became much more popular, and a significant number of packagers was using it, we are convinced many discussions would start about this first part too. Despite some problems, there is clear demand for the features that Modularity brings. In particular, we need some solution for the "too fast too slow" problem, where some packages cannot update fast enough to maintain compatibility with the rest of the distribution. === New approach: on-demand side tags === The new process proposed here provides an alternative to both parts of Modularity. The second part of Modularity is simply removed: '''users only get normal rpms'''. After rpms have been built, some or all of them are merged into the normal tags and delivered to users as usual. This is pretty much the same as the usual rebuild of specific language or graphic stacks in side tags that we do already. Dependencies between rpms must be expressed using Requires/Recommends/Provides/Suggests/Conflicts. The first part is replaced by a simpler scheme: '''we don’t define any specific syntax or repositories to build packages'''. The packager has to issue a sequence of 'fedpkg tag-build', 'fedpkg build', and 'koji wait-repo' commands as appropriate (or 'fedpkg chain-build' or …). The packager ''may'' link an older rpm into the side tag, allowing building against old versions of packages. This means that if the packager is not a committer to the package, they may only use a version which was previously built in koji. The packager selects which rpms should be merged back into the main tag. Old packages obviously cannot be merged back. To minimize disruption for users and current packagers, modules are required to be opt-in only. There are no default modules and non-modular rpms cannot depend for build or installation on modular rpms. This way there is less burden to other packagers and Modularity can evolve more easily. === Implementation === '''Ability to create side tags “on demand”, i.e. with an automated API call''' must be implemented. Koji and other infra tools would need to be able to deal with a large number of tags. Modularity already creates a large number of tags, so this should not be a significant change. Packager tools like fedpkg must grow '''new commands to create side tags'''. Both the workflow in this proposal, and modules as they currently are would be allowed to coexist. After we have some experience with both approaches, one or both could be retired. Since this proposal removes the possibility to have default modules, '''modules need to be reset'''. The details are not specified yet. One approach would be to do the equivalent of dnf module reset all during upgrade to F32, similarly to the proposed solution for https://bugzilla.redhat.com/show_bug.cgi?id=1747408. This proposal assumes that '''fedpkg will be enhanced to do extended chain builds smartly'''. This is not a precondition: even without that packagers may script something that is appropriate for them. This is one of the main advantages of the proposal: we put the “smarts” in local tooling which can be enhanced and modified without breaking what other packagers do. E.g. a solution to rebuild all 500 rust packages with new versions may be significantly different than building a compat version of postgresql, and that is OK. === Buildroot only packages === They are possible, because not all packages are merged back. The same is currently possible with side tags, and with non-rawhide builds for which packagers may never create a bodhi update, so there is no fundamental change. Whether or not we need to disallow packages which are purposefully buildroot-only by a policy is left out of this proposal on purpose. Technically, it is possible. == Benefit to Fedora == === Costs, for packages converted from modules to this scheme === Transitional costs: according to this proposal, modules may still be used. Nevertheless, if the contents of a module are to be used to build normal rpms, that module will need to switch to the new process or simply demodularized and built in the traditional way. In the case where this happens, packaging would need to be “translated” from module .yaml to normal spec files. This should mostly be about adjusting metadata, i.e. package versions and dependencies. We would lose one of the main advertised features of Modularity: the ability to build the same module in different versions. This might not be a huge loss, for two reasons. First, great majority of modules are single-version. Second, if desired it is possible to provide compat packages with different names. This is a bit of annoying work, but we have the process very well understood. Compat packages have the additional advantage that they may be made co-installable (vide python26, python2, python35, python36, python3, python38 packages currently in F31). Compat packages are exempt from package review. We may consider additional simplifications to the process to make compat packages easier. === Benefits === Vast simplification for other packagers, i.e. those who would like to depend on rpms which are currently modularized, and for users, who do not have separate modular repos, just a single repo like before. Reduced load on the build infrastructure. Easier to find rpm sources because there again is one canonical repo. Third parties building on top of Fedora can use our modules to build their own rpms without trouble. dnf does not need to handle complicated logic. It should be faster if less repos are used. Rpms are built against an up-to-date buildroot, with only the minimal compat package inserted, instead of some older-than-all-current-releases Fedora release. Normal package-level package replacement and obsoletion mechanisms are used. On-demand side tags will remove the need to open releng tickets to have them manually created. == Scope == * Proposal owners: detailed documentation how to do things in the new scheme, client-side tooling improvements. * Other developers: demodularize specific packages if wanted or necessary. * Release engineering: implement the server side of side tags on demand [RELENG TICKET WILL BE FILED LATER] * Policies and guidelines: the docs would need to be added to packager documentation. Pull requests will be submitted by proposal owners. * Trademark approval: N/A (not needed for this Change) == Upgrade/compatibility impact == This change should be fully additive to normal rpm packaging, so there should be no upgrade/compatibility issues. Some modules will need to be demodularized. This will require packager work. Once this happens, rpms built this way will be fully available to build normal rpms and modules, so the upgrade impact should be limited to the modules being rewritten. == How To Test a.k.a. examples of the new workflow == === Version bump in a language stack === It is determined that a new set of rust rpm versions should be built. A member of the rust-sig bumps the versions in dist-git for all relevant packages, creates a side tag, and does the huge chain-build. Once finished, this side tag is merged into rawhide-build. If desired, it may also be merged into f31-updates-candidate and/or f30-updates-candidate and bodhi updates submitted. rust-sig already has tooling to generate the appropriate build-order for modularity’s .yaml file. The same code can be reused to generate the order for the chain build. === Three streams of a database package === A security patch needs to be applied to three versions of postgresql. There are postgresql, postgresql82, postgresql81 dist-git repos. The patch is added in dist-git for each of the packages, and they are rebuilt as appropriate. Since we want to build each package with different compiler versions, the packager uses a bash loop to schedule separate builds of each of the three packages in each of the supported Fedora versions. (Once we have automatic creation of bodhi updates from koji builds, the packager will not have to do anything more. Right now they still need to create one big update that will then be split into separate updates for each Fedora release.) A small advantage compared to current modules is that packages are not built against an already obsolete Fedora base, so users get the benefit of toolchain improvements. === Package needs some old dep to build === Poor packager needs gcc-9.0, at the time where gcc-9.1 is the default in the build root, to build koo. They bump the version of koo in dist-git, create a side tag, tag old gcc-9.0-12.fc31 build into the tag, build koo, merge the built package back into f32-build and f31-updates-candidate. === Package needs some old dep to install === koo-gui is only compatible with python3-qt5-5.12 when python3-qt5-5.13 has been released, and cannot be built with the new version. koo maintainers may either a) arrange for python-qt5_5_12 compat package to be created and then use it in the build root and as installation dep, b) install old python3-qt5-5.12 package in build root, and bundle some contents of that package into its output rpms, and add Provides:bundled(python3-qt5)=5.12. === FAQ for “other developers” === ==== I maintain several modular streams of my package additionally to the default version in Fedora. Can I keep doing this? ==== Yes, you can keep any alternate version in modular branches. When appropriate, consider compat packages that can often be co-installable, unlike modular streams. ==== I maintain my packages as modular only. Can I keep doing this? ==== Yes, as long as those modules are “leaves”, in the sense that no non-modular rpms depend on them. ==== I would like to use an rpm which is currently only available as module as a build dependency ==== The owner of the module should be asked to demodularize the module (and either convert it to fully traditional rpms, or build some rpms using the new side-tag process). Those rpms can then be used as build- and installation-time dependencies. If the owner of the module is not interested in providing non-modular rpms, other packages may add such rpms to the distribution (and again build them in one of the two ways). ==== Which rpms can be used as build dependencies in the side tag? ==== In principle, any rpm which is available in koji (including the recursive dependencies, so it can actually be installed). In addition, the packager may build rpms and use them in the side tag build root. In practice, we expect that packagers will use one of the Fedora releases (with updates), as the base. == User Experience == Users have a clearer picture where their rpms are coming from. If they opt-in into a module, they get some additional content. Without that, they only get rpms from the normal Fedora repository. They are able to rebuild rpms locally if there are no version conflicts. In case of version conflicts, they might need to install older versions of rpms from koji to satisfy the build (which might sound bad, but really isn’t much different from current state with normal rpms.) == Dependencies == On-demand side tags. Without this, this would be completely infeasible. For limited and initial testing, side tags could be created manually. == Contingency Plan == In the implementation phase, this should not impact other rpms in any way (since it is change really happens on the build farm). Once deployed, and rpms are built using this approach, they would still be installable, even if the new process is abandonded. So there is no flag day for revert, but some packages would effectively become FTBFS. Any rpms which can only be built in this new scheme (e.g. because they require older packages that the current ones in the default build root), would have to be either fixed to build in the normal build root, or would have to be modularized. If this approach is abandoned, other approaches including the proposal in https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot may be implemented, but that is only possible in the longer term. * Contingency mechanism: 1. disable side-tag creation in koji (releng). Hopefully this would be just a single switch somewhere. Any code that is added to koji can be removed later (koji developers). 2. undo any documentation changes 3. adjust packages as described above (change owners, package owners). This is very hard to say how big the scope of this would be, because it is unclear how many packages would be involed. I think we can assume that similarly as with Modularity, it would be <10 in the first Fedora release where this is available. * Contingency deadline: TBD * Blocks release? Yes == Documentation == Will be submitted by Change owners. -- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis

7 8

Some Java packages looking for new maintainer(s)
by Fabio Valentini 11 Nov '19

11 Nov '19

Hello packagers, The Stewardship SIG is currently providing only bare-minimum maintenance for some packages, and none of our other packages depend on these anymore. So, we're looking for someone to take better care of them, preferably someone who actively uses them or maintains a package that depends on them. - ed25519-java - hibernate-jpa-2.0-api - jackson - jandex - jboss-jsp-2.3-api - jboss-transaction-1.1-api For reference, these packages directly depend on the entries in the list above: ed25519-java: - eclipse-jgit - jgit hibernate-jpa-2.0-api: - openjpa jackson: - cassandra - openstack-java-sdk jandex: - hibernate - jandex-maven-plugin jboss-jsp-2.3-api: - jboss-jsf-2.2-api - jboss-jstl-1.2-api - struts jboss-transaction-1.1-api: - jboss-connector-1.6-api If you received this email directly, you're a (co-)maintainer of one of the packages listed above, and would probably be best qualified to take care of the relevant dependency. If you want to take any of these packages off our hands, fill out the "package_adoption_request" template here: https://www.pagure.io/stewardship-sig/new_issue If nobody claims the packages within the next two weeks, we will orphan them again, setting them on their course towards retirement in about two months. Thanks, Fabio (decathorpe), for the Stewardship SIG

1 1

Use immutable CRAN URLs
by Iñaki Ucar 06 Nov '19

06 Nov '19

Hi, Currently, most of the R packages included in Fedora use the following lines in the SPEC: URL: https://cran.r-project.org/web/packages/%{packname}/ Source0: https://cran.r-project.org/src/contrib/%{packname}_%{version}.tar.gz This URL format is not recommended by CRAN, but more importantly, the Source0 format does not work anymore, as [1] noted, when a new version is released. However, there is an immutable format available, as [2] pointed out. So my proposal is to use always the following lines instead: URL: https://cran.r-project.org/package=%{packname} Source0: %{url}&version=%{version} which are both shorter and immutable, and I propose to add this to the R packaging guidelines too. If we agree on this, is there any easy way to request a system-wide change like that to all existing packages? Regards, -- Iñaki Ucar [1] https://stat.ethz.ch/pipermail/r-devel/2018-October/076988.html [2] https://stat.ethz.ch/pipermail/r-devel/2018-October/076989.html

5 28

Fedora Modularity: What's the Problem?
by Stephen Gallagher 04 Nov '19

04 Nov '19

One of the recurring themes in the ongoing Modularity threads has been that we've made references to the problems we're trying to solve, but we haven't done a good job of gathering those requirements and use-cases into a single place. To resolve this, I've written a (fairly long) blog post describing the set of problems that we are trying to solve. You can read it with nice formatting over at https://communityblog.fedoraproject.org/fedora-modularity-whats-the-problem/ or the mediocre copy-and-paste I'm including in this message. (Apologies for the HTML mail; I want to preserve the hyperlinks and formatting from the blog post) --- Fedora Modularity: What’s the Problem? <https://communityblog.fedoraproject.org/fedora-modularity-whats-the-problem/> Much has been said <https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org…> about Fedora Modularity over the past couple weeks. Much of it has been constructive; some of it the expected resistance to change that all large features encounter. Some, however, is the result of our not having painted a good picture of the problems that Modularity aims to solve. Numerous suggestions have been made on the Fedora Development mailing list that sound good on the surface but that ultimately fail to address some important use-cases. This blog post will attempt to enumerate these cases in detail so as to serve as a common reference point for the ongoing discussions. Please note as well that these are goals. There are numerous places where the implementation of Modularity at the time of this writing is not yet fully adherent to them. It’s all about the apps! Though many of the readers of this blog might be of a different mind, it’s important to remember that very few people install a Linux distribution for its own sake. Ultimately, the goal is to “scratch a particular itch” that the user is experiencing. The solutions may take many forms, but ultimately this user wants to deploy some software that solves a problem for them. This leads us to a classic problem that Linux distributions have faced: the “Too Fast/Too Slow” problem. Linux distributions are traditionally quite monolithic. The package collections they ship are generally self-consistent, providing generally whatever the latest stable major release of the software at the time of the distribution release. As the release ages, it will receive bugfixes and enhancements, but usually will remain on the same major version. This is excellent for the maintainers of the distribution, because it allows them to test that everything works together as a cohesive whole. It means that there’s one authoritative version to align to. Users, on the other hand, are most concerned about solving their problem. It matters less to them that the distribution is cohesive and more that the tools they need are available to them. The “Too Fast/Too Slow” problem is basically this: users want a solid, stable, reliable, *unchanging* system. They want it to stay that way for the life of their application. However, they also want their application to run using the set of dependencies it was designed for. If that doesn’t happen to be the same version (newer or older) as the one selected for the monolithic distribution, the user will now have to resort to alternative means to get up and running. This may be as simple as bundling a dependency or as drastic as selecting an entirely different distribution that better fits their specific need. A little background One of the precursors to Fedora Modularity was Software Collections <https://www.softwarecollections.org/> (SCLs). This was a first try at solving the Too Fast/Too Slow” problem in the Fedora/Red Hat ecosystem. provides two basic advantages: *Parallel Availability* and *Parallel Installability*. *Parallel Availability* means that more than one major release of a popular software project is available for installation. For example, the “Developer Toolset” SCLs provide access to newer versions of GCC and its related toolchain for building software. There are Python and Ruby SCLs that provide assorted runtimes for those languages and so on. *Parallel Installability* means that more than one major release of a software project can be installed on the same userspace. A few years back, the Product Management team inside Red Hat performed a large-scale survey of customers and potential customers about the user experience of Red Hat Enterprise Linux. In particular, they asked about their level of satisfaction with the software available from the enterprise distribution and their opinion on these Software Collections. Perhaps unsurprisingly, the overwhelming majority of respondents were thrilled to have supported versions of software beyond what had shipped with the base operating system. What the survey team did come away with that was an epiphany was that the respondents generally did not care about the parallel installability of the SCLs. For the most part, they maintained individual userspaces (using bare metal, traditional virtualization or containers) for each of the applications they cared about. The most common problem reported for Software Collections was that using them required changes to the applications they wanted to run. SCLs install to a separate filesystem location from more traditional RPMs and applications that rely on them need to know where to look for them. (In SCL parlance, this is called “activating” the collection.) The consequence of this relocation on disk is that users were unable to take existing applications (either FOSS or proprietary) and simply use them. Instead, they had to modify the projects to first activate the collections. This was a consistent pain point. Given this feedback, Red Hat came to the conclusion that parallel installability, while nice to have, was not a critical user requirement. Instead, the focus would be on the parallel *availability*. By dropping this requirement, it became possible to create a solution that allowed the different versions to be swapped in and take over the standard locations on the disk. Meanwhile in Fedora Of course, it’s not just Red Hat — people in Fedora are also concerned with solving this Too Fast / Too Slow problem for our users. Efforts around this kicked off in seriousness with the Fedora.next initiative <https://fedoramagazine.org/fedora-present-and-future-a-fedora-next-2014-upd…> and Fedora Project Leader Matthew Miller’s “Rings <https://lwn.net/Articles/563395/>” talk at the first Flock conference in 2013. This led to the proposal and approval by the Fedora Council of the Modularity Prototype Fedora Objective <https://fedoraproject.org/wiki/Objectives/Fedora_Modularization,_Prototype_…> and its follow-up Modularity Release Fedora Objective <https://fedoraproject.org/wiki/Objectives/Fedora_Modularization_%E2%80%94_T…> . Critical use cases for consumers First and foremost, our primary driving goal is to make it easy for our users to understand and interact with alternative software versions. In any instance where choosing between the packager experience and the user experience is in conflict, we elect to improve things for the user. Standard Locations In order to make deployment of users’ applications simpler, we need to make sure that software can be installed into the common, expected locations on the system. This includes (but is not limited to): - Libraries must be installed to /usr/lib[64]. - Headers must be installed to /usr/include. - Executables must be installed to a location in the default system $PATH - Other -devel functionality such as pkgconfig files must be installed in their standard lookup locations. - Installed services may own a well-known DBUS address. - Services may own the appropriate standard TCP/UDP ports or local socket paths. *Requirement*: Installation must occur in the same locations as traditional RPM software delivery. Don’t break the app! It is very common for Fedora to update to the latest major version of packages at each new semiannual release. This ensures that Fedora remains at the leading edge of software development, but it can wreak havoc on anyone trying to maintain a deployment on Fedora. If they are running an app that is built for PostgreSQL 9.6 and Fedora switches to carrying PostgreSQL 10 in the next major release, upgrading to that release may break their app (possibly in ways undetectable by the upgrade process). However, staying on an old version of Fedora forever has its own problems. Not least of these is the problem of security updates: Once a release has been out for about 13 months, it stops receiving errata. Moreover, new releases of the Fedora platform may have other useful enhancements (better security defaults, increased performance thanks to compiler improvements, etc.). *Requirement*: We need to allow users to “lock” themselves onto certain dependencies as long as the packager is maintaining them. These dependencies must continue to receive updates. *Requirement*: There must be appropriate and helpful UX for dealing with when those dependencies go EOL. Support the developers Developers often want to build their applications using the latest-and-greatest version of their dependencies. However, that may not have been released until after the most recent Fedora release. In non-Modular Fedora, that means waiting up to six months to be able to work on it there. *Requirement*: It must be possible to gain access to newer software than was available at the Fedora release GA. Additionally, Dev/Ops people are rapidly switching to a new paradigm of development and deployment (containers) to solve the above issue. However, most containers today are retrieved from public repositories. The public repositories are generally user-managed and have not been verified and validated for security. *Requirement*: Provide a mechanism for building *trusted* container base and application images with content alternatives. Keep it updated It’s not enough that other versions of software are available to install. They also need to be kept up to date with bug fixes and security updates. In non-Modular Fedora, users had the ability to force DNF to lock to a specific RPM NEVRA, but they wouldn’t get updates from it. *Requirement*: Alternative software must receive be able to recieve and apply updates. Make it discoverable Having alternative versions available is important but not sufficient. It is also necessary for users to be able to locate these alternatives. Some of our early explorations into this area failed this ease-of-use test because they require the user to have knowledge of external sites and then to search those sites for what they think they want. *Requirement*: Users must be able to discover what alternative software versions are available with tools that are shipped with the OS by default. Ideally, these should be the same tools that they are already comfortable with. Don’t break existing package management workflows Users are slow to adapt to changes in the way they need to behave. Requiring them to learn a new set of commands to interact with their system will likely result in frustration and possibly exodus to other distributions. *Requirement*: It must remain possible to continue to operate with only the package management commands used in traditional Fedora. We may provide additional commands to support new functionality, but we must not break the previous workflow. *Requirement*: Existing automation tools such as anaconda’s kickstart and Ansible must continue to work. Critical use-cases for packagersDependencies Because very little software today is wholly self-contained, it must be possible for Modules to depend on each other. *Requirement*: There must be a mechanism for packagers to explicitly list dependencies on other software, including alternative versions. This mechanism must support both build-time and run-time dependencies. Alternative dependencies Some software is very restrictive about which dependencies it can work with. Other software may work with several different major releases of a dependency. For example, a user may ship two Ruby-based web applications, one which is capable of running on Ruby 2.5 and the other that can run on either Ruby 2.5 or Ruby 2.6. In non-modular Fedora, only one version of Ruby would be available. If the system version was 2.5, then both applications could run fine. But if in the next release of Fedora the Ruby 2.6 release becomes the system copy, one of those applications will have to be dropped (or patched) to work with it. *Requirement*: It must be possible to build software that can be run against multiple versions of its dependencies. *Requirement*: The packaging process for creating software that supports multiple versions of their dependencies must not be significantly more difficult than packaging for a single dependency. As more and more things become modules, there is concern that such things will grow into an unbounded matrix. For this, we need to establish policies on when the use of alternative dependencies is preferable or when it is better to constrain it to a single version or small set. *Requirement*: Packaging guidelines need to provide advice on when to use multiple alternative dependencies or to select a single one. Managing private dependencies When a person decides that they want Fedora to carry a particular package and decides to do the work to accomplish this, it is not uncommon to discover that the package they care about has additional dependencies that are not yet packaged in Fedora. Traditionally, this has meant that the packager has needed to package up those dependencies and then continue to maintain them for anyone who may be using them for other purposes. This can sometimes be a significant investment in time and energy, all to support a package they don’t necessarily care about except for how it supports the primary package. Build-time Dependencies Sometimes, a package is needed only to build the software and is not required at run-time. In such cases, Modularity should offer the ability to keep those build-time dependencies entirely private and not exposed to the Fedora Package Collection at large. *Requirement*: Build-time only dependencies for an alternative version may be excluded from the installable output artifacts. These excluded artifacts may be preserved by the build-system for other purposes. *Requirement*: All sources used for generating alternative versions, regardless of final visibility, must be available to the community for purposes of modification and reproducibility. Defining the public API Similarly, there are times when an application the packager cares about depends on another package that is required at runtime, but sufficiently complex that the packager would not want to maintain it for general use. (For example, an application that links to a complicated library but only uses a few functions.) In this case, we want there to be a standard mechanism for the packager to be able to indicate that some of the output artifacts are not supported for use outside this module. If they are needed by others, they should package it themselves and/or help maintain it in a shared place. *Requirement*: Packagers must be able to encode whether their output artifacts are intended for use by other projects or if they are effectively private to the alternative version. Packagers must also have a way of finding this information out so they understand what they can and cannot rely on as a dependency. Use-case-based installation Since the earliest days of Linux, the “package” has been the fundamental unit of installable software. If you want to have some functionality on the system, you need to learn the name of the individual packages that provide that functionality (not all of which are named obviously). As we build modules, one of the goals is to try to focus installation around use-cases rather than around upstream projects. A big piece of this is that we want to have a way to install a subset of a module that supports specific use-cases. A common example being “server” and “client” cases. *Requirement*: It must be possible to install a subset of artifacts from an alternative version. These installation groups should be easily discoverable. *Recommendation*: Installation groups should be named based on the use-case they are intended to solve. This will provide a better user experience. Lifecycle isolation Another of the major issues faced by Fedora is maintaining a release schedule when all of the components within it follow vastly differing schedules. There are two main aspects to this problem: - A major version of a popular piece of software is released just after a Fedora release, so it doesn’t land in Fedora for six months. - Some software does frequent major revisions (Django, Node.js, etc.) and swapping them out every six months for the latest one means that dependent projects are constantly needing to adapt to the new breakage or find alternative mechanisms for retaining the older, working version - Some software does not handle multiple-version upgrades (Nextcloud, for example). Attempting to go from version 15 to verison 19 requires first upgrading through 16, 17, and 18. *Requirement*: It must be possible for new alternative versions of software to become available to the Fedora Package Collection between release dates. *Requirement*: It must be possible for alternative versions of software to go end-of-life during a Fedora release. This does not mean that the software must disappear from the repositories, merely that an assertion exists somewhere that after a certain date, the package will not receive updates. *Requirement*: For alternative versions whose lifecycle will continue through at least part of the next Fedora release, it must be possible to upgrade from one release to the next and remain with the fully-compatible version. Third-party additions Some third-party add-on repositories (particularly EPEL) have been limited in the past by relying on the system copies of packages in the base distribution of the release. In the particular case of EPEL, little can be done to upgrade these system copies. In order to be able to package much of the available FOSS software out there, it may be necessary to override some of the content shipped in the base system with packages known to work properly. *Requirement*: It must be possible for third party repositories to create alternative versions that override base distribution content at the user’s explicit choice. *Requirement*: It must be possible for third party repositories to create alternative versions of software that exist in the base distribution. Reduce duplication in packaging work There is plenty of software out in the wild that maintains compatibility over time and is therefore useful to carry in multiple releases of Fedora. With traditional packaging, this means carrying and building separate branches of the packages for each release of Fedora. In the case of software “stacks” which are tightly bound, this means also manually building each of its dependencies in each release of Fedora. *Requirement*: It must be possible to build multiple component software packages in the same build process. *Requirement*: It must be possible for the packager to specify the order in which packages must be built (and to indicate which ones can be built in parallel). *Requirement*: It must be possible to be build for all supported platforms using the same specification and with a single build command. Non-GoalsParallel installability As mentioned in the Background section, the goals of Modularity are specifically to *not* implement parallel-installability. We recommend instead that users should rely on other mechanisms such as virtualization or containerization to accomplish this. If parallel-installation is unavoidable, then Modularity is not the correct tool for this job. Arbitrary stream switching Module streams are intended to be compatible update streams. That means they must follow the same rules regarding RPM package-level updates within the stream. By definition, two streams of the same module exist because upgrades (or downgrades or cross-grades…) are not capable of being done in a safe, automated fashion. That does not mean that stream switching should be impossible, but it does mean that we will not build any tools intended to handle such switching in a generic manner. Stream switches should be handled on a module-by-module basis and detailed instructions and/or tools written for each such case.

7 15

Official font
by Iñaki Ucar 03 Nov '19

03 Nov '19

Hi all, I incidentally discovered today that, since quite recently, there's a Red Hat font [1]. And this led me to think about the popularity of the Ubuntu font, you know, and how nice would be to have a nice catchy official Fedora font integrated into the distro... I'm just thinking aloud, because I don't know anything about font design. But maybe someone picks up the gauntlet... ;-) [1] https://github.com/RedHatOfficial/RedHatFont Cheers, Iñaki

5 6

← Newer
1
2
3
4
5
6
7
...
35
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

devel October 2019