EPIC Planning Document
__________________________________________________________________
History / Background
Since 2007, Fedora Extra Packages for Enterprise Linux (EPEL) has
been rebuilding Fedora Project Linux packages for Red Hat
Enterprise Linux and its clones. Originally the goal was to compile
packages that RHEL did not ship but were useful in the running of
Fedora Infrastructure and other sites. Packages would be forked
from the nearest Fedora release (Fedora 3 for EPEL-4, Fedora 6 for
EPEL-5) with little updating or moving of packages in order to give
similar lifetimes as the EL packages. Emphasis was made on
backporting fixes versus upgrading, and also not making large
feature changes which would cause confusion. If a package could not
longer be supported, it would be removed from the repository to
eliminate security concerns. At the time RHEL lifetimes were
thought to be only 5-6 years so backporting did not look like a
large problem.
As RHEL and its clones became more popular, Red Hat began to extend
the lifetime of the Enterprise Linux releases from 6 years to 10
years of "active" support. This made trying to backport fixes
harder and many packages in EPEL would be "aged" out and
removed. This in turn caused problems for consumers who had tied
kickstarts and other scripts to having access to those
packages. Attempts to fix this by pushing for release upgrade
policies have run into resistance from packagers who find focusing
on the main Fedora releases a full time job already and only build
EPEL packages as one-offs. Other attempts to update policies have
run into needing major updates and changes to build tools and
scripting but no time to do so. Finally, because EPEL has not
majorly changed in 10 years, conversations about changing fall into
"well EPEL has always done it like this" from consumers, packagers,
and engineering at different places.
In order to get around many of these resistance points with
changing EPEL, I suggest that we frame the problems around a new
project called Extra Packages for Inter Communities. The goal of
this project would be to build packages from FedoraProject Linux
releases to various Enterprise Linux whether they are Red Hat
Enterprise Linux, CentOS, Scientific Linux or Oracle Enterprise
Linux.
__________________________________________________________________
Problems and Proposals
Composer Limitations:
Problem
Currently EPEL uses the Fedora build system to compose a release
of packages every couple of days. Because each day creates a new
compose, the only channels are the various architectures and a
testing where future packages can be tested. Updates are not in
a seperate because EPEL does not track releases.
EPEL packagers currently have to support a package for the 10 year
lifetime of an RHEL release. If they have to update a release, all
older versions are no longer available. If they no longer want to
support a package it is completely removed. While this sounds like
it increases security of consumers, Fedora does not remove old
packages from older releases.
Proposed Solution
EPIC will match the Enterprise Linux major/minor numbers for
releases. This means that a set of packages will be built
for say EL5 subrelease 11 (aka 5.11). Those packages would
populate for each supported architecture a release, updates
and updates-testing directory. This will allow for a set of
packages to be composed when the subrelease occurs and then
stay until the release is ended.
/pub/epic/releases/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
/pub/epic/updates/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
/pub/epic/updates/testing/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
Once a minor release is done, the old tree will be hard linked to
an appropriate archive directory.
/pub/archives/epic/releases/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
/pub/archives/epic/updates/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
/pub/archives/epic/updates/testing/5/5.11/{x86_64,source,i386,aarch64,arm,ppc6
4}/
A new one will be built and placed in appropriate sub
directories. Hard links to the latest will point to the new one,
and after some time the oldtree will be removed from the active
directory tree.
Channel Limitations:
Problem
EPEL is built against a subset of channels that Red Hat
Enterprise Linux has for customers, namely the Server, High
Availability, Optional, and some sort of Extras. Effort is
made to make sure that EPEL does not replace with newer
packages anything in those channels. However this does not
extend to packages which are in the Workstation, Desktop,
and similar channels. This can cause problems where EPEL’s
packages replace something in those channels.
Proposed Solution
EPIC will be built against the latest released CentOS minor
release using the channels which are enabled by default in
the CentOS-Base.repo. These packages are built from source
code that Red Hat delivers via a git mechanism to the CentOS
project in order to rebuild them for mass
consumption. Packages will not be allowed to replace/update
according to the standard RPM Name-Epoch-Version-Release
(NEVR) mechanism. This will allow EPIC to actually service
more clients
Build System Limitations
Problem
EPEL is built against Red Hat Enterprise Linux. Because
these packages are not meant for general consumption, the
Fedora Buildsystem does not import them but builds them
similarly to a hidden buildroot. This causes multiple
problems:
+ If EPEL has a package with the same name, it supercedes the
RHEL one even if the NEVR is newer. This means old packages
may get built against and constant pruning needs to be done.
+ If the EPEL package has a newer NEVR, it will replace the RHEL
one which may not be what the consumer intended. This may
break other software requirements.
+ Because parts of the build are hidden the package build may
not be as auditable as some consumers would like.
Proposed Solution
EPIC will import into the build system the CentOS build it
is building against. With this the build is not hidden from
view. It also makes it easier to put in rules that an EPIC
package will never replace/remove a core build
package. Audits of how a build is done can be clearly shown.
Greater Frequency Rebasing
Problem
Red Hat Enterprise Linux have been split between competing
customer needs. Customers wish to have some packages stay
steady for 10 years with only some updates to them, but they
have also found that they need rapidly updated software. In
order to bridge this, recent RHEL releases have rebased many
software packages during a minor release. This has caused
problems because EPEL packages were built against older
software ABI’s which no longer work with the latest
RHEL. This requires the EPEL software to be rebased and
rebuilt regularly. Conversely, because of how the Fedora
build system sees Red Hat Enterprise Linux packages, it only
knows about the latest packages. In the 2-4 weeks between
various community rebuilds getting their minor release
packages built, EPEL packages may be built against API’s
which are not available.
Proposed Solution
The main EPIC releases will be built against specific CentOS
releases versus the Continual Release (CR) channel. When the
next RHEL minor is announced, the EPIC releng will create
new git branch from the current minor version (aka 5.10 →
5.11). Packagers can then make major updates to versions or
other needs done. When the CentOS CR is populated with the
new rpms, packages will be built in the new tree using those
packages. After 2 weeks, the EPIC minor release will be
frozen and any new packages or fixes will occur in the
updates tree.
__________________________________________________________________
Guidelines
Packaging
EL-4
This release is no longer supported by CentOS and will not be supported
by EPIC.
EL-5
This release is no longer supported by CentOS and will not be supported
by EPIC.
EL-6
This release is supported until Nov 30 2020 (2020-11-30). The base
packaging rules for any package would be those used by the Fedora
Project during its 12 and 13 releases. Where possible, EPIC will make
macros to keep packaing more in line with current packaging rules.
EL-7
This release is supported until Jun 30 2024 (2024-06-30). The base
packaging rules for any package would be those used by the Fedora
Project during its 18 and 19 releases. Because EL7 has seen major
updates in certain core software, newer packaging rules from newer
releases is possible to follow.
EL-next
Red Hat has not publicly announced what its next release will be, when
it will be released, or what its lifetime is. When that occurs, it will
be clearer which Fedora release packaging will be based off of.
GIT structure
Currently EPEL uses only 1 branch for every major RHEL release. In
order to better match how current RHEL releases contain major
differences, EPIC will have a branch for every major.minor release.
This is to allow for people who need older versions for their usage to
better snapshot and build their own software off of it.
There are several naming patterns which need to be researched:
/<package_name>/epic/6/10/
/<package_name>/epic/6/11/
/<package_name>/epic/7/6/
/<package_name>/epic/7/7/
/<package_name>/epic-6/6.10/
/<package_name>/epic-6/6.11/
/<package_name>/epic-7/7.6/
/<package_name>/epic-7/7.7/
/<package_name>/epic-6.10/
/<package_name>/epic-6.11/
/<package_name>/epic-7.6/
/<package_name>/epic-7.7/
Git module patterns will need to match what upstream delivers for any
future EL.
Continous Integration (CI) Gating
EPIC-6
The EL-6 lifecycle is reaching its final sub releases with more
focus and growth in EL-7 and the future. Because of this gating
will be turned off EPIC-6. Testing of packages can be done at the
packagers discretion but is not required.
EPIC-7
The EL-7 lifecycle is midstream with 1-2 more minor releases with
major API changes. Due to this, it makes sense to research if
gating can be put in place for the next minor release. If the time
and energy to retrofit tools to the older EL are possible then it
can be turned on.
EPIC-next
Because gating is built into current Fedora releases, there should
be no problem with turning it on for a future release. Packages
which do not pass testing will be blocked just as they will be in
Fedora 29+ releases.
Modules
EPIC-6
Because EL-6’s tooling is locked at this point, it does not make
sense to investigate modules.
EPIC-7
Currently EL-7 does not support Fedora modules and would require
updates to yum, rpm and other tools in order to do so. If these
show up in some form in a future minor release, then trees for
modules can be created and builds done.
EPIC-next
The tooling for modules can match how Fedora approaches it. This
means that rules for module inclusion will be similar to package
inclusion. EPIC-next modules must not replace/conflict with CentOS
modules. They may use their own namespace to offer newer versions
than what is offered and those modules may be removed in the next
minor release if CentOS offers them then.
Build/Update Policy
Major Release
In the past, Red Hat has released a public beta before it finalizes
its next major version. If possible, the rebuilders have come out
with their versions of this release in order to learn what gotcha’s
they will have when the .0 release occurs. Once the packages for
the beta are built, EPIC will make a public call for packages to be
released to it. Because packagers may not want to support a beta or
they know that there will be other problems, these packages will
NOT be auto branched from Fedora.
Minor Release
The current method CentOS uses to build a minor release is to begin
rebuilding packages, patching problems and then when ready put
those packages in their /cr/ directory. These are then tested for
by people while updates are built and ISOs for the final minor
release is done.
The steps for EPIC release engineering will be the following:
1. Branch all current packages from X.Y to X.Y+1
2. Make any bugzilla updates needed
3. Rebuild all branched packages against CR
4. File FTBFS against any packages.
5. Packagers will announce major updates to mailing list
6. Packagers will build updates against CR.
7. 2 weeks in, releng will cull any packages which are still FTBFS
8. 2 weeks in, releng will compose and lock the X.Y+1 release
9. symlinks will point to the new minor release.
10. 4 weeks in, releng will finish archiving off the X.Y release
Between Releases
Updates and new packages between releases will be pushed to the
appropriate /updates/X.Y/ tree. Packagers will be encouraged to
only make minor non-api breaking updates during this time. Major
changes are possible, but need to follow this work flow:
1. Announce to the epel list that a change is required and why
2. Open a ticket to EPIC steering committee on this change
3. EPIC steering committee approves/disapproves change
4. If approved change happens but packages are in updates
5. If not approved it can be done next minor release.
Build System
Build in Fedora
Currently EPEL is built in Fedora using the Fedora Build system
which integrates koji, bodhi, greenwave, other tools together. This
could be still used with EPIC.
Build in CentOS
EPIC could be built in the CentOS BuildSystem (CBS) which also uses
koji and has some integration to the CentOS Jenkins CI system.
Build in Cloud
Instead of using existing infrastructure, EPIC is built with newly
stood up builders in Amazon or similar cloud environments. The
reasoning behind this would be to see if other build systems can
transition there eventually.
__________________________________________________________________
Definitions
Blue Sky Project
A project with a different name to help eliminate preconceptions
with the existing project.
Customer
A person who pays for a service either in money, time or goods.
Consumer
Sometimes called a user. A person who is consuming the service
without work put into it.
EPEL
Extra Packages for Enterprise Linux. A product name which was to
be replaced years ago, but no one came up with a better one.
EPIC
Extra Packages Inter Community.
RHEL
Red Hat Enterprise Linux
__________________________________________________________________
Last updated 2018-05-15 14:58:37 EDT
--
Stephen J Smoogen.
