This is a writeup on a problem we’re facing with modularity, and some ideas on
how to resolve it.
# The "Problem"
Imagine I have an **httpd module**. To simplify things, let’s say that this
module has only one stream: **2.4**. Today, in the modulemd for this module, I
declare build and runtime dependencies like this:
dependencies:
buildrequires:
platform: f26
requires:
platform: f26
This works.
Now, imagine that Fedora 27 comes along. I want the httpd module to be
installable on f27, so I update those streams to point to f27. But now I can’t
ship updates to f26 anymore! Uh oh.
## Just use more streams?
We could just ask packagers to create more streams to deal with this. httpd
wouldn’t have a 2.4 stream. It would need to have a f26-2.4 stream and a
f27-2.4 stream.
But, this gets us exactly back into the place we were **before** we had
arbitrary branching! You need to have as many branches for *every component*
as you have releases of the distro (or, as you have "products"). Noooo!
## Solution: "Input" Modulemd Syntax Changes
We’re going to extend the modulemd syntax to allow specifying multiple
dependencies in an "input" modulemd (the one that packagers modify). When
submitted to the build system, the module-build-service (MBS) will *expand*
these values under the hood, and submit **multiple** module builds to
itself based on the expansion.
Here are some examples of modulemd files (maintained by humans) that might be
submitted to the MBS.
Build on **all** active streams of the platform module.
dependencies:
buildrequires:
platform: []
Build on **only** the f27 and f26 platform streams.
dependencies:
buildrequires:
platform: [ f27, f26 ]
Build on all active streams of platform **except** for the f26 stream.
dependencies:
buildrequires:
platform: [ -f26 ]
The following syntax, which builds on **only** the f26 stream, will still be supported:
dependencies:
buildrequires:
platform: f26
--
Take the following example (described above):
dependencies:
buildrequires:
platform: [ f27, f26 ]
Submitting this modulemd to the MBS would in turn generate **two** module
builds: one of our httpd module against the f27 stream and another against the
f26 stream. Each module build would be associated with its own unique
"flattened" *output* modulemd file that specifies exactly which platform stream
it was built against.
# New Problems
Having **multiple** module builds for a single dist-git commit of a modulemd
file poses new implementation problems.
## NSV uniqueness
Today, we uniquely identify a module build in *a variety of systems* with
**<name>-<stream>-<version>** (NSV) where version is derived from the dist-git
commit timestamp. Here we’ll have an httpd-2.4 module built on f26 and an
http-2.4 module built on f27: two different module builds with the same name,
stream, and version. How can we tell them apart?
We will introduce a new value called the **context** of a built module and
include it in the modulemd metadata that gets carried with the built module.
* For user facing cases (dnf installation) it will generally be hidden. The
old NSV value will still appear. If a user ever needs to surface the value,
client tooling can find it in the modulemd metadata. The additional
identifier will give users access to explicit pointers to the content that is
installed:
* For cloning a machine.
* For comparing two installed hosts.
* For reporting bugs.
* Some build and compose time systems will have to be modified to use the
context as part of a new unique identifier. **NSVC** will be the
**<name>-<stream>-<version>-<context>**. The systems in question are PDC,
koji/brew, pungi, and bodhi.
The context value will be a **hash**, generating as the first step in the build
process (but after expansion). Consider what metadata needs to be hashed: we
think that hashing the whole modulemd is problematic, because the modulemd can
and will be modified after build time.
Therefore, the *context_hash* value will need to be derived from only the stuff
that uniquely identifies the build and runtime context of the module -- name,
stream, version and crucially, its dependencies
## Buildtime/Runtime Dep Correlation
Another problem. We now have input modulemd files for a single stream that can
expand buildtime dependencies to ‘f27’ and ‘f26’, but what about the
runtime dependencies?
Consider the following yaml file:
dependencies:
buildrequires:
platform: [f28, f27, f26]
requires:
platform: [f28, f27, f26]
With our thinking caps on, this should obviously generate three module builds
(one that buildrequires f28 and run requires f28, a second that buildrequires
f27 and run requires f27, and a third for f26). The naive cross-product of all
streams is not valuable; the MBS needs some logic to **correlate** the build
time requirements with the run-time requirements. That logic will contain
assumptions about how this will be used, and it needs to be well-defined.
Let’s try to define that. Consider a more complicated yaml file that depends
on both multiple streams of platform and on multiple hypothetical streams of a
"shared-userspace" module:
dependencies:
buildrequires:
platform: [f28, f27, f26]
shared-userspace: [fancy, nonfancy]
requires:
platform: [f28, f27, f26]
shared-userspace: [fancy, nonfancy]
Here are some rules the MBS should obey:
* If a build-time dep contains an expansion, that dep **does not** have to also
appear as a run-time dep.
* If a build-time dep contains an expansion, and if that dep *also appears* as
a runtime dep, the runtime expansion **must** match the buildtime expansion
exactly.
Now that the streams from build-time and runtime *can be mapped one-to-one*,
the cross-product of the deps is taken to produce a set of modules builds. In
our example here, we get:
1. One build with platform f28 and shared-userspace fancy.
2. One build with platform f27 and shared-userspace fancy.
3. One build with platform f26 and shared-userspace fancy.
4. One build with platform f28 and shared-userspace nonfancy.
5. One build with platform f27 and shared-userspace nonfancy.
6. One build with platform f26 and shared-userspace nonfancy.
We like this.
--
But, a new** corner case** emerges! What if the *nonfancy* branch of
shared-userspace is only compatible with f27 and f28, but not with f26?
Build #6 will always fail - or, if it doesn’t fail it will be unusable.
We thought through a number of ways that corner cases like these could be
handled automatically, and are still working through them. At the moment, we
are *leaning towards* suggesting that a failing corner **requires** a new
stream of the module in question to handle the exclusion. I.e., no convenient
syntax to say "all combinations except for this one".
In this case, you would have two streams of the httpd-2.4 module:
httpd-2.4 and httpd-2.4_f26. The first modulemd file in the
2.4 stream would be nice, like this:
dependencies:
buildrequires:
platform: [f28, f27]
shared-userspace: [fancy, nonfancy]
requires:
platform: [f28, f27]
shared-userspace: [fancy, nonfancy]
The second in the 2.4_f26 stream would exist only to support the desired
corner case:
dependencies:
buildrequires:
platform: [f26]
shared-userspace: [fancy]
requires:
platform: [f26]
shared-userspace: [fancy]
--
We’re still working through more details, especially on this last part (and how
it may affect upgrades), but figured its best to share now. Any help
appreciated!
Yours-
-Ralph
