New subject: Proposal to enable spec file preprocessing step before srpm build

8 Mar 2020


      Hello Fedora people!
I would like to offer a solution for dynamic changelog and release
generation and potentially more.
Intro
------
The solution employs a new type of macros, which can be placed into an
rpm spec file and afterward parsed by preproc utility
(https://pagure.io/rpkg-util/blob/master/f/preproc/preproc) which
executes the macros (by using bash interpreter) and replaces the
occurrences of the macros in the original file with standard output of
those macros.
The main use of those macros is intended to be reading git metadata
and using it to automatically generate rpm fields like Name, Release,
or %changelog but the macros are, in the end, a turing-complete tool
so they can be potentially used to do more than that (if desired).
The tags for those macros are {{{ and }}} and a header of an example
spec file could then look like this:
Name:    {{{ git_dir_name }}}
Version: 1.18
Summary: Remove old packages from rpm-md repository
Release: {{{ git_dir_release_branched }}}
The `git_dir_name` and `git_dir_release_branched` keywords are bash
functions that already implement some logic for automatic generation
of package name (from git repository name) and release (from commit
and tag history), respectively. There is also `git_dir_changelog`
macro to generate a changelog from content of annotated tags.
This proposal, however, is not about usage of those particular macro
functions (this can be discussed later as well as the usage of
annotated tags in particular) but only about usage of the new macro
class {{{ ... }}} recognized by the preproc utility.
The reason why we need this new macro class and we couldn't rely just
on shell expansion already offered by rpm is that %(…) (the shell
expansion rpm tag) gets carried over verbatim into the spec file which
is stored inside a built source rpm. If the code in such tag relied on
git metadata it would work only when the srpm is originally built from
dist-git sources but later when the source rpm gets rebuilt (e.g.
internally in mock), the code would fail because at that point, the
git metadata is already missing (they are not included in srpms).
Therefore we need a new type of macros in a spec file that, when
evaluated, are replaced with their stdout before the spec file is used
to build an srpm.
Implementation
---------------------
I have some packages in reviews and ready for a review to provide the
building stones. These are:
https://bugzilla.redhat.com/show_bug.cgi?id=1809644 (preproc - the preprocessor)
https://bugzilla.redhat.com/show_bug.cgi?id=1811432 (rpkg-macros -
basic set of macros that can be used by packagers)
 https://bugzilla.redhat.com/show_bug.cgi?id=1811432 (preproc-rpmspec
- simple tool that makes it easier to invoke preproc on a spec file
with the rpkg-macros included)
These packages are functional from fedora-rawhide down to epel-6.
Only preproc is essential for this proposal but I would like to, at
the same time, also provide some basic starting set of macros that
packagers can easily try and comment on. Either we can stick with
those macros, tweak them, or provide an entirely different set of
macros that uses a different logic.
Next, I was looking into koji code and found out that no changes
should be actually needed there to support this. The changes, in my
opinion, should be placed into mock here, just before the start of
srpm build: https://github.com/rpm-software-management/mock/blob/5c08f5b74/mock/py/mockb...
together with adding three new configuration options:
config_opts['rpkg_preprocessing'] = True   # by default False
config_opts['rpkg_preprocessing_requires'] = ["preproc-rpmspec"]
config_opts['rpkg_preprocessing_cmd'] = '/usr/bin/preproc-rpmspec
%(source_spec)s --output %(target_spec)s'
The first one `rpkg_preprocessing` tells us if the feature should be
enabled or not (by default, it will be probably False in mock). The
second one tells us what package should be installed into the chroot
before the start of srpm build to provide the preprocessing
functionality. And the third one is a concrete invocation of a
preprocessing command.
The logic in mock (at the corresponding place) should go as follows:
if rpkg_preprocessing is enabled; then
  if rpkg.conf file is present in git toplevel directory; then
     if the rpkg.conf file contains `preprocess_spec = True` directive
under [rpkg] section; then
        install `rpkg_preprocessing_requires` into the chroot
        run the preprocessing command as specified by the config
And these changes in mock (apart from the above packages being brought
into Fedora and EPEL) are the only changes that need to be done for
the solution to already work in Fedora (and EPEL).
I scanned master branches of all packages in Fedora dist-git and none
of them contains rpkg.conf file in the git top-level directory so
there should be no conflict there (i can do the checks on other
branches too but at this point, I am already quite certain that
'rpkg.conf' is a non-conflicting filename).
I have run all spec files from master through preproc and all of them
came out unchanged except two:
python-dns-lexicon.spec - uses {{{ }}} in comments
python-suds.spec - uses {{{ }}} in changelog at one place
All the other packages are just left untouched after the preproc run
(by the way, that wouldn't be the case for {{, }} tags if used - there
are many more conflicts there).
Those two packages would need to escape the {{{ }}} tags to keep using
them if they wanted to also start using the preprocessing
functionality by placing rpkg.conf file into dist-git repo with the
following content:
[rpkg]
preprocess_spec = True
I already have this working on my machine so now it is a matter of
opening a PR and polishing the code. Basically, I tested it on a
dist-git repo (on prunerepo package, specifically) where I added
rpkg.conf with the above content and used some preproc macros in the
spec file and I was able to successfully build an srpm with a
preprocessed spec file by invoking:
mock --buildsrpm --spec prunerepo.spec --sources .
This is pretty much what koji does when building from SCM (it just
adds more switches to the above command).
Implementation continued
-----------------------------------
The next thing that needs to be adjusted is the mass-rebuild script
(https://pagure.io/releng/blob/master/f/scripts/mass-rebuild.py) and
rpmdev-bumpspec
(https://github.com/rsrchboy/rpmdevtools/blob/master/rpmdev-bumpspec),
however also here the changes should be quite minimal.
My plan is to tweak rpmdev-bumpspec to return a dedicated exit code
when it encounters:
Release: {{{ ... }}}
and not to do any changes on release or changelog then with the
assumption that the macros will do it themselves. The mass rebuild
script when being given this exit code will not error out but instead
continue by adding a new commit
(https://pagure.io/releng/blob/682804c8/f/scripts/mass-rebuild.py#_162)
and building as usual.
This is not a 100% percent solution because one could e.g. do this in
a spec file:
{{{{ echo "Release: $(git_dir_release_branched)"  }}} in which case
regular expression checking e.g. for "Release: {{{" wouldn't work. It
might be therefore better to check for the rpkg.conf file and check if
it contains `preprocess_spec = True` in the [rpkg] section.
In either case, we can make this work by communicating with packagers
the following: "You are free to use preproc macros but if a new commit
lands, the Release field needs to change based on that". That's also
the behavior of `git_dir_release` and `git_dir_release_branched`
macros included in rpkg-macros library.
Implementation continued continued
------------------------------------------------
If the mock changes arrive to koji builders, you will be already able
to play around with the macros and I recommend using the latest
version of rpkg-utility
(https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/) that can
interpret the macros client-side and tell you exactly what the koji
builders produce before you push.
Anyway, I would like to get proper support for this in fedpkg. I think
we could immediately start supporting this on the level of the rpkg
library (https://pagure.io/rpkg) so that the other dist-git clients
get this functionality as well (in opt-in manner).
This is not a difficult task - i have already done the support in
rpkg-util so I (or maybe someone else) will just open a pull request
which should port the code from rpkg-util to the rpkg library.
Implementation finished?
---------------------------------
Is there anything else I should think about or some other scripts in
infra that should be adjusted?
About the approach
--------------------------
- the preproc macros (together with rpkg-macros) are already supported
in Copr when you use SCM method with rpkg srpm build method
- the approach is generic in the sense that it makes it very easy to
start generating other parts of spec files from git (or possibly
other) content: e.g. VCS field, Source field by packing the whole
dist-git repo content into a tarball, and %prep field - it depends on
what community and packaging committee decide is acceptable
- the approach gives packagers the same freedom they have with rpm
macros. While there can be some preconfigured macros, everybody can
roll out their own variants or entirely new macros
- the approach is suitable to implement "Git is the single source of
truth" paradigm at least with respect to the basic package information
like Release, Version or Changelog.
- the approach needs very little changes in our tooling
- it produces the same results locally which will be produced by build
system for a given commit
- guys from "Ideas and proposal for removing changelog and release
fields from spec file" are currently trying something else which
involves build system and, from my POV, magic spec file manipulations
but I still hope we can find a common language at some point and
finally start cooperating
Thanks
clime

Proposal to enable spec file preprocessing step before srpm build

Nils