packaging September 2021

packaging@lists.fedoraproject.org

21 participants
17 discussions

RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal
by Ben Cotton 13 Aug '22

13 Aug '22

https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog… == Summary == redhat-rpm-config will be updated so users of the auto framework get automated release and changelog bumping. == Owner == * Name: [[User:nim| Nicolas Mailhot]] * Email: <nicolas.mailhot at laposte.net> == Detailed Description == This is a system-wide change because all packages build with redhat-rpm-config, but it only concerns packages that opted to use this part of redhat-rpm-config (auto framework). The change will make those packages auto-bump and auto-changelog at the rpm level, in an infrastructure-independent way. == Benefit to Fedora == Autobumping removes a huge packager shore and makes timestamping in changelogs more reliable. == Scope == * Proposal owners: The feature is coded and works at the rpm level. Unfortunately, mock filters away the srpms containing the bump state, so it does not work in upper layers. * Other developers: The feature requires buy-in by mock developers (and probably koji developers) to lift the restrictions that block it above the rpm level. Also, it requires a mechanism to pass the user name and email that will be used in bumped changelogs (defining two variables in ~/.rpmmacros is sufficient at rpm level) * Mock issue: https://github.com/rpm-software-management/mock/issues/599 * Release engineering: https://pagure.io/releng/issue/9567 * Policies and guidelines: maybe eventually if things work out on the technical level * FPC issue: https://pagure.io/packaging-committee/issue/998 * Trademark approval: N/A (not needed for this Change) == Upgrade/compatibility impact == This is a pure build tooling update, it changes how things are built not what is built. == How To Test == A redhat-rpm-config packages with the changes and some example packages are available in https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-auto-… Since the mock/copr layer is currently blocking the feature, you need to install the redhat-rpm-config and forge macro packages available in this repo locally. Afterwards you can take any of the example packages in the repo and rebuild them with rpmbuild -ba to your heart content, and see the releases bump and the changelogs being updated accordingly. To get beautiful changelogs, you also need to add <pre> %buildsys_name Your name %buildsys_email Your email </pre> in ~/.rpmmacros == User Experience == N/A Packager experience change only == Dependencies == The change is a spin-off of https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macro… Therefore, it depends on the success of that other change and will probably need rebasing if the code in this other change evolves during the redhat-rpm-config merge. It also depends on mock / copr/ koji buy-in and changes, that may add their own requirements. == Contingency Plan == There is no contingency plan because the change will happen or not at all. == Documentation == There is as much documentation as the average redhat-rpm-config change (ie comments in the macro files themselves) == Release Notes == N/A Packager productivity change only -- Ben Cotton He / Him / His Senior Program Manager, Fedora & CentOS Stream Red Hat TZ=America/Indiana/Indianapolis

12 23

Mangling shebangs in text files: How to detect them, bug in the current implementation and possible solutions
by Miro Hrončok 29 Sep '21

29 Sep '21

Hello, for many releases, Fedora has the brp-mangle-sehbangs BuildRoot Policy Script that does the following: 1) Gets all executable files in the buildroot 2) Gets all "text" files from those 3a) Mangles shebangs that are "wrong" (e.g. #!/usr/bin/env node -> #!/usr/bin/node) 3b) Removes executable bits from "text" files without shebangs The idea behind this is that all "text" files that are executable need a shebang and if they don't have it, something is wrong. OTOH files that are "binary" don't need it. I intentionally put the terms "text" and "binary" in quotation marks, as the definition is somewhat fuzzy. Up until now, the script did the detection by utilizing the file tool to get the MIME type. If the MIME type starts with text/, it considered the executable to be a text file. However, a bug [1] has been discovered. Some obvious text files, such as executable JavaScript scripts, are detected as application/ (e.g. application/javascript), and hence are not considered "text". If a JavaScript executable script has the #!/usr/bin/env node shebang, the brp-mangle-sehbangs script does not mangle it. One possible solution [2] to this problem is to limit the number of bytes the MIME detection reads. My experiments showed that limiting the number of bytes to 8 always recognizes JavaScript (and other scripting languages) files as text/plain and binary files as application/octet-stream. As a side effect, it might make the BRP script faster. However, I am not sure if this approach is deterministic enough. Another solution, suggested by Florian Weimer [3], is to not detect MIME type at all, but use eu-elfclassify instead. The idea is quite simple: If (and only if) the executable file is ELF [4], it does not require a shebang. Instead of some fragile idea about what files are text and what files are binary, this is quite deterministic. It allows mangling shebangs of executable ZIP files etc. I've drafted the eu-elfclassify solution in a pull request [5]. However, we have discovered that several non-elf binary formats in Fedora are possibly legitimately executable. E.g. .exe files (for mono or wine) or other formats registered with the kernel [6]. We are presented with 3 possible actions: 1) Keep the script as it is, say the text/ MIME type limitation is how this BRP script was scoped. Affected packages would need to correct shebangs manually. 2) Limit the MIME type detection to 8 bytes and hope it will not yield incorrect results. 3) Use eu-elfclassify. Consider non-ELF executables without shebangs bogus and document this. Packages that are affected would need to opt-out. What do you think? [1] https://bugzilla.redhat.com/1998924 [2] https://bugzilla.redhat.com/1998924#c3 [3] https://bugzilla.redhat.com/1998924#c4 [4] https://en.wikipedia.org/wiki/Executable_and_Linkable_Format [5] https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/145 [6] https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

6 8

contacting package owners
by cheese＠nosuchhost.net 29 Sep '21

29 Sep '21

hi i tried to contact package owners of the package "mysqltuner" and thought it would be sufficiant to mail to PACKAGE-owner(a)fedoraproject.org and there tried mysqltuner-owner(a)fedoraproject.org but this did now work. i think that used to work some years ago. i than tried: mysqltuner-owners(a)fedoraproject.org mysqltuner-owner(a)lists.fedoraproject.org did not work i knew that we *had* some address for contaction all owners/packagers, but i did not find a solution via google (maybe my fault) and i miss a search over our documentation at https://docs.fedoraproject.org/en-US/packaging-guidelines/ therefore: *) what is currently the correct way to contact package-owners? *) we should have some simple search-box for our own documentation yours josef

4 4

Sphinx/Doxygen HTML documentation and bundling guidelines
by Ben Beasley 29 Sep '21

29 Sep '21

It was recently brought up in a package review (https://bugzilla.redhat.com/show_bug.cgi?id=2006555) that a Sphinx-generated HTML documentation package contained several JavaScript libraries, and that: 1. their licenses should be accounted for and added to the License field for the -doc subpackage 2. they should be treated as bundled libraries and marked with virtual Provides Note that the JavaScript is inserted by the Sphinx documentation build system, and is not present in the source tarball. Your “typical” Sphinx documentation has minified and unminified copies of JQuery (js-jquery) and Underscore (js-underscore), both MIT-licensed, plus several unminified utility libraries that belong to Sphinx itself (doctools.js, language_data.js, searchtools.js). The overall Sphinx project is BSD-licensed. The details vary by theme, but this is the most common case. To me, it seems that this feedback is a reasonable strict reading of the relevant packaging guidelines. If this is the right interpretation of Fedora policy, what should be done about it in general? ----- Based on looking for installed files named “searchtools.js” in packages named -doc or -docs, there are about 600 documentation packages generated with Sphinx. This might miss some that have exotic themes. As far as I can tell, only one existing package in the distribution, python-murano-package-check, tries to use virtual Provides to track the libraries that belong to Sphinx itself: Provides: bundled(js-doctools) Provides: bundled(js-jquery) = 3.1.0 Provides: bundled(js-searchtools) Provides: bundled(js-underscore) = 1.3.1 Provides: bundled(js-websupport) and this package does not actually build its documentation subpackage. Only the following packages have virtual Provides for js-jquery and js-underscore. In most cases these lack the version numbers. - arb-doc - mpdecimal-doc - python-BTrees-doc - python-latexcodec-doc - python-networkx-doc - python3-persistent-doc - sympy-doc Of the packages in the list above, only mpdecimal-doc has a License field that seems to try to account for the licenses of the JavaScript libraries. ----- Personally, I can add virtual Provides and adjust the License on the -doc subpackage for my package under review, and it’s even possible for me to replace the Underscore and JQuery libraries with symbolic links to those installed by js-jquery and js-underscore. However, changing one package, or even every package I control, is a tiny drop in a vast ocean of documentation packages. Consider, for example, that Doxygen also includes JQuery and several Doxygen-specific libraries in its output, so the same arguments about licenses and virtual Provides apply to it. A search for “dynsections.js” in packages named *-doc or *-docs gives nearly 400 Doxygen-based packages. If this is a real problem, it seems like it needs to be handled and/or documented in a way that can potentially scale to the rest of those 600 Sphinx-based packages, to the nearly 400 Doxygen-based packages, and beyond that to the output of other documentation systems. I am not sure what that would look like. All thoughts are welcome. – Ben Beasley (FAS: music)

4 7

Re: Mangling shebangs in text files: How to detect them, bug in the current implementation and possible solutions
by Miro Hrončok 23 Sep '21

23 Sep '21

On 23. 09. 21 1:40, Josh Stone wrote: > On 9/22/21 4:21 AM, Miro Hrončok wrote: >> Hello, >> >> for many releases, Fedora has the brp-mangle-sehbangs BuildRoot Policy Script >> that does the following: >> >> 1) Gets all executable files in the buildroot >> 2) Gets all "text" files from those >> 3a) Mangles shebangs that are "wrong" >> (e.g. #!/usr/bin/env node -> #!/usr/bin/node) >> 3b) Removes executable bits from "text" files without shebangs > > While we're at it, can we teach the script to ignore Rust attributes? > They're written like #![attr...], and when that's on the first line some > editors try to be helpful and make the file executable. That's > considered an error with the current script since the "shebang" doesn't > start with '/', but it would be best IMO to have it remove the > executable bit. I believe that currently the script would error: ERROR: $f has shebang which doesn't start with '/' (#![attr...]) Have you ever seen that in a Fedora package? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

Re: Mangling shebangs in text files: How to detect them, bug in the current implementation and possible solutions
by Petr Pisar 23 Sep '21

23 Sep '21

V Wed, Sep 22, 2021 at 12:47:07PM -0400, Steve Grubb napsal(a): > I find the file utility to be almost reliable. It changes how it identifies ELF > files every couple releases. So, to stabilize this, fapolicyd-cli uses it's > own logic to determine what kind of ELF file it finds. I also regularly find > text/plain files where it cannot identify the language and files that are > application/octet-stream which are also misidentified. > File's libmagic will always misdetect some files. I'd like to see rpmbuild to prefer user.mime_type extended attribute over libmagic guess. That way packagers could override the MIME type directly from a spec file: %install setfattr -n 'user.mime_type' -v 'text/x-perl' %{buildroot}%{_bindir}/GET If rpmbuild carried that attribute to RPM archive, rpm would set the attribute when installing that package and it would become available to other tools like fapolicyd-cli. Technically we could patch libmagic to do that, but I feel that libmagic upstream wouldn't like that. Maybe a place for an intermediate wrapper. -- Petr

1 0

Re: Mangling shebangs in text files: How to detect them, bug in the current implementation and possible solutions
by Miro Hrončok 22 Sep '21

22 Sep '21

On 22. 09. 21 22:48, Steve Grubb wrote: > On Wednesday, September 22, 2021 4:26:49 PM EDT Miro Hrončok wrote: >>> By chance do you have a pointer to one of those javascript files that is >>> misidentified? (Or any other for that matter). I'd like to see what's >>> going on and get a fix in place. >> >> yarnpkg package, %prepped >> >> $ file --mime-type yarn-1.22.10/bin/yarn.js >> yarn-1.22.10/bin/yarn.js: application/javascript > > application/javascript is, unfortunately, correct. This one is governed by > RFC 4329 which makes it official. > > From all the scan that we've done on fullish installs in the past, there's > only 2 others that you might run across: application/x-elc (lisp) and > application/x-java-applet. > > Maybe you just build in logic to workaround these 3 types? application/ > javascript is really the only one I can think of that is common. Yeah, maybe we should just do that. However, that would not cleanup the executable pngs. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

Re: Mangling shebangs in text files: How to detect them, bug in the current implementation and possible solutions
by Miro Hrončok 22 Sep '21

22 Sep '21

On 22. 09. 21 21:46, Steve Grubb wrote: > By chance do you have a pointer to one of those javascript files that is > misidentified? (Or any other for that matter). I'd like to see what's going on > and get a fix in place. yarnpkg package, %prepped $ file --mime-type yarn-1.22.10/bin/yarn.js yarn-1.22.10/bin/yarn.js: application/javascript -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

Re: Mangling shebangs in text files: How to detect them, bug in the current implementation and possible solutions
by Miro Hrončok 22 Sep '21

22 Sep '21

On 22. 09. 21 18:47, Steve Grubb wrote: > 4) maybe fapolicyd-cli has better detection? Or at least, its more closely > maintained. It also has it's own ELF detection so that it's stable from > release to release. Not checked whether it has better detection or not, but I see it would pull the following packages into the default buildroot: acl cryptsetup-libs dbus-broker dbus-common dbus device-mapper-libs device-mapper expat fapolicyd-dnf-plugin fapolicyd iptables-legacy-libs json-c kmod-libs libargon2 libibverbs libnl3 libpcap libseccomp lmdb-libs mpdecimal python-pip-wheel python-setuptools-wheel python3-libs python3 systemd-pam systemd-rpm-macros systemd Installed size: 54 M I don't think this is acceptable. Even systemd and Python alone are not supposed to be there. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

Self Introduction: Eric Curtin
by Eric Curtin 21 Sep '21

21 Sep '21

Hi Guys, As per: https://docs.fedoraproject.org/en-US/package-maintainers/Joining_the_Packag… here is my introduction. I am the upstream maintainer of inotify-tools and recent hire at Red Hat. I would like to join the Fedora Package Maintainers. Is mise le meas/Regards, Eric Curtin

5 5

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

packaging September 2021