python-devel August 2020

python-devel@lists.fedoraproject.org

8 participants
11 discussions

Draft of New Python Packaging Guidelines
by Petr Viktorin 15 Jun '21

15 Jun '21

Hello! Below is a draft of new Packaging Guidelines! It's full of unfinished areas (marked with XXX), but it's ready for scrutiny. A possibly updated version is on https://hackmd.io/XzJe-sHUQvWK7cSrEH_aKg?view Generally, for rules marked **SHOULD** we know of cases where they should be broken; for things marked **MUST** we don't. We have tried to only take the Good Existing Things™ (macros, practices) and revise the rest without thinking about the past much. Some used technology is unfortunately not compatible with current EPELs, but we have considered Fedora 31+. Using the current Python guidelines will still be an option for people who target EPEL 6/7/8. The main controversial idea (of many) is synchronizing Fedora's names (a.k.a. `python3dist(...)`, a.k.a. `name` in `setup.py`) with the Python Package Index (PyPI, pypi.org), which has by now become the de-facto canonical namespace for freely redistributable Python packages. We believe that this will help both Fedora and the greater Python ecosystem, but there will definitely be some growing pains. Most of Fedora Python packages already are on PyPI, but more than 250 are missing. There is software developed within Fedora (e.g. pagure, fedpkg, rust2rpm); projects that aren't primarily Python packages (e.g. perf, ceph, xen, setroubleshoot, iotop); and others. A full list is attached. The names have been temporarily blocked on PyPI to keep trolls from taking them while this is discussed. Over at the Python Discourse we are discussing how to properly handle these; once that discussion is settled they should be unblocked: https://discuss.python.org/t/pypi-as-a-project-repository-vs-name-registry-… Another general idea is that package metadata should be kept upstream; the spec file should duplicate as little of it as possible. Any adjustments should be done with upstreamable patches. The draft lives on hackmd.io, which we found easy to collaborate with. If you have an account there, we can add you. If you'd like to collaborate some other way, let us know. Petr and Miro ----------------------- Current draft for inline comments: > [IMPORTANT] > This is a DRAFT; it is not in effect yet. # Python Packaging Guidelines (draft) > [IMPORTANT] > This is a *beta* version of the Python Packaging Guidelines and the associated RPM macros. > Packagers that opt in to following this version **MUST** be prepared to change their packages frequently when the guidelines or macros are updated. > These packagers **SHOULD** join [Python SIG mailing list](https://lists.fedoraproject.org/archives/list/python-devel@lists.fedo… and monitor/start conversations there. These Guidelines represent a major rewrite and paradigm shift, and not all packages are updated to reflect this. Older guidelines are still being kept up to date, and existing packages **MAY** use them instead of this document: * 201x-era Python packaging guidelines (for packages that use e.g. `%py3_install` or `setup.py install`) * Python 2 appendix (for e.g. `%py2_install`) (requires FESCo exception) > [NOTE] > These guidelines only support Fedora 31+. For older releases (such as in EPEL 8), consult the older guidelines (XXX link). The two "Distro-wide guidelines" below apply to all software in Fedora that uses Python at build- or run-time. These rest of the Guidelines apply to packages that ship code that *can* be imported in Python. Specifically, that is all packages that install files under `/usr/lib*/python*/`. Except for the two "Distro-wide guidelines", these Guidelines do not apply to simple one-file scripts or utilities, especially if these are included with software not written in Python. However, if an application (e.g. CLI tool, script or GUI app) needs a more complex Python library, the library **SHOULD** be packaged as an importable library under these guidelines. A major goal for Python packaging in Fedora is to *harmonize with the wider Python ecosystem*, that is, the [Python Packaging Authority](https://pypa.io) (PyPA) standards and the [Python Package Index](https://pypi.org) (PyPI). Packagers **SHOULD** be prepared to get involved with upstream projects to establish best practices as outlined here. We wish to improve both Fedora and the wider Python ecosystem. > [NOTE] > Fedora's Python SIG not only develops these guidelines, but it's also involved in PyPA standards and Python packaging best practices. Check out [the wiki](https://fedoraproject.org/wiki/SIGs/Python) or [mailing list](https://lists.fedoraproject.org/archives/list/python-devel@lists.fedo… if you need help or wish to help out. ## Distro-wide guidelines ### BuildRequire python3-devel **Every** package that uses Python (at runtime and/or build time), and/or installs Python modules **MUST** explicitly include `BuildRequires: python3-devel` in its `.spec` file, even if Python is not actually invoked during build time. If the package uses an alternate Python interpreter instead of `python3` (e.g. `pypy`, `jython`, `python27`), it **MAY** instead require the coresponding `*-devel` package. The `*-devel` package brings in relevant RPM macros. It may also enable automated or manual checks: for example, Python maintainers use this requirement to list packages that use Python in some way and might be affected by planned changes. ### Mandatory macros The following macros **MUST** be used where applicable. The expansions in parentheses are provided only as reference/examples. The macros are defined for you in all supported Fedora and EPEL versions. * `%{python3}` (`/usr/bin/python3`) The Python interpreter. For example, this macro should be used for invoking Python from a `spec` file script, passed to `configure` scripts to select a Python executable, or used as `%{python3} -m pytest` to run a Python-based tool. > XXX: Use shebang opts for shebang; document pathfix macro. Document cases where you don't want this. * `%{python3_version}` (e.g. `3.9`) Version of the Python interpreter. * `%{python3_version_nodots}` (e.g. `39`) Version of the Python interpreter without the dot. * `%{python3_sitelib}` (e.g. `/usr/lib/python3.9/site-packages`) Where pure-Python modules are installed. * `%{python3_sitearch}` (e.g. `/usr/lib64/python3.9/site-packages`) Where Python3 extension modules (native code, e.g. compiled from C) are installed. The rest of this document uses these macros, along with `%{_bindir}` (`/usr/bin/`), instead of the raw path names. ## Python implementation support Fedora primarily targets *CPython*, the reference implementation of the Python language. We generally use “Python” to mean CPython. Alternate implementations like `pypy` are available, but currently lack comprehensive tooling and guidelines. When targetting these, there are no hard rules (except the general Fedora packaging guidelines). But please try to abide by the *spirit* of these guidelines. When in doubt, consider consulting the Python SIG. ## Python version support Fedora packages **MUST NOT** depend on other versions of the CPython interpreter than the current `python3`. In Fedora, Python libraries are packaged for a single version of Python, called `python3`. For example, in Fedora 32, `python3` is Python 3.8. In the past, there were multiple Python stacks, e.g. `python3.7` and `python2.7`, installable together on the same machine. That is also the case in some projects that build *on top* of Fedora, like RHEL, EPEL and CentOS. Fedora might re-introduce parallell-installable stacks in the future (for example if a switch to a new Python version needs a transition period, or if enough interested maintainers somehow appear). > XXX dots in package names! Fedora does include alternate interpreter versions, e.g. `python2.7` or `python3.5`, but these are meant only for developers that need to test upstream code. Bug and security fixes for these interpreters only cover this use case. Packages such as `pip` or `tox`, which enable setting up isolated environments and installing third-party packages into them, **MAY**, as an exception to the rule above, use these interpreters as long as this is coordinated with the maintainers of the relevant Python interpreter. ## BuildRequire pyproject-rpm-macros While these guidelines are in Beta, each Python package **MUST** have `BuildRequires: pyproject-rpm-macros` to access the beta macros. (When we go out of Beta, listing the dependency won't be necessary: we'll make `python3-devel` require it.) ## Naming Python packages have several different names, which should be kept in sync but will sometimes differ for historical or practical reasons. They are: * the Fedora *source package name* (or *component name*, `%{name}`), * the Fedora *built RPM name*, * the *project name* used on *PyPI* or by *pip*, and * the *importable module name* used in Python (a single package may have multiple importable modules). Some examples (both good and worse): | Fedora component | Built RPM | Project name | Importable module | | ----------------- | ------------------ | -------------- | -------------------- | | `python-requests` | `python3-requests` | `requests` | `requests` | | `PyYAML` | `python3-pyyaml` | `pyyaml` | `yaml` | | `python-ldap` | `python3-ldap` | `python-ldap` | `ldap`, `ldif`, etc. | | `python-pillow` | `python3-pillow` | `pillow` | `PIL` | Elsewhere in this text, the metavariables `SRPMNAME`, `RPMNAME`, `PROJECTNAME`, `MODNAME` refer to these names, respectively. ### Canonical project name Most of these names are case-sensitive machine-friendly identifiers, but the *project name* has human-friendly semantics: it is case-insensitive and treats some sets of characters (like `._-`) specially. For automated use, it needs to be [normalized](https://www.python.org/dev/peps/pep-0503/#normalized-names) to a canonical format used by Python tools and services such as setuptools, pip and PyPI. The `%{py_dist_name}` macro implements this normalization. Elsewhere in this text, the metavariable `DISTNAME` refers to the canonical form of the project name. > XXX > ```spec > # XXX specfile > %py_set_name Django > -> %distname django > -> %pojectname Django > ``` > > XXX rewrite: It is customary to define the macro `%{distname}` as the canonical project name and use it throughout the `spec` file. Some helper macros use `%{distname}` for this purpose by default. > > XXX in some places, the "original project name" must be used -- e.g. `%pypi_source` and `%autosetup` need `Django`, not `django`. > XXX The following sections should supersede the [Python section on the Naming guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Namin…. ### Library naming A built (i.e. non-SRPM) package for a *Python library* **MUST** be named with the `python3-` prefix. A source package containing primarily a *Python library* **MUST** be named with the prefix `python-`. The Fedora package's name **SHOULD** contain the *canonical project name*. If possible, the project name **SHOULD** be the same as the name of the main importable module, with underscores (`_`) replaced by dashes (`-`). A *Python library* is a package meant to be imported in Python, such as with `import requests`. Tools like *Ansible* or *IDLE*, whose code is importable but not primarily meant to be imported, are not considered libraries in this sense, so this section does not apply for them. (See the [general Libraries and Applications guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/#_lib… for details.) If the importable module name and the project name do not match, users frequently end up confused. In this case, packagers *should* ensure that upstream is aware of the problem and (especially for new packages where renaming is feasible) strive to get the package renamed. The Python SIG is available for assistance. The Fedora package name should be formed by taking the *project name* and prepending `python-` if it does not already start with `python-`. This may leads to conflicts (e.g. between [bugzilla](https://pypi.org/project/bugzilla/) and [python-bugzilla](https://pypi.org/project/python-bugzilla/)). In that case, ensure upstream is aware of the potentially confusing naming and apply best judgment. ### Application naming Packages that primarily provide executables **SHOULD** be named according to the general Fedora guidelines (e.g. `ansible`). Consider adding a virtual provide according to Library naming above (e.g. `python3-PROJECTNAME`), if it would help users find the package. ## Files to include ### Source files and bytecode cache Packages **MUST** include the source file (`*.py`) **AND** the bytecode cache (`*.pyc`) for each pure-Python module. The cache files are found in a `__pycache__` directory and have an interpreter-dependent suffix like `.cpython-39.pyc`. The cache is not necessary to run the software, but if it is not found, Python will try to create it. If this succeeds, the file is not tracked by RPM and it will linger on the system after uninstallation. If it does not succeed, users can get spurious SELinux AVC denials in the logs. Normally, byte compilation (generating the cache files) is done for you by the `brp-python-bytecompile` script (XXX link to BRP guidelines), which runs automatically after the `%install` section of the spec file has been processed. It byte compiles any `.py` files that it finds in `%{python3_sitelib}` or `%{python3_sitearch}`. You must include these files of your package (i.e. in the `%files` section). If the code is in a subdirectory (importable package), include the entire directory: ``` %files %{python3_sitelib}/foo/ ``` Adding the trailing slash is best practice for directories. However, this cannot be used for top-level scripts (those directly in e.g. `%{python3_sitelib}`), because both `%{python3_sitelib}` and `%{python3_sitelib}/__pycache__/` are owned by Python itself. Here, the `%pycached` macro can help. It expands to the given `*.py` source file and its corresponding cache file(s). For example: ``` %files %pycached %{python3_sitelib}/foo.py ``` expands roughly to: ``` %files %{python3_sitelib}/foo.py %{python3_sitelib}/__pycache__/foo.cpython-3X{,.opt-?}.pyc ``` #### Manual byte compilation If you need to bytecompile stuff outside of `%{python3_sitelib}`/`%{python3_sitearch}`, use the `%py_byte_compile` macro. For more details on `%py_byte_compile` and on the internals of bytecode compilation, please see the [appendix](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_…. > XXX: Copy the section from appendix here ### Dist-info metadata Each Python package **MUST** include *Package Distribution Metadata* conforming to [PyPA specifications](https://packaging.python.org/specifications/) (specifically, [Recording installed distributons](https://packaging.python.org/specifications/recording-install…). This applies to libraries (e.g. `python-requests`) as well as tools (e.g. `ansible`). > XXX what with splitting into subpackages? 1) dist-info always installed, 2) dist-info installed trough a metapackage? > * Ideally, do the split upstream > * Consider package split between library & tool (see poetry, fedpkg) > > e.g. > When software is split into several subpackages, it is OK to only ship metadata in one built RPM. The metadata takes the form of a `dist-info` directory installed in `%{python3_sitelib}` or `%{python3_sitearch}`, and contains information that tools like [`importlib.metadata`](https://docs.python.org/3/library/importlib.metadata.html) use to introspect installed libraries. > XXX example %files with manual dist-info entry Note that some older tools instead put metadata in an `egg-info` directory, or even a single file. This won't happen if you use the `%pyproject_wheel` macro. If your package uses a build system that generates an `egg-info` directory or file, please contact Python SIG. > XXX We need a better solution before we go out of beta. As an exception, the Python standard library **MAY** ship without this metadata. ### Explicit lists Packagers **SHOULD NOT** simply glob everything under a shared directory. In particular, the following **SHOULD NOT** be used: * `%{python3_sitelib}/*` * `%{python3_sitearch}/*` * `%{python_sitelib}/*` * `%{python_sitearch}/*` * `%{_bindir}/*` * `%pyproject_save_files *` * `%pyproject_save_files +bindir` This rule serves as a check against common mistakes which are otherwise hard to detect. It does limit some possibilities for automation. The most common mistake this rule prevents is installing a test suite system-wide as an importable module named `test`, which would then conflict with other such packages. ## PyPI parity Every Python package in Fedora **SHOULD** also be available on [the Python Package Index](https://pypi.org) (PyPI). The command `pip install PROJECTNAME` **MUST** install the same package (possibly in a different version), install nothing, or fail with a reasonable error message. If this is not the case, the packager **MUST** contact upstream about this. The goal is to get the project name registered or blocked on PyPI, or to otherwise ensure the rule is followed. > XXX Note that project names that were in Fedora but not on PyPI when these guidelines were proposed are *blocked* as we discuss how they should be handled. > This prevents potential trolls, but also legitimate owners, from taking them. This is necessary to protect users, avoid confusion, and enable automation as Fedora and upstream ecosystems grow more integrated. As always, specific exceptions can be granted by FPC (XXX link exceptions rules). > XXX Write an automated check for this. ## Provides and requirements ### Provides for importable modules For any module intended to be used in Python 3 with `import MODNAME`, the package that includes it **SHOULD** provide `python3-MODNAME`, with underscores (`_`) replaced by dashes (`-`). This is of course always the case if the package is named `python3-MODNAME`. If the subpackage has some other name, then add `%py_provides python3-MODNAME` explicitly. See the following section to learn about `%py_provides`. ### Automatic unversioned provides For any `FOO`, a package that provides `python3-FOO` **SHOULD** use `%py_provides` or an automatic generator to also provide `python-FOO`. The provide **SHOULD NOT** be added manually: if the generator or macro is not used, the provide shall not be added at all. On Fedora 33+, this is done automatically for package names by a generator. The generator can be disabled by undefining `%__pythonname_provides`. The following macro invocation will provide both `python3-FOO` and `python-FOO`: %py_provides python3-FOO > XXX: finalize `%py_provides` Using the generator or macro is important, because the specific form of the provide may change in the future. ### Machine-readable provides Every Python package **MUST** provide `python3dist(DISTNAME)` **and** `python3.Xdist(DISTNAME)`, where `X` is the minor version of the interpreter and `DISTNAME` is the *canonical project name* corresponding to the *dist-info metadata*, for example `python3.9dist(requests)`. (XXX: add links to previous sections) This is generated automatically from the dist-info metadata. The provide **SHOULD NOT** be added manually: if the generator fails to add it, the metadata **MUST** be fixed. If necessary, the automatic generator can be disabled by undefining `%__pythondist_provides`. These *Provides* are used for automatically generated *Requires*. ### Dependencies As mentioned above, each Python package **MUST** explicitly BuildRequire `python3-devel`. Packages **MUST NOT** have dependencies (either build-time or runtime) with the unversioned prefix `python-` if the corresponding `python3-` dependency can be used instead. Packages **SHOULD NOT** have explicit dependencies (either build-time or runtime) with a minor-version prefix such as `python3.8-` or `python3.8dist(`. Such dependencies **SHOULD** instead be automatically generated or a macro should be used to get the version. Packages **SHOULD NOT** have an explicit runtime dependency on `python3`. Instead of depending on `python3`, packges have an automatic dependency on `python(abi) = 3.X` when they install files to `%{python3_sitelib}` or `%{python3_sitearch}`, or they have an automatic dependency on `/usr/bin/python3` if they have executable Python scripts, or they have an automatic dependency on `libpython3.X.so.1.0()` if they embed Python. These rules help ensure a smooth upgrade path when `python3` is updated in new versions of Fedora. ### Automatically generated dependencies Packages **MUST** use the automatic Python run-time dependency generator. Packages **SHOULD** use the opt-in build-dependency generator if possible. Any necessary changes **MUST** be done by patches or modifying the source (e.g. with `sed`), rather than disabling the generator. The resulting change **SHOULD** be offered to upstream. As an exception, [filtering](https://docs.fedoraproject.org/en-US/packaging-guidelines/AutoPr… **MAY** be used for temporary workarounds and bootstrapping. Dependencies covered by the generators **SHOULD NOT** be repeated in the `.spec` file. (For example, if the generator finds a `requests` dependency, then `Requires: python3-requests` is redundant.) The automatically generated requirements are in the form `python3.Xdist(DISTNAME)`, potentially augmented with version requirements or combined together with [rich dependencies](https://rpm.org/user_doc/boolean_dependencies.html). Note that the generators only cover Python packages. Other dependencies, often C libraries like `openssl-devel`, must be specified in the `.spec` file manually. > XXX When implemented, this goes here: Alternatively, upstream Python packages may list non-Python dependencies in the `[tool.fedora.requires]`/`[tool.fedora.buildrequires]` section of `pyproject.toml`. This is non-standard and only recommended for Fedora-related packages. Where the requirements are specified in the source depends on each project's build system and preferences. Common locations are `pyproject.toml`, `setup.py`, `setup.cfg`, `config.toml`. #### Run-time dependency generator The automatic runtime dependency generator uses package metadata (as recorded in installed `*.dist-info` directories) to determine what the package depends on. In an emergency, you can opt-out from running the requires generator by adding `%{?python_disable_dependency_generator}` to the package (usually, just before the main package’s `%description`). #### Build-time dependency generator The opt-in (but strongly recommended) build-time dependency generator gathers information from [`pyproject.toml` build-system information](https://www.python.org/dev/peps/pep-0517/#source-trees) (with fallback to `setuptools`) plus a standardized [build-system hook](https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-whee… to gather further requirements. See `%pyproject_buildrequires` (XXX link) for more details. ### Test dependencies See the *Tests* section. (XXX link.) ### Extras > XXX No story here so far. > XXX Note: [python-django-storages](https://src.fedoraproject.org/rpms/python-django-st…, `drf-yasg` etc. do `%python_provide python3-%{srcname}+%{distname}` and `python3dist(%{srcname}/%{extraname})` XXX `python3dist(%{srcname}[%{extraname}])` ## Interpreter invocation ### Shebangs Shebangs lines to invoke Python **SHOULD** be `#!%{python3} -%{py3_shebang_flags}` and it **MAY** include extra flags. > XXX define py3_shebang_flags If the default flags from `%{py3_shebang_flags}` are not desirable, packages **SHOULD** explicitly redefine the macro to remove them. Using `#!%{python3}` (`#!/usr/bin/python3`) rather than e.g. `#!/usr/bin/env python` ensures that the system-wide Python interpreter is used to run the code, even if the user modifies `$PATH` (e.g. by activating a virtual environment). By default, `-%{py3_shebang_flags}` expands to `-s`, which means *don't add user site directory to `sys.path`*. That ensures user-installed Python packages (e.g. by `pip install --user`) don't interfere with the RPM installed software. Sometimes, `pip`-installed content is desirable, such as with plugins. Redefining `%{py3_shebang_flags}` to not include `s`, rather than not using the macro at all, ensures that existing or future automation won't add the flag. The `%pyproject_install` macro automatically changes all Python shebangs in `%{buildroot}%{_bindir}/*` to use `%{python3}` and add `%{py3_shebang_flags}` to the existing flags. If you're not using the macro or you need to change a shebang in a different directory, you can use the `pathfix.py` command as follows: ``` pathfix.py -n -p -k -i %{python3} -a %{py3_shebang_flags} SCRIPTNAME … ``` > XXX Ouch! Macroize this? Hell yes! > > `%py3_shebang_fix <paths>` -- also rename `%{py3_shebang_flags}` (doesn't include dash) > > can be used in `%prep` or `%install` where: * `-n` disables ceating backups * `-p` preserves timestamps * `-k` keeps existing flags * `-i` specifies the new interpreter ### Invokable Python modules Every executable `TOOL` for which the current version of Python matters **SHOULD** also be invokable by `python3 -m TOOL`. If the software doesn't provide this functionality, packagers **SHOULD** ask the upstream to add it. This applies to tools that modify the current Python environment (like installing or querying packages), use Python for configuration, or use Python to run plugins. It does not apply to tools like GIMP or Bash which support plugins in multiple languages and/or have other means to specify the interpreter. For example, `pip` can be invoked as `python3 -m pip`. This allows users to accurately specify the Python version used to run the software. This convention works across different environments that might not always set `$PATH` or install scripts consistently. ## Using Cython Tightening the general Fedora policy, packages **MUST NOT** use files pre-generated by Cython. These **MUST** be deleted in `%prep` and regenerated during the build. As an exception, these sources **MAY** be used temporarily to prevent build time circular dependencies by following the [bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#b… guidelines. Generated files (the ones that must be deleted) have a generic `.c` or `.cpp` extension. Cython source files (which should stay) usually have the `.pyx` or `.pxd` extension. Cython is a popular tool for writing extension modules for Python. If compiles a Python-like language to C, which is then fed to the C compiler. Historically, Cython was hard to use upstream as a build-time dependency. Many projects include pre-generated C files in source distribution to avoid users from needing to install the tool. Cython uses CPython's fast-changing internal API for performance reasons. For a new release of Python, Cython generally needs to be updated and the C files regenerated. In Fedora, this is frequently needed before upstreams release re-generated sources (e.g. for Alpha versins of Python). Since we do not have a problem with build-time dependencies, we always want to run the Cython step. > XXX example spec snippet ## Tests ### Running tests If a test suite exists, it **MUST** be run in the `%check` section and/or in Fedora CI. You **MAY** exclude specific failing tests. You **MUST NOT** disable the entire testsuite or ignore the result to solve a build failure. As an exception, you **MAY** disable tests with an appropriate `%if` conditional (e.g. bcond) when [bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#b…. A popular testing tool, and one which is well integrated in Fedora, is `tox`. Upstream, it is commonly used to test against multiple Python versions. In a Fedora package, BuildRequire test dependencies (see *Test dependencies* below) and run `tox` with: ``` %tox ``` This sets up the environment (`$PATH`, `$PYTHONPATH`, `$TOX_TESTENV_PASSENV`) and instructs `tox` to use the current environment rather than create new ones. For more options, see *Build macros* (XXX link to section). When upstream doesn't use `tox`, the tests need to be run directly depending on upstream choice of a test runner. A popular runner is `pytest`, which can be run against the package to be installed using: ``` export PATH=%{buildroot}%{_bindir} export PYTHONPATH=%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} /usr/bin/pytest # XXX `%{python} -m pytest` doesn't work :( -- sees PWD ``` > XXX Do we want a macro for that? We do, because not all projects have tox. > > Define %__pytest or %_pytest %pytest_command - /usr/bin/pytest > > Define %pytest (see above snippet) (not parametric) Use positional arguments to specify test directory, and `-k` to select tests (e.g. `-k "not network"` may deselect all network-related tests). ### Test dependencies One part of the Python packaging ecosystem that is still not standardized is specifying test dependencies (and development dependencies in general). The best practice to specify tests is using an extra (XXX link to section above, which should be fleshed out) like `[test]` or `[dev]`. In this case, upstream's instructions to install test dependencies might look like `pip install -e.[test]`. Projects using `tox` usually specify test dependencies in a `tox`-specific format: a [requires](https://tox.readthedocs.io/en/latest/config.html#conf-requires) key in the configuration. Both forms are handled by the `%pyproject_buildrequires` macro, see below. If upstream does not use either form, list test dependencies as manual *BuildRequires* in the `spec` file. ### Linters In `%check`, packages **SHOULD NOT** run “linters”: code style checkers, test coverage checkers and other tools that check code quality rather than functionality. Tools like `black`, `pylint`, `flake8`, or `mypy` are often “opinionated” and their “opinions” change frequently enough that they are nuisance in Fedora, where the linter is not pinned to an exact version. Furthermore, some of these tools take a long time to adapt to new Python versions, preventing early testing with Aplha and Beta releases of Python. And they are just not needed: wrongly formatted code is not important enough for the Fedora packager to bug the upstream about it. Making such an issue break a package build is entirely unreasonable. Linters *do* make sense in upstream CI. But not in Fedora. If a linter is used, disable it and remove the dependency on it. If that is not easy, talk to upstream about making it easy (for example with a configuration option or a separate `tox` environment). For packages that contain such linters, use them at runtime or extend them, you will usually need to run the linter in `%check`. Run it to test functionality, not code quality of the packaged software. ## Source files from PyPI Packages **MAY** use sources from PyPI. However, packages **SHOULD NOT** use an archive that omits test suites, licences and/or documentation present in other source archives. For example, as of this writing `pip` provides a [source tarball (“sdist”)](https://pypi.org/project/pip/#files) which omits the relatively large `tests` and `docs` directories present in [the source on GitHub](https://github.com/pypa/pip). In this case, the tarball from GitHub should be used (XXX link to source URL guidelines). When using sources from PyPI, you can use the `%pypi_source` macro to generate the proper URL. See the *Macro reference* (XXX link directly to macro) for details. ## Sample spec file The following is a viable spec file for a hypothetical Python library called `pello` that follows packaging best practices. > XXX *Need to get `Pello` uploaded as real example and on GH under fedora-python. ``` %py_set_name Pello Name: python-%{distname} Version: 1.2.3 Release: 1%{?dist} Summary: Example Python library License: MIT URL: https://github.com/fedora-python/Pello Source0: %{pypi_source} BuildArch: noarch BuildRequires: python3-devel BuildRequires: pyproject-rpm-macros %global _description %{expand: A python module which provides a convenient example. This description provides some details.} %description %_description %package -n python3-%{distname} Summary: %{summary} %description -n python3-%{distname} %_description %prep %autosetup -p1 -n %{projectname}-%{version} %generate_buildrequires %pyproject_buildrequires -t %build %pyproject_wheel %install %pyproject_install # Here, "pello" is the name of the importable module. # You may use %%{distname} here if it's the same. %pyproject_save_files pello %check %tox # Note that there is no %%files section for # the unversioned python module, python-%%{distname} %files -n python3-%{distname} -f %{pyproject_files} %doc README.md %license LICENSE %{_bindir}/pello-greeting ``` ## Macro Reference See the *Mandatory macros* section above. (XXX link to section)  * `%{python3}` (`/usr/bin/python3`) * `%{python3_version}` (e.g. `3.9`) * `%{python3_version_nodots}` (e.g. `39`) * `%{python3_sitelib}` (e.g. `/usr/lib/python3.9/site-packages`) * `%{python3_sitearch}` (e.g. `/usr/lib64/python3.9/site-packages`) ### Name macros * `%py_set_name PROJECTNAME` (e.g. `%py_set_name Django`) Sets `%{projectname}` to the given name, and `%{distname}` to the canonical version of it. These macros can then be used throughout the `spec` file. See *Naming* for details and *Sample spec file* for examples. (XXX link to sections) * `%{projectname}` (e.g. `Django`) Set by `%py_set_name` to the *project name* of the software. * `%{distname}` (e.g. `django`) Set by `%py_set_name` to the *canonical project name* of the software. ### Shebang macro * `%{py3_shebang_flags}` (`s`) Flags for `%{python3}` to use in shebangs. Redefine this macro to use a different set of flags. See *Shebangs* [XXX link section] for details. ### Convenience macros * `%{pypi_source [PROJECTNAME [VERSION [EXT]]]}` (e.g. `https://.../Django-3.0.5.tar.gz`) Evaluates to the appropriate URL for source archive hosted on PyPI. Accepts up to three optional arguments: 1. The name of the PyPI project. Defaults to `%srcname` if defined, or to `%pypi_name` if defined, or to `%name` (the package name). [XXX default to `%projectname`] 2. The version of the PyPI project. Defaults to `%version` (the package version). 3. The file extension to use. Defaults to `tar.gz`. In most cases it is not necessary to specify any arguments. * `%{python3_platform}` (e.g. `linux-x86_64`) The platform name. Used in some Python build systems. ### Build macros These macros are most useful for packaging Python projects that use the `pyproject.toml` file defined in [PEP 518](https://www.python.org/dev/peps/pep-0518/) and [PEP 517](https://www.python.org/dev/peps/pep-0517/), which specifies the package's build dependencies (including the build system, such as setuptools, flit or poetry). If `pyproject.toml` is not found, the macros automatically fall backs to using `setuptools` with configuration in `setup.cfg`/`setup.py`. A full tutorial and discussion for the macros is available in the macros' [README](https://src.fedoraproject.org/rpms/pyproject-rpm-macros/). * `%pyproject_wheel` Build the package. Commonly, this is the only macro needed in the `%build` section. This macro needs BuildRequires generated by `%pyproject_buildrequires`. * `%pyproject_install` Install the package built by `%pyproject_wheel`. This macro needs BuildRequires generated by `%pyproject_buildrequires`. * `%pyproject_buildrequires` Generate BuildRequires for the package. Used in the `%generate_buildrequires` section of the `spec` file. The macro has these options: * `-r`: Include build-time requirements (commonly needed for `%check`). * `-x EXTRA`: Include dependencies given by the given *extra* (XXX link). [XXX Multiple comma separated values cannot be given yet.] * `-t`: Include dependencies for the default *tox* environment. Implies `-r`. * `-e ENV`: Include dependencies for the given *tox* environment, and save the `ENV` name as `%{toxenv}`. Implies `-r`. Multiple comma separated values can be given, for example: %pyproject_buildrequires -e %{toxenv}-unit,%{toxenv}-integration * `%pyproject_save_files MODNAME …` Generate a list of files corresponding to the given importable modules, and save it as `%{pyproject_files}`. Note that README and licence files are not included. Also, while the macro allows including executable files (using the `+bindir` flag), this feature **MUST NOT** be used in Fedora. The `MODNAME` may be a glob pattern, which should be specific to your package. As mentioned in the *Explicit lists* section, expressions like `%pyproject_save_files *` are not acceptable. * `%{pyproject_files}` Path of the file written by `%pyproject_save_files`, to be used as: %files -n python3-%{distname} -f %{pyproject_files} * `%tox` Run tests using `tox`. A different environment may be specified with `-e`, for example: ``` %check %tox %if %{with integration_test} %tox -e %{toxenv}-integration %endif ``` Flags for the `tox` command can be specified after `--`: %tox -- --parallel 0 Additional arguments for the test runner may be specified after another `--`: %tox -- --parallel 0 -- --verbose tests/* * `%{toxenv}` The *tox* environment used by the `%tox` macro. Can be overridden manually or with `%pyproject_buildrequires -t ENV`. * `%{default_toxenv}` (e.g. `py39`) The system-wide default value of `%{toxenv}`. ### Manual Generation The following macros are available for cases where automatic generation is turned off. They can also be useful for handling files in non-standard locations where the generators don't look. * `%pycached MODNAME.py` Given a Python file, lists the file and the files with its bytecode cache. See *Source files and bytecode cache* for more information. * `%{py_provides python3-MODNAME}` > XXX See *The `%python_provide` macro* for more details. * `%{py_byte_compile INTERPRETER PATH}` > XXX`%{python3_byte_compile PATH}` XXX? Byte-compile a Python file into a `__pycache__/*.pyc`. See [Manual byte compilation](https://docs.fedoraproject.org/en-US/packaging-guidelines/Pyth… in the Appendix for usage. * `%{py_dist_name PROJECTNAME}` Given a *project name* (e.g. `PyYAML`) it will convert it to the canonical format (e.g. `pyyaml`). See *Canonical project name* for more information. * `%{py3_dist PROJECTNAME …}` Given one or more *project names*, it will convert them to the canonical format and evaluate to `python3dist(DISTNAME)`, which is useful when listing dependencies. See *Machine-readable provides* for more information. ### System Settings The following macros can be redefined for special use cases. Most of such cases are unacceptable in Fedora. * `%{__python}` (`/usr/bin/python`) Defining this macro changes the meaning of all “unversioned” Python macros such as `%{python}` or `%{python_sitelib}`. Don’t use these macros without redefining `%{__python}`. * `%{__python3}` (`/usr/bin/python3`) The python 3 interpreter. Redefining this macro changes all the `%{python3...}` macros, e.g. `%{python3_sitelib}`. * `%{python3_pkgversion}` (`3`) Distro-wide Python version, i.e. the `3` in `python3`. Projects that build on top of Fedora may define it to e.g. `3.9` to try allowing multiple Python stacks installable in parallel. ### XXX Conditionals? > XXX How to properly express: "if python_version >= 3.8"? > The current way is comparing integers from %python3_version_nodots, but that will break with 3.10/4.0 comparsion. > Do a Lua macro that splits the versions and compares them? > This looks more general, is there something in RPM we can use? ### Disabling automation The following macros can turn off Python-specific automation. Consider contacting the Python SIG if you need to do this. * `%{?python_disable_dependency_generator}` Disables the automatic dependency generator. See *Automatically generated dependencies* for details. * `%__pythonname_provides` > XXX undefine to disable %python-provides generator * `%__pythondist_requires` > XXX undefine to disable %python3dist generator ### Deprecated Macros The following macros are deprecated. See the *201x-era Python Packaging guidelines* (XXX link) for how some of them were used. * `%py3_build` * `%py3_build_wheel` * `%py3_build_egg` * `%py3_install` * `%py3_install_wheel` * `%py3_install_egg` * `%py3dir` * `%py3_other_build` * `%py3_other_install` * `%python_provide` (without `s` at the end) > XXX: `%pyX_build` breaks shebang when upstream already uses some flags (https://bugzilla.redhat.com/show_bug.cgi?id=1335203) -- should we document this in the old guidelines? ## Reviewer checklist > After the guidelines are done, distill the **MUST**s/**SHOULD**s here. > XXX: Do we need a checklist at all? > How do we keep it updated (which hasn't happened in the last N years)

8 40

Pointers on packaging an application for EL8
by Alex Corcoles 30 Aug '20

30 Aug '20

Hi, I'm dabbling in writing a small Python application (further details below to provide some context). This application: * Has no dependencies other than the Python standard library * Makes only sense to be installed using your distribution's package manager * Would initially support *only* EL8, although later on it could be expanded to support other distributions * Proper testing requires testing for each of the supported Linux distributions for the tool. The application should be packaged and deployed on a system running each supported distro and then certain commands need to be run to verify it is working fine I would like: * Making it easy for developers to hack in the package. They should be able to use any Linux system to develop. I would provide scripts using containers (podman/docker) to run tests and create packages for all supported distros. * Keep the packaging in the main source Git repo * Packaging should be fully automatized in CI/CD. All commits should generate packages for all supported distros (with a timestamped version) and push it to a repo (I would use COPR, ppa, etc.). Latest master should be easily installable using dnf, other branches should be easily installable for testing purposes. * I also like using Poetry for development (e.g. while the application itself would have 0 dependencies, development will require testing/style-checking/etc. and setting that up with venvs is friendlier to devs, instead of requiring the tools to be installed manually). My questions (as a relative noob to RPM packaging- I've created and automated some RPM packaging, but mostly by winging it): * Is any of the above a terrible idea I should reconsider? * Is there any project out there with similar goals doing things "correctly" I could "copy"? * pyproject-rpm-macros looks like something useful for what I want to do, can it be used on to package for EL8? (I only see Fedora branches) Cheers, Álex ------------------------- Context: Software such as the EL8 MariaDB server RPM requires additional steps to be configured, such as running mysql_secure_installation. If you are using MariaDB as the back end database for an application (e.g. NextCloud), you will also typically want to create a database and a user for the application. It is fairly easy to perform those steps manually. Under certain practices (such as container-heavy setups or very cattle-like deployments) it can also be easy to do that. However, if you are using automated configuration management based on idempotency, such as Ansible, Puppet, etc. it is a bit more complex to perform that. To cover that need, typically modules are developed for those configuration management tools which solve those issues. However, a module developed for MariaDB/Puppet is not ideal for Ansible users, etc. A better way would be to develop a command-line tool which can set up MariaDB using declarative configuration (e.g. a file that describes which databases/users/etc. are desired). Then this command-line tool can be easily executed using whatever configuration management tool. This tool could offer a generic way to implement idempotency (e.g. a --check flag whose return code indicates whether changes need to be performed).

2 3

Fwd: F34 Change: Python Upstream Architecture Names (Self-Contained Change)
by Miro Hrončok 27 Aug '20

27 Aug '20

Forwarding here in case you don't follow the devel-announce / devel lists. Please, keep the discussion on the devel list. -------- Forwarded Message -------- Subject: F34 Change: Python Upstream Architecture Names (Self-Contained Change) Date: Thu, 27 Aug 2020 11:13:25 -0400 From: Ben Cotton <bcotton(a)redhat.com> Reply-To: Development discussions related to Fedora <devel(a)lists.fedoraproject.org> To: devel-announce(a)lists.fedoraproject.org CC: Development discussions related to Fedora <devel(a)lists.fedoraproject.org> https://fedoraproject.org/wiki/Changes/Python_Upstream_Architecture_Names == Summary == Use CPython upstream architecture naming in Fedora's Python ecosystem (mostly in filenames) instead of the previously patched Fedora names. For example, have <code>/usr/lib64/python3.9/lib-dynload/array.cpython-39-powerpc64le-linux-gnu.so</code> instead of <code>/usr/lib64/python3.9/lib-dynload/array.cpython-39-ppc64le-linux-gnu.so</code>. This makes packaging of Python itself a tad trickier, but it moves Fedora's Python closer to upstream and solves interoperability problems with ppc64le manylinux wheels. The change has impact only on '''ppc64le''' and '''armv7hl''' (considering the architectures built by koji.fedoraproject.org). Packages assuming the filenames always contain <code>%{_arch}-linux%{_gnu}</code> will need to be adapted. == Owner == * Name: [[User:Churchyard|Miro Hrončok]] * Name: [[User:Lbalhar|Lumír Balhar]] * Email: python-maint(a)redhat.com == Detailed Description == === The Saga === ==== A Long Time Ago in the Fedora Galaxy ==== Many releases ago, when Fedora wasn't being built for power and arm yet, the Python maintainers mapped the Python "platform triplet" or "multiarch name" to <code>%{_arch}-linux%{_gnu}</code>. This worked. For example, on x86_64, this is <code>x86_64-linux-gnu</code> on Fedora and this is consistent with the "platform triplet" used in filenames in upstream. ==== The Phantom Technical Debt ==== Later around the year 2015, as more architectures were added, Python build scripts were patched to use "the Fedora's architecture names": * <code>powerpc</code> was changed to <code>ppc</code> https://src.fedoraproject.org/rpms/python3/c/0efd3d31 * the arm triplet was patched https://src.fedoraproject.org/rpms/python3/c/2f6352e7 * even the mips one https://src.fedoraproject.org/rpms/python3/c/4bc70e0c At the time, that was a reasonable decision: the idea of cross-Linux builds was sci-fi, and Fedora was not trying to stay close to upstream as it is now (we had around 60 patches; today we're down to around 6). ==== Rise of the Manylinux Wheels ==== In the meantime, cross-Linux builds become a thing. The [https://www.python.org/dev/peps/pep-0513/ manylinux1 standard] was created in 2016, allowing to build Python wheels with compiled extension modules on one Linux platform and ship them to many. The first manylinux version only supported x86_64 and i686 and hence it was not impacted by Fedora's patching decisions. The manylinux standard arguably made the upstream Python packaging ecosystem a much nicer place. Installing packages with compiled extension modules was no longer such a pain. One could just run <code>pip install numpy</code> and not worry about a disturbing lack of a Fortran compiler. For that reason, manylinux wheels become widely adopted by the most popular projects. ==== A New Architecture ==== With the third manylinux version -- [https://www.python.org/dev/peps/pep-0599/ manylinux2014] (created in 2019, named after the oldest Linux it supports -- CentOS 7), support for more architectures was introduced: x86_64, i686, aarch64, armv7l, ppc64, ppc64le, s390x. The adaption of new architectures is somehow slow, because the [https://github.com/pypa/manylinux#manylinux2014 official manylinux2014 containers] only currently (August 2020) exist for x86_64, i686, aarch64, ppc64le and s390x. ==== Revenge of the Patches ==== We have discovered [https://github.com/pypa/manylinux/issues/687 a problem with the ppc64le manylinux2014 wheels]: The CentOS 7 manylinux2014 container images ship upstream Python without RHEL/CentOS/EPEL patches. When an extension module is built there, it is named with an upstream named suffix: <code>.cpython-XY(m)-powerpc64le-linux-gnu.so</code>. The wheel is installable on Fedora (with Fedora's patches), but the module won't (even be considered for) import, because Fedora's Pythons expect the extension to be <code>.cpython-XY(m)-ppc64le-linux-gnu.so</code>. In theory, we have the same problem on armv7hl, but there are no manylinux2014 containers available for that platform, so there are no such wheels out there yet (to our knowledge). The same problem also exists the other way around, albeit it's arguably less severe. It is possible to build manylinux wheels on (some version of) Fedora or EL (using the Python from that distribution). However extension modules from such ppc64le wheels won't import on other Linux distributions. ==== The Workaround Awakens ==== To allow importing extension modules from ppc64le manylinux wheels, we have [https://src.fedoraproject.org/rpms/python3.9/pull-request/24 patched Pythons (3.5+) in Fedora] to consider both "Fedora's" and upstream extension suffixes when importing extension modules. This workarounds works well for users installing manylinux wheels on Fedora, but does not solve the problem when building the wheels on Fedora. === The Change === With this change proposal, we plan to [https://src.fedoraproject.org/rpms/python3.9/pull-request/28 switch to use the upstream architecture names] and keep the workaround to preserve backwards compatibility. When we do that the following will happen: # The Python standard library extension module suffixes will change to <code>.cpython-39-powerpc64le-linux-gnu.so</code> and <code>.cpython-39-arm-linux-gnueabihf.so</code>. Python will still import extension modules with the legacy suffixes <code>.cpython-39-ppc64le-linux-gnu.so</code> and <code>.cpython-39-arm-linux-gnu.so</code>. Other architectures not built by koji.fedoraproject.org will also be renamed, see [https://src.fedoraproject.org/rpms/python3.9/pull-request/28#comment-0 the pull request for a complete regex]. This will happen for Python 3.7, 3.8, 3.5, 3.6 and 3.9. # The newly built Python packages with extension modules will also change the suffixes. Packages that assume the platform triplet is always <code>%{_arch}-linux%{_gnu}</code> (e.g. in the <code>%files</code> section) will need to be adapted (see the [[#New_Macros|New Macros]] section). A mix of legacy and upstream suffixes will co-exist and work together. # When safe, we will drop the workaround to support the legacy names. For example, when we initially package Python 3.10, it will be packaged without the workaround. On the other hand, older Python versions might never be able to drop it, because users will carry their own built extension modules from previous releases. === New Macros === For packagers' convenience we will add 2 new Python macros: '''%python3_ext_suffix''' Defined as: %python3_ext_suffix %(%{__python3} -Ic "import sysconfig; print(sysconfig.get_config_var('EXT_SUFFIX'))") Values will be: * <code>.cpython-39-x86_64-linux-gnu.so</code> * <code>.cpython-39-powerpc64le-linux-gnu.so</code> on Fedora 34+ / <code>.cpython-39-ppc64le-linux-gnu.so</code> on older Fedoras * <code>.cpython-39-arm-linux-gnueabihf.so</code> on Fedora 34+ / <code>.cpython-39-arm-linux-gnu.so</code> on older Fedoras * etc. Beware that due to chnages in Python, this macro only works with Python 3. For Python 3.4, the value is <code>.cpython-34m.so</code> on all architectures (similarly on older Python 3 versions, but we don't have such in Fedora or EPEL). '''%python3_platform_triplet''' Defined as: %python3_platform_triplet %(%{__python3} -Ic "import sysconfig; print(sysconfig.get_config_var('MULTIARCH'))") Values will be: * <code>x86_64-linux-gnu</code> * <code>powerpc64le-linux-gnu</code> / <code>ppc64le-linux-gnu</code> * <code>arm-linux-gnueabihf</code> / <code>arm-linux-gnu</code> * etc. Beware that due to changes in Python, this macro only works with Python versions 3.6 and newer. Both macros will be backported to stable Fedoras and EPEL 7+ and will have the corresponding <code>%python_</code> variant. == Feedback == Eighth_Doctor: it's the cleverest written change proposal I've seen in over a decade :) == Benefit to Fedora == Users of ppc64le and armv7hl Fedora (and future RHEL) will have a closer-to-upstream Python experience and will no longer suffer from compatibility issues when they install or build manylinux wheels. The upstream-downstream balance will be restored. == Scope == * Proposal owners: ** Review and merge https://src.fedoraproject.org/rpms/python3.9/pull-request/28 ** Add the new macros to all Fedoras and EPEL 7+ and mention them in the guidelines ** Monitor builds of Python packages with extension modules, fix issues if they arise. ** Backport to Python 3.5-3.8 ** Drop the workaround from Python 3.10 once packaged * Other developers: Mostly nothing, adapt the <code>%files</code> section if needed * Release engineering: a check of an impact with Release Engineering is not needed * Policies and guidelines: the new macros will be documented in the Python packaging guidelines * Trademark approval: not needed for this Change * Alignment with Objectives: no == Upgrade/compatibility impact == No significant user visible upgrade/compatibility problem is anticipated. Filenames will be different, but the old filenames are still supported. Scripts that hardcode filename assumptions might break. == How To Test == On ppc64le, try to install a manylinux wheel and import from it. It should work on any Python ≥ 3.5. E.g.: pip install simple-manylinux-demo python -c 'from dummyextension import extension' On ppc64le, try to build a manylinux wheel and import from it on another Linux. It should work on any Python ≥ 3.5. E.g.: pip wheel . # on some project with extension module auditwheel repair ...whl wormhole send ...whl # or any other way On another ppc64le Linux (such as Debian or openSUSE): wormhole receive ... pip install ...whl python -c 'from ... import ...' You can also build a regular (non-manylinux) wheel on Fedora 33/32 and install and import it on Fedora 34. It should work. The other way around will most likely also work, unless Fedora 34 has an incompatible glibc update. == User Experience == Users of ppc64le and armv7hl Fedora (and future RHEL) will have a closer-to-upstream Python experience and will no longer suffer from compatibility issues when they install or build manylinux wheels. == Dependencies == No known dependencies. May the force be with us. == Contingency Plan == * Contingency mechanism: Revert the change and rebuild all affected packages. * Contingency deadline: Soft before the mass rebuild, so we could leverage it for the revert-rebuilds. Hard before the beta freeze. * Blocks release? No * Blocks product? No == Documentation == This page is the documentation.

1 0

Python spec template violates guidelines?
by Michel Alexandre Salim 26 Aug '20

26 Aug '20

Per https://pagure.io/packaging-committee/issue/782, "Forbid %{pythonX_site(lib|arch)}/* in %files" Python packages should not blindly glob contents of the sitelib/sitearch directories. This makes sense, in fact, I just got bit by this packaging python- sphinx-hoverxref (https://bugzilla.redhat.com/show_bug.cgi?id=1872508) -- for some reason a `tests` directory also get copied to buildroot and thus get packaged. `rpmdev-newspec python-foo` still produces a spec with globbing though. This is just a matter of omission, I presume? I can put up a PR fixing this. Cheers, -- Michel Alexandre Salim profile: https://keybase.io/michel_slm chat via email: https://delta.chat/ GPG key: 5DCE 2E7E 9C3B 1CFF D335 C1D7 8B22 9D2F 7CCC 04F2

3 5

[HEADS UP] Sphinx updated to 3.2
by Miro Hrončok 21 Aug '20

21 Aug '20

I've updated python-sphinx to 3.2.1 in rawhide and f33. It only breaks one package (python-graphql-core) -- all other packages either build fine with new Sphinx or FTBFS for unrelated reasons. I'll have a look at python-graphql-core to see how it can be fixed. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

[RELEASE] Python 3.9.0rc1 is now available
by Miro Hrončok 12 Aug '20

12 Aug '20

This has been updated in Fedora rawhide. The Fedora 33 update waits for the successful compose: https://bodhi.fedoraproject.org/updates/FEDORA-2020-3d34a62850 The updates appear to be ABI compatible, so there is no need for a targeted rebuild. Fedora 32 and 31 updates are pending: https://bodhi.fedoraproject.org/updates/FEDORA-2020-ce74609c79 https://bodhi.fedoraproject.org/updates/FEDORA-2020-85cca90ce8 -------- Forwarded Message -------- Subject: [Python-Dev] [RELEASE] Python 3.9.0rc1 is now available Date: Tue, 11 Aug 2020 23:48:59 +0200 From: Łukasz Langa <lukasz(a)langa.pl> To: python-committers(a)python.org, python-dev(a)python.org, python-list(a)python.org, python-announce(a)python.org Python 3.9.0 is*almost*ready. This release,*3.9.0rc1*, is the penultimate release preview. You can get it here: https://www.python.org/downloads/release/python-390rc1/ Entering the release candidate phase, only reviewed code changes which are clear bug fixes are allowed between this release candidate and the final release. The second candidate and the last planned release preview is currently planned for 2020-09-14. Please keep in mind that this is a preview release and its use is*not*recommended for production environments. Calls to action Core developers: all eyes on the docs now * Are all*your*changes properly documented? * Did you notice*other*changes you know of to have insufficient documentation? Community members We*strongly encourage*maintainers of third-party Python projects to prepare their projects for 3.9 compatibility during this phase. As always, report any issues tothe Python bug tracker <https://bugs.python.org/>. Installer news This is the first version of Python to default to the 64-bit installer on Windows. The installer now also actively disallows installation on Windows 7. Python 3.9 is incompatible with this unsupported version of Windows. Major new features of the 3.9 series, compared to 3.8 Some of the new major new features and changes in Python 3.9 are: * PEP 584 <https://www.python.org/dev/peps/pep-0584/>, Union Operators in|dict| * PEP 585 <https://www.python.org/dev/peps/pep-0585/>, Type Hinting Generics In Standard Collections * PEP 593 <https://www.python.org/dev/peps/pep-0593/>, Flexible function and variable annotations * PEP 602 <https://www.python.org/dev/peps/pep-0602/>, Python adopts a stable annual release cadence * PEP 615 <https://www.python.org/dev/peps/pep-0615/>, Support for the IANA Time Zone Database in the Standard Library * PEP 616 <https://www.python.org/dev/peps/pep-0616/>, String methods to remove prefixes and suffixes * PEP 617 <https://www.python.org/dev/peps/pep-0617/>, New PEG parser for CPython * BPO 38379 <https://bugs.python.org/issue38379>, garbage collection does not block on resurrected objects; * BPO 38692 <https://bugs.python.org/issue38692>, os.pidfd_open added that allows process management without races and signals; * BPO 39926 <https://bugs.python.org/issue39926>, Unicode support updated to version 13.0.0; * BPO 1635741 <https://bugs.python.org/issue1635741>, when Python is initialized multiple times in the same process, it does not leak memory anymore; * A number of Python builtins (range, tuple, set, frozenset, list, dict) are now sped up usingPEP 590 <https://www.python.org/dev/peps/pep-0590>vectorcall; * A number of Python modules (_abc, audioop, _bz2, _codecs, _contextvars, _crypt, _functools, _json, _locale, operator, resource, time, _weakref) now use multiphase initialization as defined byPEP 489 <https://www.python.org/dev/peps/pep-0489/>; * A number of standard library modules (audioop, ast, grp, _hashlib, pwd, _posixsubprocess, random, select, struct, termios, zlib) are now using the stable ABI defined byPEP 384 <https://www.python.org/dev/peps/pep-0384/>.

1 0

[HEADS UP] pyproject-rpm-macros %pyproject_buildrequires -t / %tox will now fail without a tox config
by Miro Hrončok 11 Aug '20

11 Aug '20

Hello, tox was updated in Feodra 33+ to 3.19. I've managed to get a backwards incompatible change of behavior into that release. Previously, when there was no tox.ini, but there was setup.cfg, tox proceeded without checking the content of setup.cfg. This lead to a dangerous behvior: `tox` (and `%tox`) would exit with 0 if there were no tox-configured tests. This is now changed: If setup.cfg is present, but contains no tox-related configuration, tox will fail: ERROR: tox config file (either pyproject.toml, tox.ini, setup.cfg) not found This manifests like this with `%pyproject_buildrequires -t`: ... Handling tox-current-env >= 0.0.2 from tox itself Requirement satisfied: tox-current-env >= 0.0.2 (installed: tox-current-env 0.0.2) ERROR: tox config file (either pyproject.toml, tox.ini, setup.cfg) not found Traceback (most recent call last): File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 322, in main generate_requires( File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 269, in generate_requires generate_tox_requirements(toxenv, requirements) File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 231, in generate_tox_requirements r.check_returncode() File "/usr/lib64/python3.9/subprocess.py", line 456, in check_returncode raise CalledProcessError(self.returncode, self.args, self.stdout, subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'tox', '--print-deps-to-file', '/tmp/tmpgvu_6190', '-qre', 'py39']' returned non-zero exit status 1. error: Bad exit status from /var/tmp/rpm-tmp.n92yaC (%generate_buildrequires) The mitigation is to not use tox if upstream doesn't have tox configuration. This change broke python-latexcodec and python-pybtex-docutils: https://koschei.fedoraproject.org/build/8698931 https://koschei.fedoraproject.org/build/8698932 I'll fix them. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

[python-engineio] test failures on F33 koji
by Mukundan Ragavan 09 Aug '20

09 Aug '20

I am trying to fix FTBFS and upgrade engineio to 3.13 (current version in rawhide is 3.11). Some tests keep failing on koji [0] but not when I build locally using mock. This is what I see in build log - _ ERROR collecting tests/common/test_async_eventlet.py _____________ /usr/lib/python3.9/site-packages/dns/resolver.py:743: in read_resolv_conf f = stack.enter_context(open(f)) E FileNotFoundError: [Errno 2] No such file or directory: '/etc/resolv.conf' During handling of the above exception, another exception occurred: <snip> E dns.resolver.NoResolverConfiguration: Resolver configuration could not be read or specified no nameservers. Pointers to fix this will be greatly appreciated. Thanks! [0] https://koji.fedoraproject.org/koji/taskinfo?taskID=48843424

2 1

[HEADS UP] pytest update to 6.0.x is approaching
by Miro Hrončok 08 Aug '20

08 Aug '20

Hello, recently we have updated pytest to 5, but version 6 is out and I feel confident that the update is safe, hence I plan to merge and ship pytest 6.0.1 in Fedora 33+ in couple days: https://src.fedoraproject.org/rpms/pytest/pull-request/17 I've analyzed the build failures with pytest 6.0.0rc1: https://src.fedoraproject.org/rpms/pytest/pull-request/17#comment-50314 And later with 6.0.1: https://src.fedoraproject.org/rpms/pytest/pull-request/17#comment-52937 Most of the failures are caused by PytestDeprecationWarning treated as error. If your package FTBFS, you should see the reason in one of the comments there ^ As a short term mitigation, you can disable this treatment, see https://docs.pytest.org/en/stable/changelog.html#breaking-changes > PytestDeprecationWarning are now errors by default. > > Following our plan to remove deprecated features with as little disruption > as possible, all warnings of type PytestDeprecationWarning now generate > errors instead of warning messages. > > The affected features will be effectively removed in pytest 6.1, so please > consult the Deprecations and Removals section in the docs for directions on > how to update existing code. > > https://docs.pytest.org/en/latest/deprecations.html > > In the pytest 6.0.X series, it is possible to change the errors back into > warnings as a stopgap measure by adding this to your pytest.ini file: > > [pytest] > filterwarnings = > ignore::pytest.PytestDeprecationWarning > > But this will stop working when pytest 6.1 is released. > > If you have concerns about the removal of a specific feature, please add > a comment to https://github.com/pytest-dev/pytest/issues/5584 And if nothing else works, you can still use pytest 4, see: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org… -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

PSA: Python 2.6 and 3.4 retired on Fedora 33+
by Miro Hrončok 06 Aug '20

06 Aug '20

Hey Pythonistas, I've just retired Python 2.6 and Python 3.4 from rawhide (Fedora 33+). - https://fedoraproject.org/wiki/Changes/RetirePython26 - https://fedoraproject.org/wiki/Changes/RetirePython34 If you use any of the mentioned Python versions on Fedora rawhide for CI purposes, you might need to update the CI configuration. The Python versions remain available on Fedora 31 and 32 until they go EOL in November 2020 / May 2021. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel August 2020