python-devel July 2021

python-devel@lists.fedoraproject.org

11 participants
14 discussions

package request: PyWaffle
by Matthew Miller 20 Dec '21

20 Dec '21

Hi! I'm using this https://github.com/gyli/PyWaffle for some visualizations for Fedora Project stats. I'm kind of out of the loop on the state of the art of python packaging, and wondered if some kind Python SIG person would like to take it on for me.

4 12

Macro to smoke-test-import a Python module in %check
by Miro Hrončok 04 Aug '21

04 Aug '21

Hello Python RPM packagers, based on some discussion in the "F35 Change: Python Packaging Guidelines overhaul (System-Wide Change proposal)" thread [0], I've drafted a macro that can help to test-import a Python module in %check when no other tests exist or are when they cannot be executed during build [1]. The semantics is quite simple: %check %py3_check_import mymodule mymodule.submodule When all listed modules are successfully imported, "nothing happens", when at lest one fails to import, the entire build fails. The above line is translated very-roughly to `python3 -c 'import mymodule, mymodule.submodule'` (see the implementation [0] for more accurate representation). Given the Python semantics, it is possible to use commas as module separators as well (but no parentheses). The %buildroot's %pythn3_site{lib,arch} is added to PYTHONPATH. The runtime dependencies are obviously needed for this to work, so they need to be manually BuildRequired or even better, generated in %generate_buildrequires via `%pyproject_buildrequires -r` to use this macro. The macro can be used repeatedly when importing multiple modules at once is undesired (e.g. when only one module can be imported from the same Python interpreter): %check %py3_check_import mymodule.either %py3_check_import mymodule.or Before I merge this and backport to all Fedoras and EPELs, I'd like to know: - Do you have better ideas for the macro name? - Do you have better ideas for the macro semantics? [0] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org… [1] https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/99 -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

4 6

Package with deps not detected
by Robert-André Mauchin 31 Jul '21

31 Jul '21

Hi, I'm trying to convert one of my Python package to the new guidelines. With the former guidelines, it had the following BR: BuildRequires: python3-devel BuildRequires: python3dist(click) BuildRequires: python3dist(pytest) BuildRequires: python3dist(setuptools) However when converting to the new guidelines, I'm encountering the following the following error: Handling wheel from get_requires_for_build_wheel Requirement satisfied: wheel (installed: wheel 0.36.2) Handling tox-current-env >= 0.0.6 from tox itself Requirement satisfied: tox-current-env >= 0.0.6 (installed: tox-current-env 0.0.6) ERROR: tox config file (either pyproject.toml, tox.ini, setup.cfg) not found There is a setup.cfg, but there is no dependency specified anywhere in it. The deps are specified in the setup.py but don't seem to be detected: https://github.com/click-contrib/click-default-group/blob/master/setup.py Is there any workaround for this situation? Should I write my own pyproject.toml in %prep? Best regards, Robert-André

2 1

Review swap request: python-xds-protos
by Ben Beasley 23 Jul '21

23 Jul '21

I’d like to get python-xds-protos (https://bugzilla.redhat.com/show_bug.cgi?id=1980041) reviewed so I can update grpc to version 1.39.0 in Rawhide in time for Fedora 35. This is a new dependency for grpc—a weirdly circular one that’s ultimately generated from sources inside grpc, but is separately versioned and separately distributed (generated sources only) on PyPI, so I am treating it as a separate package. I’m happy to review another package, or a handful of simple packages, in exchange. I’m particularly accustomed to reviewing Python, C, C++, NodeJS, JavaScript, and web asset packages, but I’m happy to jump in and review something else as long as reading the relevant domain-specific packaging guidelines is enough to get me through the review.

2 1

New release: pyproject-rpm-macros 0-46, new features and breaking changes
by Miro Hrončok 23 Jul '21

23 Jul '21

Hello, we have just released pyproject-rpm-macros 0-46 with new features and breaking changes (2 affected Fedora packages, python-platformdirs and python-shellingham). F35: https://bodhi.fedoraproject.org/updates/FEDORA-2021-a33e4b0558 F34: https://bodhi.fedoraproject.org/updates/FEDORA-2021-fcf5e2adc6 F33: https://bodhi.fedoraproject.org/updates/FEDORA-2021-3d06d86fe9 We aim to release version 1 before Fedora 35 is released: That should remove the "provisional" status of the macros: we are extensively probing the design decisions and plan to maintain a stable interface and behavior after that. tl;dr version: %pyproject_buildrequires now supports x.* versions. In %pyproject_buildrequires, invalid requirements now *fail* the build. %pyproject_buildrequires fallbacks to setuptools only if setup.py exists. The requirements files parser is now more robust. %pyproject_save_files now escapes weird characters. %pyproject_save_files marks license files with %license. The %{_pyproject_ghost_distinfo} and %{_pyproject_record} macros are private. Details below: --------------- %pyproject_buildrequires now supports x.* versions. When a project requires e.g. "pkg==6.*", %pyproject_buildrequires now correctly handles that by requiring "pytohn3dist(pkg) >= 6 with pytohn3dist(pkg) < 7". --------------- In %pyproject_buildrequires, invalid requirements now *fail* the build. Previously, they were skipped with a warning. Only one package in Fedora is affected and a PR exists (python-platformdirs#1). Failing the build for dependencies that cannot be processed via Python tools (such as pip) is harmless: The package would eventually fail to build later in the process anyway. However, some dependency declarations work with pip while we cannot process them via the macros. From the README: > When a dependency is specified via an URL or local path, for example as: > > https://github.com/ActiveState/appdirs/archive/8eacfa312d77aba28d483fbfb6f6… > /some/path/foo-1.2.3.tar.gz > git+https://github.com/sphinx-doc/sphinx.git@96dbe5e3 > > The %pyproject_buildrequires macro is unable to convert it to an appropriate RPM requirement and will fail. If the URL contains the packageName @ prefix as specified in PEP 508, the requirement will be generated without a version constraint: > > appdirs@https://github.com/ActiveState/appdirs/archive/8eacfa312d77aba28d483fbfb6f6fc54099622be.zip > foo@file:///some/path/foo-1.2.3.tar.gz > > Will be converted to: > > python3dist(appdirs) > python3dist(foo) > > Alternatively, when an URL requirement parsed from a text file given as positional argument to %pyproject_buildrequires contains the #egg=packageName fragment, as documented in pip's documentation: > > git+https://github.com/sphinx-doc/sphinx.git@96dbe5e3#egg=sphinx > > The requirements will be converted to package names without versions, e.g.: > > python3dist(sphinx) > > However upstreams usually only use direct URLs for their requirements as workarounds, so be prepared for problems. ------------ %pyproject_buildrequires fallbacks to setuptools only if setup.py exists. In line with the recent development of pip (and build), when the build backend is not specified (e.g. pyproject.toml is missing or does not specify it) the setuptools.build_meta.__legacy__ build backend is only used if setup.py exists. Projects without setup.py that intent to use setuptools need to explicitly declare it in pyoproject.toml or add a setup.py stub, see e.g. python-shellingham#5, the only affected Fedora package. --------------- The requirements files parser is now more robust. The %pyproject_buildrequires macro accepts requirements files as optional position arguments. With this release, the parser for the files (also used for tox dependencies) is more robust and should process comments, escaped newlines and environment variables better than ever. However, the parser is not (yet) 100% compatible with `pip install -r <file>` behavior (and might as well never be). If you encounter incompatibilities, please let us know. --------------- %pyproject_save_files now escapes weird characters. When a path contains spaces (incl. newlines), quotes (") or percentage signs (%), it is now correctly escaped by %pyproject_save_files. Note that due to the limitations of RPM, we are unable to represent a path with both spaces and quotes. An error is raised when that happens, instead of producing wrong results. --------------- %pyproject_save_files marks license files with %license. PEP 639 (still a draft) introduces a new metadata field for license files (the License-File field). Setuptools 57+ generate the field from the license_file(s) setup options. %pyproject_save_files newly recognizes this field and marks the listed files with the %license tag. Other build backends might follow later. If your package supports this, you no longer need to list the license file manually. I recommend using `rpm -ql --licensefiles` to verify this before removing the manually listed LICENSE file. Note that in Fedora 33 and 34 we don't have setuptools 57+, so I only recommend doing this for packages that don't do merges to stable Fedoras. --------------- The %{_pyproject_ghost_distinfo} and %{_pyproject_record} macros now start with an underscore to indicate they are "private": You should not use them in the spec file unless working around some bug in pyproject-rpm-macros. If you ever need to use the macros (or their value) in spec, consider contacting us first. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

Bring your Python package to Nest and we'll help you convert it to the new Python macros
by Miro Hrončok 23 Jul '21

23 Jul '21

Hello Fedorans and especially Pythonistas, The Python Maintenance team from Red Hat just submitted a proposal [1] for a hackfest on Nest with Fedora [2]. With the new Python packaging guidelines [3], changes of your spec files are necessary to benefit from all the new fancy stuff, such as automatically generated build (incl. test) requires, packaging packages that use poetry or flit, using tox in %check or semi-automatically generating the %files section. On this hackfest, we plan to be available for you for half a day either on Thursday, August 5 or Friday, August 6. You hop on or hop off to Hopin as needed, ask us questions in the chat or via audio/video and we'll try our best to help you get started with the new tools and macros. Please, let the Nest organizers know in the ticket [1] if you are interested, so they can make a better decision whether to include this hackfest or not. See you on Nest either way! [1] https://pagure.io/flock/issue/346 [2] https://flocktofedora.org/ [3] https://fedoraproject.org/wiki/Changes/PythonPackagingGuidelines202x -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

An unsuccessful case study: Using pyproject-rpm-macros with PyQt-builder and sip 5
by Miro Hrončok 22 Jul '21

22 Jul '21

Hello, since Scott is working on update to sip 5 (thank you!) I've figured this is a good time check if we can use the PyQt-builder Python build backend with our pyproject-rpm-macros. tl;dr It is broken in various ways and I am unsure where to attempt to fix things. ------------- This was my case study: I've used the python-pyqtchart package as a test subject, as it is fairly simple and has an open pull request to port it to sip5 [1]. I've pretty much reconstructed the spec file as I would normally do: %generate_buildrequires %pyproject_buildrequires %build %pyproject_wheel %install %pyproject_install To ease things up, I've kept the non-Python-packages BuilRequires intact, so the package still BuildRequires python3-qt5-devel, qt5-qtcharts-devel and qt5-qtbase-private-devel. I've removed the BuildRequires on python3dist(sip) and python3dist(pyqt-builder) as I've expected those to be generated. The simplified pyproject.toml file looks like this: [build-system] requires = ["sip >=5.3, <7", "PyQt-builder >=1.6, <2"] build-backend = "sipbuild.api" [tool.sip.metadata] name = "PyQtChart" version = "5.15.2" ... requires-dist = "PyQt5 (>=5.15)" [tool.sip] project-factory = "pyqtbuild:PyQtProject" [tool.sip.project] tag-prefix = "QtChart" [tool.sip.bindings.QtChart] qmake-QT = ["charts"] I've also tried with version 5.15.4, but not much is different. I've used the copr with sip5 [2] to build the package. ------------- %pyproject_buildrequires works and generates requirements on python3dist(sip) and python3dist(pyqt-builder) \o/ %pyproject_buildrequires -r doesn't work. It says: ValueError: build backend cannot provide build metadata (incl. runtime requirements) before buld Except for the typo [3], this is the expected behavior for a build backend that does not support the prepare_metadata_for_build_wheel hook. Bummer, but fair. We can RFE PyQt-builder to add that hook. ------------- %pyproject_wheel fails very badly. The traceback is unparseable by humans. It ends with: pip._internal.exceptions.InstallationSubprocessError: Command errored out with exit status 1: ... Check the logs for full command output. I have no idea where are the logs. We might want to improve this in pip. After some digging, I've realized the problem is that PyQt-builder calls "qmake" and that command was not available. I've looked into the code and the "qmake" name is hardcoded, but searched on $PATH. The "original" non-pyproject spec has: sip-build ... --qmake="%{_qt5_qmake}" But there is no (known) way to pass such options to the backend. I was able to make it work by exporting PATH="%{_qt5_bindir}:$PATH". I was wondering whether the Fedora's sip5 (or PyQt-builder) package might change the default to make this work, but maybe there is no reasonable default here, as PyQt-builder is Qt version agnostic. Hence I don't know how to make it work out of the box. Not sure where to improve this experience. Anyway, workaround exists. ------------- During %pyproject_wheel, "make" is called internally by the backend. The build is not parallelized. I was able to workaround this by using MAKEFLAGS='%{?_smp_mflags}'. We might want to set this environment variable to '%{_make_output_sync} %{?_smp_mflags} %{_make_verbose}' in the %pyproject_wheel macro, so any backed that utilizes make gets the default arguments Fedora packages should use. Anyway, this is not a blocker to build the package. ------------- Processing files: python-pyqtchart-debugsource-5.15.4-1.fc35.x86_64 RPM build errors: error: Empty %files file /builddir/build/BUILD/PyQtChart-5.15.4 /debugsourcefiles.list This is a problem I get next. It likely means the build flags are not propagated. Our macros set the CFLAGS and LDFLAGS environment variables, but the build does not respect them. I've tried exporting CXXFLAGS as well, but no dice. Apparently, the build backend does not pass the variables to the build. I've tried a workaround like this: export MAKEFLAGS='%{?_smp_mflags} CPPFLAGS="%{build_cxxflags}"' But it makes the internal make call just dump the targets instead of actually building, so I guess I am doing it wrong. Either way, I think we need to fix PyQt-builder/sip to respect the flags from the environment variables. Funnily enbough, when I break make like this, it creates the wheel anyway, but with no dist-info metadata in it. I guess if users break make by setting wrong MAKEFLAGS, the wheel should not be created at all :/ To move forward, I've used %global debug_package %{nil}, but I consider not using the Fedora's flag a blocker before we recommend the macros for packages that use PyQt-builder. ------------- The next problem I got was: error: File not found: /builddir/build/BUILDROOT/python-pyqtchart-5.15.4-1.fc35.x86_64/usr/share/qt5/qsci/api/python/PyQtChart.api The "original" non-pyproject specfile had: sip-build ... --api-dir=%{_qt5_datadir}/qsci/api/python And as said previously, I don't know a way to pass such options to the backend. I am not skilled enough in Qt and SIP, so I have no idea if we need to ship the .api file or not. But if we need or want to, we should be able to. When the option is not used, the api file is not installed anywhere. I was unable to figure this out yet. ------------- That's it. I don't really know where to fix things, and I'd appreciate some pointers. This is not a very high priority for me right now, but once the new Python guidelines [4] are in place, I think we should make this work, possibly with some reasonable worarounds. [1] https://src.fedoraproject.org/rpms/python-pyqtchart/pull-request/6 [2] https://copr.fedorainfracloud.org/coprs/swt2c/pyqt5-sip5/ [3] https://src.fedoraproject.org/rpms/pyproject-rpm-macros/pull-request/167 [4] https://fedoraproject.org/wiki/Changes/PythonPackagingGuidelines202x -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

4 4

python-testtools 2.5.0 built for Rawhide
by Michel Alexandre Salim 16 Jul '21

16 Jul '21

I've upgraded python-testtools in Rawhide to 2.5.0 The upstream changelog contains some breaking changes, so I'm not planning to upgrade F34 (let alone F33) unless there's a need for it. It's mostly removing deprecated functionality though: https://github.com/testing-cabal/testtools/releases/tag/2.5.0 As a bonus, all patches have been dropped, this release builds cleanly on Python 3.10 (despite the release notes only saying it's tested on 3.9) and all tests pass. -- Michel Alexandre Salim profile: https://keyoxide.org/michel@michel-slm.name

1 0

Major update of Python packages next week: Flask 2.0, Werkzeug 2.0, Jinja 3.0, ItsDangerous 2.0, and MarkupSafe 2.0
by Miro Hrončok 15 Jul '21

15 Jul '21

Hello Fedorans and especially Pythonistas, we plan to marge and ship the following upgrades in Rawhide next week: https://src.fedoraproject.org/rpms/python-flask/pull-request/8 https://src.fedoraproject.org/rpms/python-werkzeug/pull-request/8 https://src.fedoraproject.org/rpms/python-jinja2/pull-request/9 https://src.fedoraproject.org/rpms/python-itsdangerous/pull-request/2 https://src.fedoraproject.org/rpms/python-markupsafe/pull-request/6 The packages come from the Pallets Projects stack and you can read all about the upgrade in https://palletsprojects.com/blog/flask-2-0-released/ (Note that Click was alrady upgraded to 8.0 a while ago.) This breaks some packages, but not a large stack: https://src.fedoraproject.org/rpms/python-flask/pull-request/8#comment-0 https://src.fedoraproject.org/rpms/python-flask/pull-request/8#comment-80471 Don't hesitate to ask for help if this breaks your package. There should always be a path forward. If not, compat packages can be introduced as a last resort (and I can help with the initial packging of those if needed). Thanks for your attention, -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 1

pep8 was replaced by what ?
by Sérgio Basto 15 Jul '21

15 Jul '21

Hi, https://src.fedoraproject.org/rpms/python-pep8 was retired 2 years ago , have we any replacement ? for pep8 for example: pep8 --config pep8.conf fedora-review/java_guidelines.py Thank you, -- Sérgio M. B.

1 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel July 2021