On 1/30/20 8:39 AM, Miro Hrončok wrote:

On 30. 01. 20 16:32, Orion Poplawski wrote:

...

Folks -

Looks like RHEL 8.2 will have python 3.8 in addition to python 3.6.  From the 8.2 beta:

Red Hat Enterprise Linux 8 for x86_64 - AppStream Beta (RPMs) Name           Stream          Profiles               Summary python27       2.7 [d][e]      common [d]             Python programming language, version 2.7 python36       3.6 [d][e]      build, common [d]      Python programming language, version 3.6 python38       3.8 [d][e]      build, common [d]      Python programming language, version 3.8

Currently, %python_pkgversion is set to 3 in /usr/lib/rpm/macros.d/macros.python-srpm from python-srpm-macros.

python3-devel is still provided only by python36-devel, so presumably all EPEL8 python packages will continue to be built against python 3.6. But I imagine that people will soon be asking for python 3.8 versions of EPEL packages.  How can we provide those?  Does this have to be done in some modular fashion - which seems to come back to the discussion of whether or not every package has to become its own module or whether to group them together somehow.  Or since both python modules are "default" modules and we can install both python36-devel and python38-devel at the same time, perhaps we can define the python3_other* macros again for python38 and just go that way?

Thoughts?

The idea is that the versions fo stuff we build in RHEL for different python versions is different and I'd like to keep that idea in EPEL as well. Building a python38-foo package from it's own spec should work as follows:

BR python38-rpm-macros BR python38-devel BR python38-bar etc...

Regular specfile follows.

You can even have a single specfile that build for different Python version based on local override of %python_pkgversion in the buildroot.

Building both versions from single spec file---single build would require a new set of macros, yes (or hardcoding stuff). However I'd not call them python3_other* unless you want to end up with python3_other_other* the next time this happens.

This along with some more info from rhel 8.2 beta yields more questions for me. While I do agree that building the python38 packages from separate specs probably is the best route (biggest reason being it allows for updating of individual module versions) I'm hoping we can brainstorm ways to make this less onerous on individual packagers.

Looks like python38-devel is a module in RHEL 8.2 that provides a bunch of stuff needed for building modules (python38-devel, python38-pytest, etc):

Red Hat CodeReady Linux Builder Beta for RHEL 8 x86_64 (RPMs) Name Stream Profiles Summary python38-devel 3.8 [e] Python programming language, version 3.8

Since this isn't a default module, does this again mean the python38 packages in EPEL 8 need to be modules as well? Or can we provide a buildroot that enables this module?

My current pie-in-the-sky idea is that we could do:

- create a epel8-python38 branch for all of the python-* packages with epel8 branches. - epel8-python38 buildroot would enable python38-devel and install python38-rpm-macros and define python3_pkgversion to 38. - This would imply an epel8-python38 repo. It's possible that some packages from epel8-python38 wouldn't be able to be installed unless the python38-devel module was enabled. - This might lead to an explosion of repos if we try to work around other modules in RHEL8 like this (php 7.3, perl, ruby 2.6)

Otherwise I think we will need python38 packages in EPEL8 to be modular. If the module route is the way to go, I really do think that we should try to not have every package be its own module, though the other extreme (all packages in one module) probably is untenable as well. In any case, this will require a lot more coordination among packagers (not necessarily a bad thing).

Thoughts? Plans?

On 2/13/20 02:21, Tomas Orsava wrote:

On 2/13/20 5:18 AM, Orion Poplawski wrote:

...

On 1/30/20 8:39 AM, Miro Hrončok wrote:

...

On 30. 01. 20 16:32, Orion Poplawski wrote:

...

Folks -

Looks like RHEL 8.2 will have python 3.8 in addition to python 3.6.  From the 8.2 beta:

Red Hat Enterprise Linux 8 for x86_64 - AppStream Beta (RPMs) Name           Stream          Profiles               Summary python27       2.7 [d][e]      common [d]             Python programming language, version 2.7 python36       3.6 [d][e]      build, common [d]      Python programming language, version 3.6 python38       3.8 [d][e]      build, common [d]      Python programming language, version 3.8

Currently, %python_pkgversion is set to 3 in /usr/lib/rpm/macros.d/macros.python-srpm from python-srpm-macros.

python3-devel is still provided only by python36-devel, so presumably all EPEL8 python packages will continue to be built against python 3.6.  But I imagine that people will soon be asking for python 3.8 versions of EPEL packages.  How can we provide those?  Does this have to be done in some modular fashion - which seems to come back to the discussion of whether or not every package has to become its own module or whether to group them together somehow.  Or since both python modules are "default" modules and we can install both python36-devel and python38-devel at the same time, perhaps we can define the python3_other* macros again for python38 and just go that way?

Thoughts?

The idea is that the versions fo stuff we build in RHEL for different python versions is different and I'd like to keep that idea in EPEL as well. Building a python38-foo package from it's own spec should work as follows:

BR python38-rpm-macros BR python38-devel BR python38-bar etc...

Regular specfile follows.

You can even have a single specfile that build for different Python version based on local override of %python_pkgversion in the buildroot.

Building both versions from single spec file---single build would require a new set of macros, yes (or hardcoding stuff). However I'd not call them python3_other* unless you want to end up with python3_other_other* the next time this happens.

This along with some more info from rhel 8.2 beta yields more questions for me.  While I do agree that building the python38 packages from separate specs probably is the best route (biggest reason being it allows for updating of individual module versions) I'm hoping we can brainstorm ways to make this less onerous on individual packagers.

Looks like python38-devel is a module in RHEL 8.2 that provides a bunch of stuff needed for building modules (python38-devel, python38-pytest, etc):

Hi! Just a little correction, despite the name suggesting otherwise, the "python38-devel" package is not in the python38-devel module, but in the python38 module itself, which has a default stream.

The python38-devel module contains only python38-pytest and it's dependencies (pyparsing, atomicwrites, attrs, packaging, py, more-itertools, pluggy, wcwidth). And the reason it's not default is not an intention but a current technical limitation of the automatically generated "-devel" modules shipped in the CRB.

...

Red Hat CodeReady Linux Builder Beta for RHEL 8 x86_64 (RPMs) Name               Stream         Profiles              Summary python38-devel     3.8 [e]                              Python programming language, version 3.8

Since this isn't a default module, does this again mean the python38 packages in EPEL 8 need to be modules as well?  Or can we provide a buildroot that enables this module?

My current pie-in-the-sky idea is that we could do:

• create a epel8-python38 branch for all of the python-* packages with epel8

branches.

• epel8-python38 buildroot would enable python38-devel and install

python38-rpm-macros and define python3_pkgversion to 38.

• This would imply an epel8-python38 repo.  It's possible that some packages

from epel8-python38 wouldn't be able to be installed unless the python38-devel module was enabled.

• This might lead to an explosion of repos if we try to work around other

modules in RHEL8 like this (php 7.3, perl, ruby 2.6)

Otherwise I think we will need python38 packages in EPEL8 to be modular.  If the module route is the way to go, I really do think that we should try to not have every package be its own module, though the other extreme (all packages in one module) probably is untenable as well.  In any case, this will require a lot more coordination among packagers (not necessarily a bad thing).

Thoughts?  Plans?

Since pytest is usually a build dependency, then theoretically you could have non-modular python38 packages in EPEL, since all the important stuff is in python38 with a default stream.

Tomas

Except that we are going to need python38-pytest, etc. in the EPEL8 buildroot if we are going to build (most of) the packages in the first place. That's the problem I'm running into now trying to build python38-jmespath:

https://koji.fedoraproject.org/koji/taskinfo?taskID=83570866 DEBUG util.py:444: No matching package to install: 'python38-pytest'

So I'm back to my original questions:

* Do we just make EPEL python38- packages as modules or try to hack up some kind of build system support for it? * If modules, every package a module or one (or a few) modules?

On Wed, 2 Mar 2022 at 17:00, Orion Poplawski orion@nwra.com wrote:

On 3/2/22 14:14, epel-devel@lists.fedoraproject.org wrote:

...

Except that we are going to need python38-pytest, etc. in the EPEL8

buildroot

...

if we are going to build (most of) the packages in the first place.

That's

...

the problem I'm running into now trying to build python38-jmespath:

https://koji.fedoraproject.org/koji/taskinfo?taskID=83570866 DEBUG util.py:444: No matching package to install: 'python38-pytest'

So I'm back to my original questions:

• Do we just make EPEL python38- packages as modules or try to hack up

some

...

kind of build system support for it?

• If modules, every package a module or one (or a few) modules?

I have been told that the lack of python38-pytest is an issue with the buildroot generation and so have filed:

https://pagure.io/releng/issue/10679

Hopefully this will make everything simpler as nothing has to be a module or special in any way.

So this problem seems to be something with koji and that set of packages in that sub-directory. I tested this with the following:

1. koji mock-config --tag epel8-next-build --arch x86_64 > e8-next.cfg 2. cp e8-next.cfg e8-next-infra.cfg 3. change ` https://kojipkgs.fedoraproject.org/repos/epel8-next-build/4499598/x86_64%60 to ` https://infrastructure-iad.fedoraproject.org/repo/centos/stream8-kojitarget/... in the infra one. This points to the core repo that koji will try to import as a buildroot. 4. `mock -r e8-next.cfg --enable-network --install dnf; mock -r e8-next.cfg --enable-network --shell` ``` # dnf list python38-py* Last metadata expiration check: 0:00:04 ago on Fri Mar 4 09:37:44 2022. Available Packages python38-PyMySQL.noarch 0.10.1-1.module_el8.4.0+677+b84873a2 build python38-pycparser.noarch 2.19-3.module_el8.4.0+647+0ba99ce8 build python38-pysocks.noarch 1.7.1-4.module_el8.4.0+665+abc3a503 build python38-pytz.noarch 2019.3-3.module_el8.4.0+647+0ba99ce8 build python38-pyyaml.x86_64 5.4.1-1.module_el8.6.0+929+89303463 build ``` 5. Do the same with the other config so we see what the main is presenting to koji: ``` # dnf list python38-py* Last metadata expiration check: 0:00:01 ago on Fri Mar 4 09:39:58 2022. Available Packages python38-PyMySQL.noarch 0.10.1-1.module_el8.4.0+677+b84873a2 build python38-py.noarch 1.8.0-8.module_el8.5.0+742+dbad1979 build python38-pycparser.noarch 2.19-3.module_el8.4.0+647+0ba99ce8 build python38-pyparsing.noarch 2.4.5-3.module_el8.5.0+742+dbad1979 build python38-pysocks.noarch 1.7.1-4.module_el8.4.0+665+abc3a503 build python38-pytest.noarch 4.6.6-3.module_el8.5.0+742+dbad1979 build python38-pytz.noarch 2019.3-3.module_el8.4.0+647+0ba99ce8 build python38-pyyaml.x86_64 5.4.1-1.module_el8.6.0+929+89303463 build ```

Notice the differing packages. For some reason part of koji is seeing some problem with the packages in a set of directories and is not allowing access to them.

On 30. 01. 20 16:44, Pat Riehecky wrote:

While I lack good answers, perhaps another question.  What a the thoughts on using python `.pth` files for python modules that work in multiple interpreters?

In theory this would permit bit for bit identical libraries in multiple interpreters at once?

Where would you put the files on the filesystem level?

How would we handle different bytecode caches and extension modules?

On 30. 01. 20 16:52, Pat Riehecky wrote:

On 1/30/20 9:47 AM, Miro Hrončok wrote:

...

On 30. 01. 20 16:44, Pat Riehecky wrote:

...

While I lack good answers, perhaps another question.  What a the thoughts on using python `.pth` files for python modules that work in multiple interpreters?

In theory this would permit bit for bit identical libraries in multiple interpreters at once?

Where would you put the files on the filesystem level?

How would we handle different bytecode caches and extension modules?

Perhaps something like /usr/lib/python-epel?  I thought python 3.6+ used the __pycache__ directory that was able to distinguish between the various python bytecodes.

Yes, but do we always ship all possible versions?

I believe extension modules must be compiled for a specific interpreter... I could be wrong there... I don't recall ever building one myself.

That is the case, exactly.