Hi,
This is following up on a brief conversation I was part of in IRC (#fedora-devel) earlier:
I've never been quite sure how to name packages written in Python when they are just applications. Many "applications" written in Python still install themselves using distutils or setuptools, and so install a "module" into the site_packages directory. By a strict interpretation of our Python packaging guidelines [1] [2], python 3 "modules" MUST be packaged as "python3-name".
Since technically any nontrivial Python application that can be installed via distutils or setuptools likely installs a module, this strict interpretation of the guidelines would mean that software that happens to be written in Python but does not intend to provide a library must be named with the python3-* prefix.
I am not sure that this is intended (since there's definitely software in the distribution written in Python not named with a python prefix). If this is not intended, then it would be nice if we could reword the guideline to be more clear.
If it is intended, I'm prepared to argue that the current guideline is too strict and should be revisited, both from a practical position (users looking for software called "foobar" who many not care whether or not it's written in Python will be confused if it is packaged under the name "python-foobar" in Fedora), and from a philosophical one (we don't put the "lib" prefix on every library package if upstream doesn't use that as their name because we generally try to respect the way upstream names their software).
As a counter-proposal I would suggest that, as we currently do, we require the python3-prefix to be provided by the package, but explicitly leave it to the packager+reviewer's discretion whether or not the prefix must be part of the real name, too. Some other languages do already do this: nodejs [3] and ocaml [4] both explicitly have "if this primarily provides a tool or application" clauses in their naming guidelines. I think it makes sense to have something similar for Python, to help avoid confusion.
Ben Rosser
[1] https://fedoraproject.org/wiki/Packaging:Naming#Python_modules [2] https://fedoraproject.org/wiki/Packaging:Python#Provides [3] https://fedoraproject.org/wiki/Packaging:Node.js?rd=Node.js/#Naming_Guidelin... [4] https://fedoraproject.org/wiki/Packaging:Naming#OCaml_modules
On 10 August 2017 at 10:49, Ben Rosser rosser.bjr@gmail.com wrote:
As a counter-proposal I would suggest that, as we currently do, we require the python3-prefix to be provided by the package, but explicitly leave it to the packager+reviewer's discretion whether or not the prefix must be part of the real name, too. Some other languages do already do this: nodejs [3] and ocaml [4] both explicitly have "if this primarily provides a tool or application" clauses in their naming guidelines. I think it makes sense to have something similar for Python, to help avoid confusion.
This is technically already the de facto policy (if it wasn't, a lot of Fedora's build tooling would have very different names), but the Python packaging policy doesn't spell it out explicitly.
So while I think you're right that we should make it officially that it's OK for application packagers to let it be an implementation detail whether something is written in Python or not, I also think we would ideally take more steps than we do today to make sure that such use of Python actually *is* an implementation detail, rather than something that can leak out and cause compatibility issues with upstream Python modules.
For example, some time ago, the "mock" command line tool had to rename its supporting Python package to "mockbuild" in order to resolve a file conflict when the python-mock package was first added to the distro.
In that vein, the potentially more robust approach I've been considering for managing that kind of situation is the idea of recommending that these "the use of Python is an implementation detail" applications be restructured to take advantage of the native virtual environment support in Python 3 in order to put their application specific modules in a private virtual environment, rather than installing them directly into the system Python as generally importable packages.
Specifically, having venv available by default allows us to do this as part of an RPM build:
1. Use "python3 -m venv --system-site-packages --without-pip <the_venv_dir>" to create an empty symlinked virtual environment that can see system packages 2. From *outside* the venv, run "pip3 install --prefix <the_venv_dir>" to install the application specific module(s) 3. Copy any generated "bin" scripts out into the FHS binary directory (their shebang lines will automatically be set up to auto-activate the private venv, since they'll reference the venv's Python symlink, *not* the system level path)
Having a private venv like that available would also give application packagers a relatively clean way to inject "before anything else runs" behaviour via *.pth files, which will let them do things like implicitly adjusting sys.path to include additional directories, as well as tinkering with __main__.__requires__ in order to manipulate the behaviour of pkg_resources parallel loading support.
The main reason I hadn't gotten around to actually proposing that is that there are a few significant open questions where I'm not sure what the right answer would be:
1. Where would these private venvs actually live? 2. Does the answer to (1) differ for pure Python app modules vs binary extension modules? 3. Would we allow daisy chaining of these private venvs via *.pth files? (see https://github.com/berdario/pew#add) 4. How would we make this manageable across Fedora/EPEL/SCLo spec files?
Cheers, Nick.
On 08/10/2017 02:49 AM, Ben Rosser wrote:
Hi,
This is following up on a brief conversation I was part of in IRC (#fedora-devel) earlier:
I've never been quite sure how to name packages written in Python when they are just applications. Many "applications" written in Python still install themselves using distutils or setuptools, and so install a "module" into the site_packages directory. By a strict interpretation of our Python packaging guidelines [1] [2], python 3 "modules" MUST be packaged as "python3-name".
Since technically any nontrivial Python application that can be installed via distutils or setuptools likely installs a module, this strict interpretation of the guidelines would mean that software that happens to be written in Python but does not intend to provide a library must be named with the python3-* prefix.
I am not sure that this is intended (since there's definitely software in the distribution written in Python not named with a python prefix). If this is not intended, then it would be nice if we could reword the guideline to be more clear.
If it is intended, I'm prepared to argue that the current guideline is too strict and should be revisited, both from a practical position (users looking for software called "foobar" who many not care whether or not it's written in Python will be confused if it is packaged under the name "python-foobar" in Fedora), and from a philosophical one (we don't put the "lib" prefix on every library package if upstream doesn't use that as their name because we generally try to respect the way upstream names their software).
Hello,
You're right that this is currently missing from the guidelines. Would you be interested in writing up a draft change?
Python modules are a special case of "Addon Packages" [0]:
If a new package is considered an "addon" package that enhances or adds a new functionality to an existing Fedora package without being useful on its own, its name should > reflect this fact.
So, if you're not extending Python, but providing a standalone app, you don't need to use the python3- prefix.
Specifically, the de-facto policy (which we should write down) is that if nothing else is expected to import the module, then don't use the python3- prefix. So, by not using the prefix, you're signaling that the Python module is an implementation detail and can be removed (or moved, for example to a different Python implementation) at any time.
If others *are* expected to import the module, some people make a "python3-foobar" sub-package, and make the main "foobar" package require it.
[0] https://fedoraproject.org/wiki/Packaging:Naming?rd=Packaging:NamingGuideline...
As a counter-proposal I would suggest that, as we currently do, we require the python3-prefix to be provided by the package, but explicitly leave it to the packager+reviewer's discretion whether or not the prefix must be part of the real name, too. Some other languages do already do this: nodejs [3] and ocaml [4] both explicitly have "if this primarily provides a tool or application" clauses in their naming guidelines. I think it makes sense to have something similar for Python, to help avoid confusion.
That's a nice idea.
I'll note that modules installed using Python's mechanisms (setuptools, flit, pip, etc.) automatically provide "python3dist(name)". If we write new guidelines, I think we should promote that rather than the "python3-" prefix.
On Thu, Aug 10, 2017 at 11:46:42AM +0200, Petr Viktorin wrote:
On 08/10/2017 02:49 AM, Ben Rosser wrote:
Hi,
This is following up on a brief conversation I was part of in IRC (#fedora-devel) earlier:
I've never been quite sure how to name packages written in Python when they are just applications. Many "applications" written in Python still install themselves using distutils or setuptools, and so install a "module" into the site_packages directory. By a strict interpretation of our Python packaging guidelines [1] [2], python 3 "modules" MUST be packaged as "python3-name".
Since technically any nontrivial Python application that can be installed via distutils or setuptools likely installs a module, this strict interpretation of the guidelines would mean that software that happens to be written in Python but does not intend to provide a library must be named with the python3-* prefix.
I am not sure that this is intended (since there's definitely software in the distribution written in Python not named with a python prefix). If this is not intended, then it would be nice if we could reword the guideline to be more clear.
If it is intended, I'm prepared to argue that the current guideline is too strict and should be revisited, both from a practical position (users looking for software called "foobar" who many not care whether or not it's written in Python will be confused if it is packaged under the name "python-foobar" in Fedora), and from a philosophical one (we don't put the "lib" prefix on every library package if upstream doesn't use that as their name because we generally try to respect the way upstream names their software).
Hello,
You're right that this is currently missing from the guidelines. Would you be interested in writing up a draft change?
Python modules are a special case of "Addon Packages" [0]:
If a new package is considered an "addon" package that enhances or adds a new functionality to an existing Fedora package without being useful on its own, its name should > reflect this fact.
So, if you're not extending Python, but providing a standalone app, you don't need to use the python3- prefix.
Specifically, the de-facto policy (which we should write down) is that if nothing else is expected to import the module, then don't use the python3- prefix. So, by not using the prefix, you're signaling that the Python module is an implementation detail and can be removed (or moved, for example to a different Python implementation) at any time.
This guidelines are just too vague, and I think this is the source of many disagreements over naming. I think we should stop guessing, and assume that if a package installs any importable module, something might start using that module and behave accordingly.
I'd use the following rules: 1. If "primarily an application", use upstream name for both the srpm and main binary rpm
2. Except for packages which are strongly python-version dependent, and install executables for both python versions (sympy, sphinx, ipython, ...). In this case use python2- and python3- prefixes for the binary packages, and have one of those packages Provide the original name.
3. If "primarily a library", use python- prefix for srpm, and python2-, python3- prefixes for the binary packages
4. For all 1., 2., and 3., always add '%python_provide python2-foobar' or '%python_provide python3-foobar' or '%python_provide python%{py3_pkgversion}-foobar'.
This way we provide consistent naming for installation (dnf install python2-foobar and dnf install python3-foobar DTRT), and "applications" still follow upstream naming.
If others *are* expected to import the module, some people make a "python3-foobar" sub-package, and make the main "foobar" package require it.
[0] https://fedoraproject.org/wiki/Packaging:Naming?rd=Packaging:NamingGuideline...
As a counter-proposal I would suggest that, as we currently do, we require the python3-prefix to be provided by the package, but explicitly leave it to the packager+reviewer's discretion whether or not the prefix must be part of the real name, too.
Yes! I think what I wrote above matches this.
Some other languages do already do this: nodejs [3] and ocaml [4] both explicitly have "if this primarily provides a tool or application" clauses in their naming guidelines. I think it makes sense to have something similar for Python, to help avoid confusion.
That's a nice idea.
I'll note that modules installed using Python's mechanisms (setuptools, flit, pip, etc.) automatically provide "python3dist(name)". If we write new guidelines, I think we should promote that rather than the "python3-" prefix.
python3dist(name) provides have no %_isa suffix.
Right now packages which are multilib and have such provides are uncommon, but this could become problematic if the use of those tags becomes widespread.
For example: $ dnf repoquery --provides python3-qhexedit2-qt5.x86_64 python3-qhexedit2-qt5 = 0.8.3-2.fc26 python3-qhexedit2-qt5(x86-64) = 0.8.3-2.fc26 python3.6dist(qhexedit-qt5) = 0.8.3 python3dist(qhexedit-qt5) = 0.8.3 $ dnf repoquery --provides python3-qhexedit2-qt5.i686 python3-qhexedit2-qt5 = 0.8.3-2.fc26 python3-qhexedit2-qt5(x86-32) = 0.8.3-2.fc26 python3.6dist(qhexedit-qt5) = 0.8.3 python3dist(qhexedit-qt5) = 0.8.3
If I install foobar.i686, it'd have to depend on python3-qhexedit2-qt5(x86-64), because python3.6dist(qhexedit-qt5) is not strong enough.
Afaics, %_isa never came up in the original discussion. Petr, Tomas, Miro, you were owners of the change. Can you shine some light on this question? (If you say that this is just not important enough, I can live with that ;))
Zbyszek
"ZJ" == Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl writes:
ZJ> This guidelines are just too vague, and I think this is the source ZJ> of many disagreements over naming.
Well, there is https://fedoraproject.org/wiki/Packaging:Guidelines#Libraries_and_Applicatio... which I don't think is particularly vague.
ZJ> I think we should stop guessing, and assume that if a package ZJ> installs any importable module, something might start using that ZJ> module and behave accordingly.
Which is covered by the "Mixed Use Packages" section, isn't it?
This question isn't unique to Python, which is why it's in the main packaging guidelines. If you believe that Python should be stricter here or has special requirements, then you can certainly propose some Python-specific guidelines. Languages which have more complete automatic dependency generation don't particularly need to care outside of the requirement to subpackage for purposes of dependency minimization.
- J<
python-devel@lists.fedoraproject.org