Hello all,
I recently upstreamed some fixes to a new RPM dependency generator that will be available as an option for distributions to enable. The new generator uses python .egg data to generate Provides and Requires in the form of pythonXegg(Y), where X is the Python major version and Y is the module.
The code for the new generator is available in RPM's GitHub[1]. I've also made a package in Copr[2] for members of the Python SIG to test out and provide feedback.
I would greatly appreciate it if the Python SIG would try it out. It's available for Fedora 22, 23, and rawhide through the Copr.
There is one issue I haven't quite been able to figure out, though. When being run using Python 3 (Fedora 23+ default), it appears to add an unversioned "python(abi)" Require to modules on top of the versioned one. This does not occur when the code is being run in Python 2 (Fedora 22 default), and I cannot quite figure out why this is happening.
If anyone has any idea, please let me know, as I would definitely like this to fix this issue so that it is perfectly functional on both Python 2 and Python 3.
The Python interpreter being used can be switched in /usr/lib/rpm/fileattrs/pythoneggs.attr
Best regards, Neal
[1]: https://github.com/rpm-software-management/rpm/blob/master/scripts/pythonegg... [2]: https://copr.fedoraproject.org/coprs/ngompa/rpm-depgen-pythoneggs/
On 17 November 2015 at 00:38, Neal Gompa ngompa13@gmail.com wrote:
Hello all,
I recently upstreamed some fixes to a new RPM dependency generator that will be available as an option for distributions to enable. The new generator uses python .egg data to generate Provides and Requires in the form of pythonXegg(Y), where X is the Python major version and Y is the module.
I'm not clear on what you mean by depending on an egg. Eggs are a binary format that isn't compatible with Linux distro packaging policies, since they lose too much structural information regarding where files should be installed for policy compliance - that's one of the major reasons we defined the wheel format as a next generation replacement: https://www.python.org/dev/peps/pep-0427/.
Is the goal to enable installation commands like "dnf install python3egg(Django)" and have that install the distro's Django package?
If so, then there's some relevant work currently under way upstream to improve the interaction between Python installation tools and build systems to improve the metadata extraction process, rather than relying on implementation details of setuptools.
Regards, Nick.
On Tue, Nov 17, 2015 at 3:26 AM, Nick Coghlan ncoghlan@gmail.com wrote:
I'm not clear on what you mean by depending on an egg. Eggs are a binary format that isn't compatible with Linux distro packaging policies, since they lose too much structural information regarding where files should be installed for policy compliance - that's one of the major reasons we defined the wheel format as a next generation replacement: https://www.python.org/dev/peps/pep-0427/.
I don't think I've ever heard of this, much less seen it in use. And it looks like the format was approved well over two years ago, so I'm *really* surprised nothing has happened since then. That said, the dependency generator only reads egg info data about the module name and the modules it requires and translates them into RPM Provides/Requires. The other information is too unreliable to use.
Is the goal to enable installation commands like "dnf install python3egg(Django)" and have that install the distro's Django package?
Essentially, yes.
If so, then there's some relevant work currently under way upstream to improve the interaction between Python installation tools and build systems to improve the metadata extraction process, rather than relying on implementation details of setuptools.
That's great, and I hope to see something soon out of that. For a little bit of history on the pythonegg dependency generator, it was developed by Per Øyvind Karlsen for Mandriva to have more Perl-like automatic dependency generation for Python. He brought it forward and upstreamed it to RPM in August, and I made several fixes to it which were upstreamed earlier this month.
Variants of this generator are in use in both Mandriva and Mageia, and I wanted to give the Python SIG in Fedora the opportunity to try it out and see if we might want to enable it in Fedora.
Essentially, for things that have egg info that includes the requirements and the name of the module, it makes it simpler to figure out what how to install Python packages in a more cross-distro manner.
On 17 November 2015 at 22:05, Neal Gompa ngompa13@gmail.com wrote:
On Tue, Nov 17, 2015 at 3:26 AM, Nick Coghlan ncoghlan@gmail.com wrote:
I'm not clear on what you mean by depending on an egg. Eggs are a binary format that isn't compatible with Linux distro packaging policies, since they lose too much structural information regarding where files should be installed for policy compliance - that's one of the major reasons we defined the wheel format as a next generation replacement: https://www.python.org/dev/peps/pep-0427/.
I don't think I've ever heard of this, much less seen it in use. And it looks like the format was approved well over two years ago, so I'm *really* surprised nothing has happened since then.
We don't currently have real-time stats, but when Donald Stufft ran PyPI's download stats back in April, wheels represented about 20% of all downloads, while eggs had dropped to 2-3% (from ~5% a year earlier): https://caremad.io/2015/04/a-year-of-pypi-downloads/
Variants of this generator are in use in both Mandriva and Mageia,
Ah, that's likely to have an impact. The upstream distutils-sig and python-dev lists don't include any Mageia/Mandriva users that I'm aware of, so that's likely to delay downstream community awareness of changes in upstream recommendations. We currently only have close collaborations with Debian, Ubuntu, Fedora and RHEL/CentOS due to overlapping developer communities.
and I wanted to give the Python SIG in Fedora the opportunity to try it out and see if we might want to enable it in Fedora.
Essentially, for things that have egg info that includes the requirements and the name of the module, it makes it simpler to figure out what how to install Python packages in a more cross-distro manner.
Having a plugin to automatically generate appropriate upstream Provides/Requires to allow installation based on PyPI distribution names definitely has the potential to be useful. It was only the choice of "python2egg" and "python3egg" as the name that threw me, since that doesn't align with the upstream community terminology (the egg format is specific to setuptools and easy_install)
Regards, Nick.
On Nov 17, 2015, at 7:54 AM, Nick Coghlan ncoghlan@gmail.com wrote:
On 17 November 2015 at 22:05, Neal Gompa ngompa13@gmail.com wrote:
On Tue, Nov 17, 2015 at 3:26 AM, Nick Coghlan ncoghlan@gmail.com wrote:
I'm not clear on what you mean by depending on an egg. Eggs are a binary format that isn't compatible with Linux distro packaging policies, since they lose too much structural information regarding where files should be installed for policy compliance - that's one of the major reasons we defined the wheel format as a next generation replacement: https://www.python.org/dev/peps/pep-0427/.
I don't think I've ever heard of this, much less seen it in use. And it looks like the format was approved well over two years ago, so I'm *really* surprised nothing has happened since then.
We don't currently have real-time stats, but when Donald Stufft ran PyPI's download stats back in April, wheels represented about 20% of all downloads, while eggs had dropped to 2-3% (from ~5% a year earlier): https://caremad.io/2015/04/a-year-of-pypi-downloads/
Another stat that I just computed, Eggs have had.. what 15 years? or so as a format and in that time they have accumulated 41,060 eggs uploaded to PyPI. In the uh, 2 or 3 years that Wheels have been around they have accumulated 38,720 total files on PyPI. We’re close to having Wheels beat Eggs in both the download counts and the number of files available.
Variants of this generator are in use in both Mandriva and Mageia,
Ah, that's likely to have an impact. The upstream distutils-sig and python-dev lists don't include any Mageia/Mandriva users that I'm aware of, so that's likely to delay downstream community awareness of changes in upstream recommendations. We currently only have close collaborations with Debian, Ubuntu, Fedora and RHEL/CentOS due to overlapping developer communities.
and I wanted to give the Python SIG in Fedora the opportunity to try it out and see if we might want to enable it in Fedora.
Essentially, for things that have egg info that includes the requirements and the name of the module, it makes it simpler to figure out what how to install Python packages in a more cross-distro manner.
Having a plugin to automatically generate appropriate upstream Provides/Requires to allow installation based on PyPI distribution names definitely has the potential to be useful. It was only the choice of "python2egg" and "python3egg" as the name that threw me, since that doesn't align with the upstream community terminology (the egg format is specific to setuptools and easy_install)
This sounds like it’s actually use the “other” Egg (because for maximum confusion setuptools eggs are actually like 3 different things). Right now both distutils and setuptools will generate an egg-info metadata when installing projects (though distutils generates a file and setuptools a directory). However wheels install a dist-info directory.
I don’t have a better suggestion for name other than I’d prefer not to bake more things onto the “egg” terminology since we (Python packaging upstream) are trying to phase it out.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Tue, Nov 17, 2015 at 8:05 AM, Donald Stufft donald@stufft.io wrote:
On Nov 17, 2015, at 7:54 AM, Nick Coghlan ncoghlan@gmail.com wrote:
On 17 November 2015 at 22:05, Neal Gompa ngompa13@gmail.com wrote:
and I wanted to give the Python SIG in Fedora the opportunity to try it out and see if we might want to enable it in Fedora.
Essentially, for things that have egg info that includes the requirements and the name of the module, it makes it simpler to figure out what how to install Python packages in a more cross-distro manner.
Having a plugin to automatically generate appropriate upstream Provides/Requires to allow installation based on PyPI distribution names definitely has the potential to be useful. It was only the choice of "python2egg" and "python3egg" as the name that threw me, since that doesn't align with the upstream community terminology (the egg format is specific to setuptools and easy_install)
This sounds like it’s actually use the “other” Egg (because for maximum confusion setuptools eggs are actually like 3 different things). Right now both distutils and setuptools will generate an egg-info metadata when installing projects (though distutils generates a file and setuptools a directory). However wheels install a dist-info directory.
I don’t have a better suggestion for name other than I’d prefer not to bake more things onto the “egg” terminology since we (Python packaging upstream) are trying to phase it out.
Is the format inside of the .dist-info directory the same as the older .egg-info and .egg-link directories? If so, it should be easy to add to read that information too.
As for naming, I'm all ears for a better name, because if the "egg" name is going away, I'd rather it not continue to say that.
On Nov 17, 2015, at 8:25 AM, Neal Gompa ngompa13@gmail.com wrote:
On Tue, Nov 17, 2015 at 8:05 AM, Donald Stufft donald@stufft.io wrote:
On Nov 17, 2015, at 7:54 AM, Nick Coghlan ncoghlan@gmail.com wrote:
On 17 November 2015 at 22:05, Neal Gompa ngompa13@gmail.com wrote:
and I wanted to give the Python SIG in Fedora the opportunity to try it out and see if we might want to enable it in Fedora.
Essentially, for things that have egg info that includes the requirements and the name of the module, it makes it simpler to figure out what how to install Python packages in a more cross-distro manner.
Having a plugin to automatically generate appropriate upstream Provides/Requires to allow installation based on PyPI distribution names definitely has the potential to be useful. It was only the choice of "python2egg" and "python3egg" as the name that threw me, since that doesn't align with the upstream community terminology (the egg format is specific to setuptools and easy_install)
This sounds like it’s actually use the “other” Egg (because for maximum confusion setuptools eggs are actually like 3 different things). Right now both distutils and setuptools will generate an egg-info metadata when installing projects (though distutils generates a file and setuptools a directory). However wheels install a dist-info directory.
I don’t have a better suggestion for name other than I’d prefer not to bake more things onto the “egg” terminology since we (Python packaging upstream) are trying to phase it out.
Is the format inside of the .dist-info directory the same as the older .egg-info and .egg-link directories? If so, it should be easy to add to read that information too.
Currently yes. In the future we’ll probably evolve it and then they’ll drift apart. Part of doing that though will be defining a way to determine what “version” of standard the directory is currently using (with the current version being a not very well defined version 0 or something).
As for naming, I'm all ears for a better name, because if the "egg" name is going away, I'd rather it not continue to say that.
-- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ python-devel mailing list python-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Tue, Nov 17, 2015 at 8:34 AM, Donald Stufft donald@stufft.io wrote:
On Nov 17, 2015, at 8:25 AM, Neal Gompa ngompa13@gmail.com wrote: Is the format inside of the .dist-info directory the same as the older .egg-info and .egg-link directories? If so, it should be easy to add to read that information too.
Currently yes. In the future we’ll probably evolve it and then they’ll drift apart. Part of doing that though will be defining a way to determine what “version” of standard the directory is currently using (with the current version being a not very well defined version 0 or something).
The dependency generator uses pkg_resources (specifically Distribution, FileMetadata, PathMetadata) to read data in the .egg-info directory. That should still work with .dist-info, right? If it does, then I can easily support it to read and parse the data. If there's something else I have to do, please let me know, so I can add it.
On Tue, Nov 17, 2015 at 9:06 AM, Nick Coghlan ncoghlan@gmail.com wrote:
On 17 November 2015 at 23:25, Neal Gompa ngompa13@gmail.com wrote:
As for naming, I'm all ears for a better name, because if the "egg" name is going away, I'd rather it not continue to say that.
My suggestions would be either:
python2dist(name)/python3dist(name)or:
python2(name)/python3(name)The "dist" suffix comes from:
- "distutils", the standard library's software distribution utilities
- the "sdist" name used for uploading source archives to PyPI
- the "-Dist" suffix used in the never-really-adopted metadata 1.2 spec [1]
While dropping the suffix entirely seems like a potentially attractive option to me, it may also be ambiguous as to whether it's referring to import package names (eg. "python2(pkg_resources)") or distribution package names (e.g. "python2(setuptools)").
Cheers, Nick.
I think I'll go for pythonXdist(Y), as that seems to be the least ambiguous option. I've made the change in my local code, and I'll submit it as a PR to rpm as soon as I have confirmation that pkg_resources works for reading .dist-info.
On Nov 17, 2015, at 9:26 AM, Neal Gompa ngompa13@gmail.com wrote:
On Tue, Nov 17, 2015 at 8:34 AM, Donald Stufft donald@stufft.io wrote:
On Nov 17, 2015, at 8:25 AM, Neal Gompa ngompa13@gmail.com wrote: Is the format inside of the .dist-info directory the same as the older .egg-info and .egg-link directories? If so, it should be easy to add to read that information too.
Currently yes. In the future we’ll probably evolve it and then they’ll drift apart. Part of doing that though will be defining a way to determine what “version” of standard the directory is currently using (with the current version being a not very well defined version 0 or something).
The dependency generator uses pkg_resources (specifically Distribution, FileMetadata, PathMetadata) to read data in the .egg-info directory. That should still work with .dist-info, right? If it does, then I can easily support it to read and parse the data. If there's something else I have to do, please let me know, so I can add it.
That’s correct. There’s a EggInfoMetadata class and a DistInfoMetadata class in pkg_resources.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Tue, Nov 17, 2015 at 9:35 AM, Donald Stufft donald@stufft.io wrote:
On Nov 17, 2015, at 9:26 AM, Neal Gompa ngompa13@gmail.com wrote:
The dependency generator uses pkg_resources (specifically Distribution, FileMetadata, PathMetadata) to read data in the .egg-info directory. That should still work with .dist-info, right? If it does, then I can easily support it to read and parse the data. If there's something else I have to do, please let me know, so I can add it.
That’s correct. There’s a EggInfoMetadata class and a DistInfoMetadata class in pkg_resources.
So the code to pull the metadata works like this:
if lower.endswith('.egg') or \ lower.endswith('.egg-info') or \ lower.endswith('.egg-link') or \ lower.endswith('.dist-info'): from pkg_resources import Distribution, FileMetadata, PathMetadata dist_name = basename(f) if isdir(f): path_item = dirname(f) metadata = PathMetadata(path_item, f) else: path_item = f metadata = FileMetadata(f) dist = Distribution.from_location(path_item, dist_name, metadata)
And then for pulling the name for Provides, it uses dist.key and Requires is iterating over dist.requires() to get the desired info.
Do I need to actually check the path and force it to use one particular class over the other?
On Nov 17, 2015, at 9:53 AM, Neal Gompa ngompa13@gmail.com wrote:
On Tue, Nov 17, 2015 at 9:35 AM, Donald Stufft donald@stufft.io wrote:
On Nov 17, 2015, at 9:26 AM, Neal Gompa ngompa13@gmail.com wrote:
The dependency generator uses pkg_resources (specifically Distribution, FileMetadata, PathMetadata) to read data in the .egg-info directory. That should still work with .dist-info, right? If it does, then I can easily support it to read and parse the data. If there's something else I have to do, please let me know, so I can add it.
That’s correct. There’s a EggInfoMetadata class and a DistInfoMetadata class in pkg_resources.
So the code to pull the metadata works like this:
if lower.endswith('.egg') or \ lower.endswith('.egg-info') or \ lower.endswith('.egg-link') or \ lower.endswith('.dist-info'): from pkg_resources import Distribution, FileMetadata, PathMetadata dist_name = basename(f) if isdir(f): path_item = dirname(f) metadata = PathMetadata(path_item, f) else: path_item = f metadata = FileMetadata(f) dist = Distribution.from_location(path_item, dist_name, metadata)
And then for pulling the name for Provides, it uses dist.key and Requires is iterating over dist.requires() to get the desired info.
Do I need to actually check the path and force it to use one particular class over the other?
I think you could just do ``pkg_resources.get_distribution(“name”)``?
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 17 November 2015 at 23:25, Neal Gompa ngompa13@gmail.com wrote:
As for naming, I'm all ears for a better name, because if the "egg" name is going away, I'd rather it not continue to say that.
My suggestions would be either:
python2dist(name)/python3dist(name)
or:
python2(name)/python3(name)
The "dist" suffix comes from:
* "distutils", the standard library's software distribution utilities * the "sdist" name used for uploading source archives to PyPI * the "-Dist" suffix used in the never-really-adopted metadata 1.2 spec [1]
While dropping the suffix entirely seems like a potentially attractive option to me, it may also be ambiguous as to whether it's referring to import package names (eg. "python2(pkg_resources)") or distribution package names (e.g. "python2(setuptools)").
Cheers, Nick.
[1] https://www.python.org/dev/peps/pep-0345/
On Nov 17, 2015 6:06 AM, "Nick Coghlan" ncoghlan@gmail.com wrote:
On 17 November 2015 at 23:25, Neal Gompa ngompa13@gmail.com wrote:
As for naming, I'm all ears for a better name, because if the "egg" name is going away, I'd rather it not continue to say that.
My suggestions would be either:
python2dist(name)/python3dist(name)
I'd favor this form due to the import names conflict that Nick mentions (not that I think anyone will code an auto generator that uses import names once this is accepted.. this seems good enough to carry the day.)
One thing I would change, though: instead of python2dist(name) use python2.7dist(name). A module built for python 2.7 won't be able to import a module built for python2.6 (the python interpreter won't look in the directories in which the files are placed) so without both the major.minor numbers we won't be able to rely on the auto generated requires in the spec file.
-Toshio
On Tue, Nov 17, 2015 at 9:44 AM, Toshio Kuratomi a.badger@gmail.com wrote:
On Nov 17, 2015 6:06 AM, "Nick Coghlan" ncoghlan@gmail.com wrote:
On 17 November 2015 at 23:25, Neal Gompa ngompa13@gmail.com wrote:
As for naming, I'm all ears for a better name, because if the "egg" name is going away, I'd rather it not continue to say that.
My suggestions would be either:
python2dist(name)/python3dist(name)I'd favor this form due to the import names conflict that Nick mentions (not that I think anyone will code an auto generator that uses import names once this is accepted.. this seems good enough to carry the day.)
One thing I would change, though: instead of python2dist(name) use python2.7dist(name). A module built for python 2.7 won't be able to import a module built for python2.6 (the python interpreter won't look in the directories in which the files are placed) so without both the major.minor numbers we won't be able to rely on the auto generated requires in the spec file.
-Toshio
That's already guaranteed by the auto-generated python(abi) requires, and that would also make it hugely problematic to use in spec files in any distro agnostic manner, so I don't think I will do that.
On Nov 17, 2015 6:47 AM, "Neal Gompa" ngompa13@gmail.com wrote:
On Tue, Nov 17, 2015 at 9:44 AM, Toshio Kuratomi a.badger@gmail.com
wrote:
One thing I would change, though: instead of python2dist(name) use python2.7dist(name). A module built for python 2.7 won't be able to
import
a module built for python2.6 (the python interpreter won't look in the directories in which the files are placed) so without both the
major.minor
numbers we won't be able to rely on the auto generated requires in the
spec
file.
-Toshio
That's already guaranteed by the auto-generated python(abi) requires, and that would also make it hugely problematic to use in spec files in any distro agnostic manner, so I don't think I will do that.
I could just be being dumb here but it doesn't seem like it would. The python(abi) will require a specific major.minor of the python package itself. It will not require a python-setuptools (for instance) that was built for the same version of python as the package you are installing. If it was true, then you wouldn't need the separate python2dist and python3dist as python(abi) would take care of the difference between packages built for python2 and python3.
example autogenerated deps:
python-setuptools
Provides: python2dist(setuptools)
Requires: python(abi) = 2.7
python2.6-setuptools
Provides: python2dist(setuptools)
Requires: python(abi) = 2.6
python2.6-foo
Requires: python2dist(setuptools)
Requires: python(abi) = 2.6
On my system, I have installed, python-2.7.0 (which provides python(abi)==2.7), python2.6-2.6.0 (which provides python(abi)==2.6), and python-setuptools (deps listed above). I want to install python2.6foo. The depsolver will satisfy python2dist(setuptools) with my already installed python-setuptools package and python(abi) == 2.6 with python2.6-2.6.0. Thus, the library will not function correctly because python-2.6 will not be able to import setuptools.
-Toshio
On Tue, Nov 17, 2015 at 10:43 AM, Toshio Kuratomi a.badger@gmail.com wrote:
On Nov 17, 2015 6:47 AM, "Neal Gompa" ngompa13@gmail.com wrote:
That's already guaranteed by the auto-generated python(abi) requires, and that would also make it hugely problematic to use in spec files in any distro agnostic manner, so I don't think I will do that.
I could just be being dumb here but it doesn't seem like it would. The python(abi) will require a specific major.minor of the python package itself. It will not require a python-setuptools (for instance) that was built for the same version of python as the package you are installing. If it was true, then you wouldn't need the separate python2dist and python3dist as python(abi) would take care of the difference between packages built for python2 and python3.
example autogenerated deps:
python-setuptools
Provides: python2dist(setuptools)
Requires: python(abi) = 2.7
python2.6-setuptools
Provides: python2dist(setuptools)
Requires: python(abi) = 2.6
python2.6-foo
Requires: python2dist(setuptools)
Requires: python(abi) = 2.6
On my system, I have installed, python-2.7.0 (which provides python(abi)==2.7), python2.6-2.6.0 (which provides python(abi)==2.6), and python-setuptools (deps listed above). I want to install python2.6foo. The depsolver will satisfy python2dist(setuptools) with my already installed python-setuptools package and python(abi) == 2.6 with python2.6-2.6.0. Thus, the library will not function correctly because python-2.6 will not be able to import setuptools.
I see the problem you are describing, but how do you solve it currently?
That said, I *think* I could autogenerate Provides for pythonX.Ydist(M) and pythonXdist(M), while only having requires generated with pythonX.Ydist(M). Would that solve the problem while allowing BuildRequires using pythonXdist(M) to pick up the latest one? I'm not entirely sure it would...
On Nov 17, 2015 8:18 AM, "Neal Gompa" ngompa13@gmail.com wrote:
I see the problem you are describing, but how do you solve it currently?
Currently we use manually specified dependencies with package names here. So when python2.6-foo is built, the packager specifies a dependency on python2.6-setuptools.
That said, I *think* I could autogenerate Provides for pythonX.Ydist(M) and pythonXdist(M), while only having requires generated with pythonX.Ydist(M). Would that solve the problem while allowing BuildRequires using pythonXdist(M) to pick up the latest one? I'm not entirely sure it would...
Nope. In the spec you would need both major and minor to make auto generated dependencies work correctly in all circumstances.
If your particular distribution only shipped a single version of python2 and a single version of python 3 on that distribution you could use pythonXdist but the packages would break on other distributions that shipped multiple runtimes.
Something to consider: if you're worried about how to specify manual deps in the spec file portably it may be best to continue using package names there. That way built packages take advantage of the auto dep generator in most cases, most packagers can specify buildrequires using the generic package name that's stable across distro releases (and sometimes across distros) and truly special cases can pin a specific compatible runtime by manually specifying pythonX.Ydist(foo).
Also, if doing it this way makes the pythonXdist unnecessary it would be nice to leave it out. The size of yum metadata is a big bottleneck. I believe someone once enabled erlang auto deps that increased the size so much that we (fedora) removed that auto dep generator from our packaging (overriding the maintainer). It feels like an even worse problem these days when the packageset is bigger and dnf is downloading the full filelist.
-Toshio
"NC" == Nick Coghlan ncoghlan@gmail.com writes:
NC> If so, then there's some relevant work currently under way upstream NC> to improve the interaction between Python installation tools and NC> build systems to improve the metadata extraction process, rather NC> than relying on implementation details of setuptools.
If that's the case, could someone bang out a few paragraphs that we could use as a blueprint for some packaging guidelines? An example spec, or even just the file layout and some idea of how autogenerated dependencies would work would be enough. I know this stuff is a bit new, but we've been doing a really big overhaul of the python stuff and we'd like to at least design to accommodate this rather than having to do it all over again once this new format comes out.
- J<
On 18 November 2015 at 02:29, Jason L Tibbitts III tibbs@math.uh.edu wrote:
"NC" == Nick Coghlan ncoghlan@gmail.com writes:
NC> If so, then there's some relevant work currently under way upstream NC> to improve the interaction between Python installation tools and NC> build systems to improve the metadata extraction process, rather NC> than relying on implementation details of setuptools.
If that's the case, could someone bang out a few paragraphs that we could use as a blueprint for some packaging guidelines? An example spec, or even just the file layout and some idea of how autogenerated dependencies would work would be enough. I know this stuff is a bit new, but we've been doing a really big overhaul of the python stuff and we'd like to at least design to accommodate this rather than having to do it all over again once this new format comes out.
The main policy changes would be to update these two section to mention keeping dist-info directories:
* https://fedoraproject.org/wiki/Packaging:Python#Files_to_include * https://fedoraproject.org/wiki/Packaging:Python#Reviewer_checklist
It would also be desirable to state a preference for dist-info over egg-info.
Thinking about it a bit further, I don't think the latest round of upstream changes should impact the metadata analysis step, as the metadata querying changes are designed to support getting at the metadata without building and installing the package first, and that's not a consideration for the RPM use case.
However, they could potentially affect the py2/3_build macros, as we're looking to finally migrate away from *requiring* the presence of a setup.py file in every source tree, and instead allow out-of-tree build tools, with machine readable instructions for bootstrapping them into the build environment.
The draft PEP for that is at https://github.com/pypa/interoperability-peps/pull/54/files and upstream discussions are on distutils-sig
I'd been thinking using "pip install" instead of "setup.py install" in the build macros would be sufficient, but I now realise that isn't the case - if a project uses flit (for example) as its build utility, then we're going to need to generate a suitable BuildRequires in pyp2rpm and similar tools (perhaps using the "BuidlRequires: pythonX.Ydist(flit)" format). The build macros themselves could still delegate the task of working out the right build command to invoke to pip, though.
Regards, Nick.
On Wed, Nov 18, 2015 at 2:48 AM, Nick Coghlan ncoghlan@gmail.com wrote:
On 18 November 2015 at 02:29, Jason L Tibbitts III tibbs@math.uh.edu wrote:
> "NC" == Nick Coghlan ncoghlan@gmail.com writes:
NC> If so, then there's some relevant work currently under way upstream NC> to improve the interaction between Python installation tools and NC> build systems to improve the metadata extraction process, rather NC> than relying on implementation details of setuptools.
If that's the case, could someone bang out a few paragraphs that we could use as a blueprint for some packaging guidelines? An example spec, or even just the file layout and some idea of how autogenerated dependencies would work would be enough. I know this stuff is a bit new, but we've been doing a really big overhaul of the python stuff and we'd like to at least design to accommodate this rather than having to do it all over again once this new format comes out.
The main policy changes would be to update these two section to mention keeping dist-info directories:
- https://fedoraproject.org/wiki/Packaging:Python#Files_to_include
- https://fedoraproject.org/wiki/Packaging:Python#Reviewer_checklist
It would also be desirable to state a preference for dist-info over egg-info.
Thinking about it a bit further, I don't think the latest round of upstream changes should impact the metadata analysis step, as the metadata querying changes are designed to support getting at the metadata without building and installing the package first, and that's not a consideration for the RPM use case.
However, they could potentially affect the py2/3_build macros, as we're looking to finally migrate away from *requiring* the presence of a setup.py file in every source tree, and instead allow out-of-tree build tools, with machine readable instructions for bootstrapping them into the build environment.
The draft PEP for that is at https://github.com/pypa/interoperability-peps/pull/54/files and upstream discussions are on distutils-sig
I'd been thinking using "pip install" instead of "setup.py install" in the build macros would be sufficient, but I now realise that isn't the case - if a project uses flit (for example) as its build utility, then we're going to need to generate a suitable BuildRequires in pyp2rpm and similar tools (perhaps using the "BuidlRequires: pythonX.Ydist(flit)" format). The build macros themselves could still delegate the task of working out the right build command to invoke to pip, though.
The main issue I see with that is how to make it so that python upgrades aren't obnoxiously painful. If BuildRequires use pythonXdist(module) format, but all *generated* runtime requirements use pythonX.Ydist(module) format, this problem goes away. But as Toshio mentioned, how do we solve that in a multi-version environment (like Enterprise Linux, for instance)?
Using pythonX.Ydist(module) for BuildRequires effectively locks the module to a specific Python version until each and every maintainer upgrades them. That is an awful thing to have to do, and no other programming environment in any RPM-based distribution requires that. Most of the time, this is an unnecessary burden on the package maintainers.
My view on pythonXdist(module) vs pythonX.Ydist(module) for BuildRequires is that DNF/Zypper may actually solve this issue for us. Perhaps presenting it with pythonXdist(module) and a package that provides the appropriate "python(ABI) = X.Y" as part of the builddep grab will actually pick the right one (after all, each module would Require a specific "python(ABI)" anyway). I'm not sure if Yum would do the same, though (I hope it does!). I suppose the key is whether or not the depsolver analyzes the whole request before creating its proposed transaction, rather than iteratively solving and presenting the results.
On Wed, Nov 18, 2015 at 5:27 AM, Neal Gompa ngompa13@gmail.com wrote:
On Wed, Nov 18, 2015 at 2:48 AM, Nick Coghlan ncoghlan@gmail.com wrote:
I'd been thinking using "pip install" instead of "setup.py install" in the build macros would be sufficient, but I now realise that isn't the case - if a project uses flit (for example) as its build utility, then we're going to need to generate a suitable BuildRequires in pyp2rpm and similar tools (perhaps using the "BuidlRequires: pythonX.Ydist(flit)" format). The build macros themselves could still delegate the task of working out the right build command to invoke to pip, though.
The main issue I see with that is how to make it so that python upgrades aren't obnoxiously painful. If BuildRequires use pythonXdist(module) format, but all *generated* runtime requirements use pythonX.Ydist(module) format, this problem goes away. But as Toshio mentioned, how do we solve that in a multi-version environment (like Enterprise Linux, for instance)?
Really, for this I think continuing to use package names is the right thing to do. Package names uniquely identify the package built for the default python version for that distribution+release which is what we want to make rebuilding a package on a newer release with a newer version of python as the default. Attempting to build this into the generated dependencies duplicates the features that relying on the name gives us.
Using pythonX.Ydist(module) for BuildRequires effectively locks the module to a specific Python version until each and every maintainer upgrades them. That is an awful thing to have to do, and no other programming environment in any RPM-based distribution requires that. Most of the time, this is an unnecessary burden on the package maintainers.
Note: we do want this when we're specifically building a package for a non-default version of python. If we want to have python2.7 in EPEL6 or python2.6 in Fedora24 or (python3.5 and python3.4 for that matter), then the package stack built for that non-default python needs to specify that the packages it requires also need to be those built for that stack. So all of those packages can BuildRequire the X.Y autogenerated deps.
It's for the default stack that we want to have some notion of "default" within the dependencies
My view on pythonXdist(module) vs pythonX.Ydist(module) for BuildRequires is that DNF/Zypper may actually solve this issue for us. Perhaps presenting it with pythonXdist(module) and a package that provides the appropriate "python(ABI) = X.Y" as part of the builddep grab will actually pick the right one (after all, each module would Require a specific "python(ABI)" anyway). I'm not sure if Yum would do the same, though (I hope it does!). I suppose the key is whether or not the depsolver analyzes the whole request before creating its proposed transaction, rather than iteratively solving and presenting the results.
No depsolver should solve this in its core code. it's a special case that relies on information and decisions outside of the package deps' literal meanings. Making the special case mandatory would definitely lead to issues. For instance, say python-foo ships a utility along with its library and there is no python3-foo. python3-bar requires the utility to build. If dnf is made to only satisfy the dep if the package also matches on python(abi) then this will not be buildable because the depsolver is trying to be too smart instead of doing what the packager told it to do.
-Toshio
On Wed, Nov 18, 2015 at 11:32 AM, Toshio Kuratomi a.badger@gmail.com wrote:
On Wed, Nov 18, 2015 at 5:27 AM, Neal Gompa ngompa13@gmail.com wrote:
On Wed, Nov 18, 2015 at 2:48 AM, Nick Coghlan ncoghlan@gmail.com wrote:
I'd been thinking using "pip install" instead of "setup.py install" in the build macros would be sufficient, but I now realise that isn't the case - if a project uses flit (for example) as its build utility, then we're going to need to generate a suitable BuildRequires in pyp2rpm and similar tools (perhaps using the "BuidlRequires: pythonX.Ydist(flit)" format). The build macros themselves could still delegate the task of working out the right build command to invoke to pip, though.
The main issue I see with that is how to make it so that python upgrades aren't obnoxiously painful. If BuildRequires use pythonXdist(module) format, but all *generated* runtime requirements use pythonX.Ydist(module) format, this problem goes away. But as Toshio mentioned, how do we solve that in a multi-version environment (like Enterprise Linux, for instance)?
Really, for this I think continuing to use package names is the right thing to do. Package names uniquely identify the package built for the default python version for that distribution+release which is what we want to make rebuilding a package on a newer release with a newer version of python as the default. Attempting to build this into the generated dependencies duplicates the features that relying on the name gives us.
Using pythonX.Ydist(module) for BuildRequires effectively locks the module to a specific Python version until each and every maintainer upgrades them. That is an awful thing to have to do, and no other programming environment in any RPM-based distribution requires that. Most of the time, this is an unnecessary burden on the package maintainers.
Note: we do want this when we're specifically building a package for a non-default version of python. If we want to have python2.7 in EPEL6 or python2.6 in Fedora24 or (python3.5 and python3.4 for that matter), then the package stack built for that non-default python needs to specify that the packages it requires also need to be those built for that stack. So all of those packages can BuildRequire the X.Y autogenerated deps.
It's for the default stack that we want to have some notion of "default" within the dependencies
Based on the feedback from you guys, I've made the changes to move to pythonX.Ydist() in the dependency generator. That code has been submitted as a pull request to the RPM GitHub repository[0]. I also added a switch for those who want pythonXdist() Provides, but it is opt-in rather than opt-out. The option is only for distributions that intend to carry only one Python runtime per major version.
I've built a new package with the changes requested in my Copr repository[1]. The new package is called "rpm-depgen-pythondistdeps" and it obsoletes the older "rpm-depgen-pythoneggs" package. A "dnf update" should get it, but if not, just do "dnf install rpm-depgen-pythondistdeps" and it will do the rest.
[0]: https://github.com/rpm-software-management/rpm/pull/33 [1]: https://copr.fedoraproject.org/coprs/ngompa/rpm-depgen-pythoneggs/
On 22 November 2015 at 04:18, Neal Gompa ngompa13@gmail.com wrote:
Based on the feedback from you guys, I've made the changes to move to pythonX.Ydist() in the dependency generator. That code has been submitted as a pull request to the RPM GitHub repository[0]. I also added a switch for those who want pythonXdist() Provides, but it is opt-in rather than opt-out. The option is only for distributions that intend to carry only one Python runtime per major version.
Very cool, thank you!
Cheers, Nick.
On Sun, Nov 22, 2015 at 10:11 PM, Nick Coghlan ncoghlan@gmail.com wrote:
On 22 November 2015 at 04:18, Neal Gompa ngompa13@gmail.com wrote:
Based on the feedback from you guys, I've made the changes to move to pythonX.Ydist() in the dependency generator. That code has been submitted as a pull request to the RPM GitHub repository[0]. I also added a switch for those who want pythonXdist() Provides, but it is opt-in rather than opt-out. The option is only for distributions that intend to carry only one Python runtime per major version.
Very cool, thank you!
Cheers, Nick.
I do still have one quirk that I can't quite figure out what is causing it. The quirk I mentioned at the beginning of this thread, where it adds an unversioned python(abi) Requires when being executed under Python 3.x. This doesn't happen on Python 2.7, and I don't know why.
Anyone have any thoughts on why that's happening? I'm totally bamboozled on it, so I'd love some help to fix it. :)
On Sun, Nov 22, 2015 at 10:17 PM, Neal Gompa ngompa13@gmail.com wrote:
On Sun, Nov 22, 2015 at 10:11 PM, Nick Coghlan ncoghlan@gmail.com wrote:
On 22 November 2015 at 04:18, Neal Gompa ngompa13@gmail.com wrote:
Based on the feedback from you guys, I've made the changes to move to pythonX.Ydist() in the dependency generator. That code has been submitted as a pull request to the RPM GitHub repository[0]. I also added a switch for those who want pythonXdist() Provides, but it is opt-in rather than opt-out. The option is only for distributions that intend to carry only one Python runtime per major version.
Very cool, thank you!
Cheers, Nick.
I do still have one quirk that I can't quite figure out what is causing it. The quirk I mentioned at the beginning of this thread, where it adds an unversioned python(abi) Requires when being executed under Python 3.x. This doesn't happen on Python 2.7, and I don't know why.
Anyone have any thoughts on why that's happening? I'm totally bamboozled on it, so I'd love some help to fix it. :)
Err, to be clear, the unversioned python(abi) *is unwanted*.
python-devel@lists.fedoraproject.org
-
Donald Stufft -
Jason L Tibbitts III -
Neal Gompa -
Nick Coghlan -
Toshio Kuratomi