I have a pair of python packages that bundle a series of python packages under a subpackage extern. In particular, python-astropy bundles six, configobj, pytest and ply.
What I do is patching the source, so that instead of
from astropy.extern.six import ...
I have
from six import ...
This works for astropy itself, but other third-party packages require astropy.extern to exist and work, i.e, its part of the API of astropy.
I have read this
http://fedoraproject.org/wiki/User:Toshio/Unbundling_Python_Modules
but before trying to implement changes, my question is: is a valid approach to remove the bundled library and make a files ystem link to the system library?
Thanks, Sergio
On Mon, 9 Jun 2014 11:57:07 +0200, Sergio Pascual wrote:
I have a pair of python packages that bundle a series of python packages under a subpackage extern. In particular, python-astropy bundles six, configobj, pytest and ply.
What I do is patching the source, so that instead of
from astropy.extern.six import ...
I have
from six import ...
This works for astropy itself, but other third-party packages require astropy.extern to exist and work, i.e, its part of the API of astropy.
I have read this
http://fedoraproject.org/wiki/User:Toshio/Unbundling_Python_Modules
but before trying to implement changes, my question is: is a valid approach to remove the bundled library and make a files ystem link to the system library?
IMO, it would be cleaner to adjust the "import" statements and make them try importing the system modules before falling back to the bundled modules. Such a change could be merged upstream, and then you would only need to delete the bundled modules.
Of course, a primary question is why are these modules bundled? Is it only out of convenience (for the users/developers)? Are there strict dependencies on specific versions of these modules?
----- Original Message -----
On Mon, 9 Jun 2014 11:57:07 +0200, Sergio Pascual wrote:
I have a pair of python packages that bundle a series of python packages under a subpackage extern. In particular, python-astropy bundles six, configobj, pytest and ply.
What I do is patching the source, so that instead of
from astropy.extern.six import ...
I have
from six import ...
This works for astropy itself, but other third-party packages require astropy.extern to exist and work, i.e, its part of the API of astropy.
I have read this
http://fedoraproject.org/wiki/User:Toshio/Unbundling_Python_Modules
but before trying to implement changes, my question is: is a valid approach to remove the bundled library and make a files ystem link to the system library?
IMO, it would be cleaner to adjust the "import" statements and make them try importing the system modules before falling back to the bundled modules. Such a change could be merged upstream, and then you would only need to delete the bundled modules.
Of course, a primary question is why are these modules bundled? Is it only out of convenience (for the users/developers)? Are there strict dependencies on specific versions of these modules?
Another viable approach could be (not tested) replacing contents of astropy.extern.six with single line "from six import *". That'd unbundle six and wouldn't break depending libraries.
Slavek
2014-06-09 12:44 GMT+02:00 Bohuslav Kabrda bkabrda@redhat.com:
Another viable approach could be (not tested) replacing contents of astropy.extern.six with single line "from six import *". That'd unbundle six and wouldn't break depending libraries.
No, it doesn't work, python-2.x does relative imports in preference to absolute imports, see the last point here
http://fedoraproject.org/wiki/User:Toshio/Unbundling_Python_Modules
in fact, the directory approach doesn't work for me either with six.py, import six tries to import itself
Slavek
packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging
----- Original Message -----
2014-06-09 12:44 GMT+02:00 Bohuslav Kabrda < bkabrda@redhat.com > :
Another viable approach could be (not tested) replacing contents of astropy.extern.six with single line "from six import *". That'd unbundle six and wouldn't break depending libraries.
No, it doesn't work, python-2.x does relative imports in preference to absolute imports, see the last point here
http://fedoraproject.org/wiki/User:Toshio/Unbundling_Python_Modules
in fact, the directory approach doesn't work for me either with six.py, import six tries to import itself
Right. You could try using imp module to import "six" from each of "site.getsitepackages" (although you'd have to do this by importlib for python3, where "imp" is deprecated). After having the module object imported, you could bind its variables to globals of astropy.extern.six.
Slavek
Slavek
--
packaging mailing list
packaging@lists.fedoraproject.org
-- packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging
2014-06-09 12:36 GMT+02:00 Michael Schwendt mschwendt@gmail.com:
On Mon, 9 Jun 2014 11:57:07 +0200, Sergio Pascual wrote:
I have a pair of python packages that bundle a series of python packages under a subpackage extern. In particular, python-astropy bundles six, configobj, pytest and ply.
What I do is patching the source, so that instead of
from astropy.extern.six import ...
I have
from six import ...
This works for astropy itself, but other third-party packages require astropy.extern to exist and work, i.e, its part of the API of astropy.
I have read this
http://fedoraproject.org/wiki/User:Toshio/Unbundling_Python_Modules
but before trying to implement changes, my question is: is a valid
approach
to remove the bundled library and make a files ystem link to the system library?
IMO, it would be cleaner to adjust the "import" statements and make them try importing the system modules before falling back to the bundled modules. Such a change could be merged upstream, and then you would only need to delete the bundled modules.
So, do you mean changing things like:
from ...extern.six.moves.http_client import HTTPSConnection
to
try: from six.moves.http_client import HTTPSConnection except ImportError: from ...extern.six.moves.http_client import HTTPSConnection
everywhere in the code? Surely upstream is not going to to accept it.
What they could accept is a mechanism in extern.six to load the system package when is available and the bundled package when it is not. The same for the other packages (ply, pytest, etc). But that's what I'm trying to avoid by linking.
Of course, a primary question is why are these modules bundled? Is it only out of convenience (for the users/developers)? Are there strict dependencies on specific versions of these modules?
It's a matter of convenience. The only hard requirement of astropy is numpy.
The bundled versions are kept up to date, sometimes versions are new than Fedora's.
I must say that upstream is very helpful and they make very easy to unbundle the C libraries shipped in the code. It's just the python packages where I'm not completely happy how I'm unbundle them.
-- packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging
On Mon, 9 Jun 2014 13:43:53 +0200, Sergio Pascual wrote:
So, do you mean changing things like:
from ...extern.six.moves.http_client import HTTPSConnection
to
try: from six.moves.http_client import HTTPSConnection except ImportError: from ...extern.six.moves.http_client import HTTPSConnection
everywhere in the code? Surely upstream is not going to to accept it.
Well, you've mentioned already that the "extern" module path is considered part of the API, so changing the import statements would need to happen within those files. It depends a bit on how exactly the bundling is done, i.e. whether entire Python module directory trees are copied without modifications. If upstream needed to adjust the copied modules everytime a new version is needed, that could get tiresome (without using helper scripts).
What they could accept is a mechanism in extern.six to load the system package when is available and the bundled package when it is not. The same for the other packages (ply, pytest, etc). But that's what I'm trying to avoid by linking.
For every symlink you would be missing a strict dependency on the target file, unless you would add them as RPM Requires on file paths. If the structure of the system Python module changed, that would break the symlinks. And if a bundled module is newer/older than its corresponding system module, there could be new/old files with no target to link with.
On Mon, Jun 09, 2014 at 01:43:53PM +0200, Sergio Pascual wrote:
So, do you mean changing things like:
from ...extern.six.moves.http_client import HTTPSConnection
to
try: from six.moves.http_client import HTTPSConnection except ImportError: from ...extern.six.moves.http_client import HTTPSConnection
everywhere in the code? Surely upstream is not going to to accept it.
Some upstreams do accept this.
What they could accept is a mechanism in extern.six to load the system package when is available and the bundled package when it is not. The same for the other packages (ply, pytest, etc). But that's what I'm trying to avoid by linking.
This can be done as well. Take a look at how kitchen.pycompat* is coded for instance:
http://bzr.fedorahosted.org/bzr/kitchen/devel/annotate/head:/kitchen/pycompa...
Note that import * isn't guaranteed to get all of the publically available API because __all__ might not be complete in a given module. If you run into that you have to do something like:
try: from module import * from module import not_in__all__ except ImportError: from _module import * from _module import not_in__all__
Of course, a primary question is why are these modules bundled? Is it only out of convenience (for the users/developers)? Are there strict dependencies on specific versions of these modules?It's a matter of convenience. The only hard requirement of astropy is numpy.
Why are these modules considered part of the astropy API? That seems like bad practice. It may be that the packages which use astropy.extern.* should also be taught to use
try: from system module except ImportError: from bundled location
-Toshio
On Mon, Jun 09, 2014 at 12:36:57PM +0200, Michael Schwendt wrote:
On Mon, 9 Jun 2014 11:57:07 +0200, Sergio Pascual wrote:
I have a pair of python packages that bundle a series of python packages under a subpackage extern. In particular, python-astropy bundles six, configobj, pytest and ply.
What I do is patching the source, so that instead of
from astropy.extern.six import ...
I have
from six import ...
This works for astropy itself, but other third-party packages require astropy.extern to exist and work, i.e, its part of the API of astropy.
I have read this
http://fedoraproject.org/wiki/User:Toshio/Unbundling_Python_Modules
but before trying to implement changes, my question is: is a valid approach to remove the bundled library and make a files ystem link to the system library?
IMO, it would be cleaner to adjust the "import" statements and make them try importing the system modules before falling back to the bundled modules. Such a change could be merged upstream, and then you would only need to delete the bundled modules.
+1
There's a hierarchy of things that are somewhat acceptable in regards to bundling. The least desirable is probably to get an exception from the FPC. The most desirable is to unbundle via a build-time switch or runtime detection that upstream has merged.
Symlinks fall in the middle of that. There's some pretty easy recipes for making python modules fit into the most desirable category so I would not recommend dipping down to the level of symlinks.
-Toshio
2014-06-10 2:34 GMT+02:00 Toshio Kuratomi a.badger@gmail.com:
On Mon, Jun 09, 2014 at 12:36:57PM +0200, Michael Schwendt wrote:
On Mon, 9 Jun 2014 11:57:07 +0200, Sergio Pascual wrote:
I have a pair of python packages that bundle a series of python
packages
under a subpackage extern. In particular, python-astropy bundles six, configobj, pytest and ply.
What I do is patching the source, so that instead of
from astropy.extern.six import ...
I have
from six import ...
This works for astropy itself, but other third-party packages require astropy.extern to exist and work, i.e, its part of the API of astropy.
I have read this
http://fedoraproject.org/wiki/User:Toshio/Unbundling_Python_Modules
but before trying to implement changes, my question is: is a valid
approach
to remove the bundled library and make a files ystem link to the system library?
IMO, it would be cleaner to adjust the "import" statements and make them try importing the system modules before falling back to the bundled modules. Such a change could be merged upstream, and then you would only need to delete the bundled modules.
+1
Great, thank you for the clarification
There's a hierarchy of things that are somewhat acceptable in regards to bundling. The least desirable is probably to get an exception from the FPC. The most desirable is to unbundle via a build-time switch or runtime detection that upstream has merged.
In this sense, this is the PR I've sent astropy to get runtime detection of six
https://github.com/astropy/astropy/pull/2623
Symlinks fall in the middle of that. There's some pretty easy recipes for making python modules fit into the most desirable category so I would not recommend dipping down to the level of symlinks.
-Toshio
-- packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging
packaging@lists.fedoraproject.org
-
Bohuslav Kabrda -
Michael Schwendt -
Sergio Pascual -
Toshio Kuratomi