Hey there,
So I’m one of the authors of PEP453, the original implementor of ensurepip, and a pip maintainer. I know that pip (and other language level package managers) have a sort of love/hate (sometimes more of one or the other!) relationship with the downstream Linux packagers. I know that PEP453 also puts more focus on this in general since it's now a recommendation for pip to be installed if Python is.
So I'd like to say that If there are problems I'd like to help fix them. Especially if there are problems with pip that make you nervous/upset/sad with having it tied so closely to Python. Also generally about making less headache for distros where pip is involved (pip and the OS package manager stomping on each other etc).
To start off this goal I've filed https://github.com/pypa/pip/issues/1668 to figure out how pip can get our defaults to the point where most users will be installing to ~/.local/ instead of the system location. If there's more things pip can do I'd love to know about them, or if ensurepip or the PEP453 processes have something I can help with too :)
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Hi Donald,
welcome to our mailing list.
----- Original Message -----
From: "Donald Stufft" donald@stufft.io To: python-devel@lists.fedoraproject.org Sent: Friday, March 21, 2014 11:19:49 PM Subject: PEP453 // ensurepip // pip
Hey there,
So I’m one of the authors of PEP453, the original implementor of ensurepip, and a pip maintainer. I know that pip (and other language level package managers) have a sort of love/hate (sometimes more of one or the other!) relationship with the downstream Linux packagers. I know that PEP453 also puts more focus on this in general since it's now a recommendation for pip to be installed if Python is.
So I'd like to say that If there are problems I'd like to help fix them. Especially if there are problems with pip that make you nervous/upset/sad with
I would say this one https://github.com/pypa/pip/issues/1351 is important for us as packagers. It makes me nervous/upset and sad altogether :-).
having it tied so closely to Python. Also generally about making less headache for distros where pip is involved (pip and the OS package manager stomping on each other etc).
To start off this goal I've filed https://github.com/pypa/pip/issues/1668 to figure out how pip can get our defaults to the point where most users will be installing to ~/.local/ instead of the system location. If there's more things pip can do I'd love to know about them, or if ensurepip or the PEP453 processes have something I can help with too :)
Nice, I have put my two cents in it.
Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
python-devel mailing list python-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel
Thank you for bringing up this long-awaited and postponed issue for discussion!
Regards,
Robert Kuska ----------------------------------------------------- rkuska @ #fedora-devel on freenode #brno #gulag #software-collections on brq.redhat
On Mar 24, 2014, at 5:38 AM, Robert Kuska rkuska@redhat.com wrote:
Hi Donald,
welcome to our mailing list.
----- Original Message -----
From: "Donald Stufft" donald@stufft.io To: python-devel@lists.fedoraproject.org Sent: Friday, March 21, 2014 11:19:49 PM Subject: PEP453 // ensurepip // pip
Hey there,
So I’m one of the authors of PEP453, the original implementor of ensurepip, and a pip maintainer. I know that pip (and other language level package managers) have a sort of love/hate (sometimes more of one or the other!) relationship with the downstream Linux packagers. I know that PEP453 also puts more focus on this in general since it's now a recommendation for pip to be installed if Python is.
So I'd like to say that If there are problems I'd like to help fix them. Especially if there are problems with pip that make you nervous/upset/sad with
I would say this one https://github.com/pypa/pip/issues/1351 is important for us as packagers. It makes me nervous/upset and sad altogether :-).
Awesome, well that’s on the list for 1.6 so that should be the next feature release of pip.
having it tied so closely to Python. Also generally about making less headache for distros where pip is involved (pip and the OS package manager stomping on each other etc).
To start off this goal I've filed https://github.com/pypa/pip/issues/1668 to figure out how pip can get our defaults to the point where most users will be installing to ~/.local/ instead of the system location. If there's more things pip can do I'd love to know about them, or if ensurepip or the PEP453 processes have something I can help with too :)
Nice, I have put my two cents in it.
Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
python-devel mailing list python-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel
Thank you for bringing up this long-awaited and postponed issue for discussion!
Regards,
Robert Kuska
rkuska @ #fedora-devel on freenode #brno #gulag #software-collections on brq.redhat
On the topic of re-wheeling (Sorry I just joined so I don’t have it in my history to reply to).
I’m assuming Fedora unbundles the stuff that pip bundles (sorry :/) and I’m guessing that since the Rewheeling is going to pull in the system versions that it’s going to pull in a copy of pip with those things unbundled. If that’s the case you’re going to need to install those things into the virtualenv itself.
However I think that the copy of pip inside of a venv should keep stuff bundled if at all possible. One of the reasons we did this was so that when using pip inside of a venv we don’t make any assertions about what other things you can have installed. So for example we depend on requests, we don’t want someone who is using an older (or newer!) requests inside of their venv to be unable to install it because pip itself uses it. Also another reason we did that is because if you uninstall one of those things and it breaks pip you don’t really have a good way to unbreak it except destroy the venv or install it manually. This is less of a concern for the system installed pip because you have yum or whatever that can be used to fix it and y’all integrate things already to ensure compat :)
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
----- Original Message -----
I would say this one https://github.com/pypa/pip/issues/1351 is important for us as packagers. It makes me nervous/upset and sad altogether :-).
Awesome, well that’s on the list for 1.6 so that should be the next feature release of pip.
Cool, thanks a lot!
having it tied so closely to Python. Also generally about making less headache for distros where pip is involved (pip and the OS package manager stomping on each other etc).
To start off this goal I've filed https://github.com/pypa/pip/issues/1668 to figure out how pip can get our defaults to the point where most users will be installing to ~/.local/ instead of the system location. If there's more things pip can do I'd love to know about them, or if ensurepip or the PEP453 processes have something I can help with too :)
Nice, I have put my two cents in it.
On the topic of re-wheeling (Sorry I just joined so I don’t have it in my history to reply to).
I’m assuming Fedora unbundles the stuff that pip bundles (sorry :/) and I’m guessing that since the Rewheeling is going to pull in the system versions that it’s going to pull in a copy of pip with those things unbundled. If that’s the case you’re going to need to install those things into the virtualenv itself.
Just yesterday, I took ownership of python-pip in Fedora and I'm quite surprised that we don't unbundle anything. I'll have to investigate this. It actually makes sense now that I think of it, since the rewheeled pip has worked for us in virtualenv - had we unbundled the bundled code, it would have failed.
However I think that the copy of pip inside of a venv should keep stuff bundled if at all possible. One of the reasons we did this was so that when using pip inside of a venv we don’t make any assertions about what other things you can have installed. So for example we depend on requests, we don’t want someone who is using an older (or newer!) requests inside of their venv to be unable to install it because pip itself uses it. Also another reason we did that is because if you uninstall one of those things and it breaks pip you don’t really have a good way to unbreak it except destroy the venv or install it manually. This is less of a concern for the system installed pip because you have yum or whatever that can be used to fix it and y’all integrate things already to ensure compat :)
Maybe the reasons for you to bundle are also the reasons why Fedora's pip doesn't have these libraries unbundled. As I've said, I need to investigate this.
Regards, Slavek
Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Mar 25, 2014, at 2:59 AM, Bohuslav Kabrda bkabrda@redhat.com wrote:
----- Original Message -----
I would say this one https://github.com/pypa/pip/issues/1351 is important for us as packagers. It makes me nervous/upset and sad altogether :-).
Awesome, well that’s on the list for 1.6 so that should be the next feature release of pip.
Cool, thanks a lot!
having it tied so closely to Python. Also generally about making less headache for distros where pip is involved (pip and the OS package manager stomping on each other etc).
To start off this goal I've filed https://github.com/pypa/pip/issues/1668 to figure out how pip can get our defaults to the point where most users will be installing to ~/.local/ instead of the system location. If there's more things pip can do I'd love to know about them, or if ensurepip or the PEP453 processes have something I can help with too :)
Nice, I have put my two cents in it.
On the topic of re-wheeling (Sorry I just joined so I don’t have it in my history to reply to).
I’m assuming Fedora unbundles the stuff that pip bundles (sorry :/) and I’m guessing that since the Rewheeling is going to pull in the system versions that it’s going to pull in a copy of pip with those things unbundled. If that’s the case you’re going to need to install those things into the virtualenv itself.
Just yesterday, I took ownership of python-pip in Fedora and I'm quite surprised that we don't unbundle anything. I'll have to investigate this. It actually makes sense now that I think of it, since the rewheeled pip has worked for us in virtualenv - had we unbundled the bundled code, it would have failed.
Ah I just assumed y’all did, that actually makes sense now why you hadn’t run into that issue yet :)
However I think that the copy of pip inside of a venv should keep stuff bundled if at all possible. One of the reasons we did this was so that when using pip inside of a venv we don’t make any assertions about what other things you can have installed. So for example we depend on requests, we don’t want someone who is using an older (or newer!) requests inside of their venv to be unable to install it because pip itself uses it. Also another reason we did that is because if you uninstall one of those things and it breaks pip you don’t really have a good way to unbreak it except destroy the venv or install it manually. This is less of a concern for the system installed pip because you have yum or whatever that can be used to fix it and y’all integrate things already to ensure compat :)
Maybe the reasons for you to bundle are also the reasons why Fedora's pip doesn't have these libraries unbundled. As I've said, I need to investigate this.
For what it’s worth, we do attempt to make it easy for the system level pip to have it’s copies of stuff unbundled, but if you keep it bundled and use rewheel it’ll be easier to do modifications to it than having the system pip and the “bundled” pip have different sources. Because the pip source just bundles stuff as part of it’s source while the venv source has it inside a Wheel so you have to do an unpack, modify, repack dance that rewheel will do for you.
FWIW the technique used by Python for PEP453 matches what we already were doing in the virtualenv package, so if you modify the behavior there, it might be reasonable to also modify it in virtualenv to use rewheel. That would also cut down on the number of different sources of pip.
Regards, Slavek
Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
python-devel mailing list python-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Has this critical pip issue been addressed?
https://lists.fedoraproject.org/pipermail/devel/2012-February/162518.html
On Tue, Mar 25, 2014 at 7:10 AM, Donald Stufft donald@stufft.io wrote:
On Mar 25, 2014, at 2:59 AM, Bohuslav Kabrda bkabrda@redhat.com wrote:
----- Original Message -----
I would say this one https://github.com/pypa/pip/issues/1351 is
important
for us as packagers. It makes me nervous/upset and sad altogether :-).
Awesome, well that's on the list for 1.6 so that should be the next
feature
release of pip.
Cool, thanks a lot!
having it tied so closely to Python. Also generally about making less headache for distros where pip is involved (pip and the OS package manager
stomping
on each other etc).
To start off this goal I've filed
https://github.com/pypa/pip/issues/1668
to figure out how pip can get our defaults to the point where most users
will
be installing to ~/.local/ instead of the system location. If there's
more
things pip can do I'd love to know about them, or if ensurepip or the PEP453 processes have something I can help with too :)
Nice, I have put my two cents in it.
On the topic of re-wheeling (Sorry I just joined so I don't have it in
my
history to reply to).
I'm assuming Fedora unbundles the stuff that pip bundles (sorry :/) and
I'm
guessing that since the Rewheeling is going to pull in the system versions that it's
going
to pull in a copy of pip with those things unbundled. If that's the case you're
going to
need to install those things into the virtualenv itself.
Just yesterday, I took ownership of python-pip in Fedora and I'm quite
surprised that we don't unbundle anything. I'll have to investigate this. It actually makes sense now that I think of it, since the rewheeled pip has worked for us in virtualenv - had we unbundled the bundled code, it would have failed.
Ah I just assumed y'all did, that actually makes sense now why you hadn't run into that issue yet :)
However I think that the copy of pip inside of a venv should keep stuff bundled if at all possible. One of the reasons we did this was so that when using pip
inside of
a venv we don't make any assertions about what other things you can have
installed.
So for example we depend on requests, we don't want someone who is using an
older
(or newer!) requests inside of their venv to be unable to install it because pip
itself
uses it. Also another reason we did that is because if you uninstall one of those
things
and it breaks pip you don't really have a good way to unbreak it except destroy the
venv or
install it manually. This is less of a concern for the system installed pip
because you
have yum or whatever that can be used to fix it and y'all integrate things
already to
ensure compat :)
Maybe the reasons for you to bundle are also the reasons why Fedora's
pip doesn't have these libraries unbundled. As I've said, I need to investigate this.
For what it's worth, we do attempt to make it easy for the system level pip to have it's copies of stuff unbundled, but if you keep it bundled and use rewheel it'll be easier to do modifications to it than having the system pip and the "bundled" pip have different sources. Because the pip source just bundles stuff as part of it's source while the venv source has it inside a Wheel so you have to do an unpack, modify, repack dance that rewheel will do for you.
FWIW the technique used by Python for PEP453 matches what we already were doing in the virtualenv package, so if you modify the behavior there, it might be reasonable to also modify it in virtualenv to use rewheel. That would also cut down on the number of different sources of pip.
Regards, Slavek
Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
python-devel mailing list python-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel
Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
python-devel mailing list python-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel
On Mar 25, 2014, at 8:52 AM, Neal Becker ndbecker2@gmail.com wrote:
Has this critical pip issue been addressed?
https://lists.fedoraproject.org/pipermail/devel/2012-February/162518.html
Not that i’m aware of, although it’s actually an issue where distutils and setuptools supports two different standards here :(
Probably this needs to be resolved in Python itself to get distutils to make directories instead of a file. Or do the same hack that pip does in order to force setuptools monkey patching to take place and have setuptools drop them.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Well just to be clear then, without resolving this issue (which, IMO, is a bug in cpio that should be fixed) it is absolutely unsafe to ever use pip for non-local installs on fedora.
On Tue, Mar 25, 2014 at 9:02 AM, Donald Stufft donald@stufft.io wrote:
On Mar 25, 2014, at 8:52 AM, Neal Becker ndbecker2@gmail.com wrote:
Has this critical pip issue been addressed?
https://lists.fedoraproject.org/pipermail/devel/2012-February/162518.html
Not that i'm aware of, although it's actually an issue where distutils and setuptools supports two different standards here :(
Probably this needs to be resolved in Python itself to get distutils to make directories instead of a file. Or do the same hack that pip does in order to force setuptools monkey patching to take place and have setuptools drop them.
Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
python-devel mailing list python-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel
On Mar 25, 2014, at 9:29 AM, Neal Becker ndbecker2@gmail.com wrote:
Well just to be clear then, without resolving this issue (which, IMO, is a bug in cpio that should be fixed) it is absolutely unsafe to ever use pip for non-local installs on fedora.
It might be because I’m still half asleep and trying to wake up, but I’m confused as to what is happening here, pip should never install a egg info file, it always imports setuptools so it always gets an egg-info directory. Are you seeing something different?
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
I think it was clearly explained here:
https://lists.fedoraproject.org/pipermail/devel/2012-February/162518.html
I did not do the research into this explanation - it looks like others have though. I'm just the poor victim who's systems were trashed by this nasty bug, which causes fedora update to abort leaving you with a mess.
On Tue, Mar 25, 2014 at 9:37 AM, Donald Stufft donald@stufft.io wrote:
On Mar 25, 2014, at 9:29 AM, Neal Becker ndbecker2@gmail.com wrote:
Well just to be clear then, without resolving this issue (which, IMO, is
a bug in cpio that should be fixed) it
is absolutely unsafe to ever use pip for non-local installs on fedora.
It might be because I'm still half asleep and trying to wake up, but I'm confused as to what is happening here, pip should never install a egg info file, it always imports setuptools so it always gets an egg-info directory. Are you seeing something different?
Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
python-devel mailing list python-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel
On 03/25/2014 11:42 PM, Neal Becker wrote:
I think it was clearly explained here:
https://lists.fedoraproject.org/pipermail/devel/2012-February/162518.html
I did not do the research into this explanation - it looks like others have though. I'm just the poor victim who's systems were trashed by this nasty bug, which causes fedora update to abort leaving you with a mess.
If it was numpy (as it appears it was), then numpy adds its *own* hackery on top of distutils, and then also tries to detect when setuptools is in use. There's a reason our PyCon packaging panel last year ended up being subtitled "./setup.py install must die" :P
As Toshio notes, there's definitely value in the approach of saying "this dir is for pip, this dir is for the system package manager". Debian/Ubuntu does that by making the system packages install into dist-packages instead of site-packages, but I wonder if we could so some tweaks in site and sysconfig to instead redirect pip to /usr/local.
I'll think about that one a bit more - could be a good thing to hack on at the PyCon US sprints next month.
Cheers, Nick.
python-devel@lists.fedoraproject.org
-
Bohuslav Kabrda -
Donald Stufft -
Neal Becker -
Nick Coghlan -
Robert Kuska