At the present time, running sudo pip3 in Fedora is not safe. Pip shares its installation directory with dnf, can remove dnf-managed files and generally break the Python 3 interpreter.
Our first attempt to make sudo pip safe on Fedora [0] was based on using a different binary (/usr/libexec/system-python) for RPM packaging purposes and changing the behavior of /usr/bin/python3 (and pip that uses its settings) to install under /usr/local by default. Switching all the Python 3 packages to system-python turned out to be much more problematic than expected and we had to postpone the Change for F27.
We decided to try a different approach. The main idea is to detect an ongoing RPM build and modify the behavior of the Python 3 executable only when in normal user environment so that RPM builds won't be affected at all.
The adjustment of the behavior can be done on different levels. The first option is to set the sys.prefix variable to /usr/local when the interpreter is initialized. This will affect all the install methods, but the solution can cause some problems in applications that rely on the value of sys.prefix. A prototype of this implementation can be seen here [1].
The other possibility is to limit the pip install location change to distutils and pip [2]. This is the "safer" option, but does not cover all corner cases. For example, Python software built locally using cmake or similar tools will be installed into /usr/lib and can conflict with system tools. Debian chose to implement similar solution.
I would like to ask which solution would work for your applications better, and what is in your opinion the right way to go. I will appreciate any costructive comments or ideas on how to improve these patches. If you want to check some specific use-case you can use the images docker.io/mcyprian/sudo_pip_sys_prefix and docker.io/mcyprian/sudo_pip_install_prefix to try out the two mentioned implementations.
[0] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe?rd=User:Torsava/... [1] https://github.com/mcyprian/python3/commit/6077c96460c03ddc334bae654fffcce44... [2] https://github.com/mcyprian/python3/commit/ac2f266a90871d67ba8d0b70d2b0ebe47...
Michal Cyprian
Forwarding to fedora-devel as well
Charalampos Stratakis Associate Software Engineer Python Maintenance Team, Red Hat
----- Original Message ----- From: "Michal Cyprian" mcyprian@redhat.com To: python-devel@lists.fedoraproject.org Sent: Wednesday, April 26, 2017 11:19:00 AM Subject: What is your opinion on "sudo pip" fix for Fedora 27?
At the present time, running sudo pip3 in Fedora is not safe. Pip shares its installation directory with dnf, can remove dnf-managed files and generally break the Python 3 interpreter.
Our first attempt to make sudo pip safe on Fedora [0] was based on using a different binary (/usr/libexec/system-python) for RPM packaging purposes and changing the behavior of /usr/bin/python3 (and pip that uses its settings) to install under /usr/local by default. Switching all the Python 3 packages to system-python turned out to be much more problematic than expected and we had to postpone the Change for F27.
We decided to try a different approach. The main idea is to detect an ongoing RPM build and modify the behavior of the Python 3 executable only when in normal user environment so that RPM builds won't be affected at all.
The adjustment of the behavior can be done on different levels. The first option is to set the sys.prefix variable to /usr/local when the interpreter is initialized. This will affect all the install methods, but the solution can cause some problems in applications that rely on the value of sys.prefix. A prototype of this implementation can be seen here [1].
The other possibility is to limit the pip install location change to distutils and pip [2]. This is the "safer" option, but does not cover all corner cases. For example, Python software built locally using cmake or similar tools will be installed into /usr/lib and can conflict with system tools. Debian chose to implement similar solution.
I would like to ask which solution would work for your applications better, and what is in your opinion the right way to go. I will appreciate any costructive comments or ideas on how to improve these patches. If you want to check some specific use-case you can use the images docker.io/mcyprian/sudo_pip_sys_prefix and docker.io/mcyprian/sudo_pip_install_prefix to try out the two mentioned implementations.
[0] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe?rd=User:Torsava/... [1] https://github.com/mcyprian/python3/commit/6077c96460c03ddc334bae654fffcce44... [2] https://github.com/mcyprian/python3/commit/ac2f266a90871d67ba8d0b70d2b0ebe47...
Michal Cyprian _______________________________________________ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-leave@lists.fedoraproject.org
On 26 April 2017 at 19:19, Michal Cyprian mcyprian@redhat.com wrote:
The other possibility is to limit the pip install location change to distutils and pip [2]. This is the "safer" option, but does not cover all corner cases. For example, Python software built locally using cmake or similar tools will be installed into /usr/lib and can conflict with system tools. Debian chose to implement similar solution.
I would like to ask which solution would work for your applications better, and what is in your opinion the right way to go.
From my perspective, the main goal of the change is to make it possible to fully recover from "sudo pip install <package>" by doing "sudo pip uninstall <package>", and the approach Debian took is sufficient to achieve that in almost all cases. So +1 from me for replicating something similar to Debian's approach in Fedora rather than blazing new trails by fiddling with `sys.prefix` directly.
For the latter concern with CMake/autotools/Scons/etc, anyone doing local builds of C/C++ applications already needs to be careful not to be interfere with system packages, so I'm comfortable leaving that in the category where running C/C++ build system install commands outside a package build process without due care and attention is a generally bad idea (e.g. a "sudo make altinstall" of the Python 3.5 maintenance branch will clobber the system Python binaries in F25, and a "sudo make install" from the main Python development branch will mess with the default symlinks if you didn't set a suitable `/opt/` prefix).
Cheers, Nick.
On Apr 27, 2017, at 04:52 PM, Nick Coghlan wrote:
From my perspective, the main goal of the change is to make it possible to fully recover from "sudo pip install <package>" by doing "sudo pip uninstall <package>", and the approach Debian took is sufficient to achieve that in almost all cases. So +1 from me for replicating something similar to Debian's approach in Fedora rather than blazing new trails by fiddling with `sys.prefix` directly.
All of Debian's deltas from upstream pip are here:
https://anonscm.debian.org/cgit/python-modules/packages/python-pip.git/tree/...
We also have a somewhat more elaborate machinery to unbundle packages using a tool called dirtbike to turn installed (at build time) packages into wheels. That's a separate issue I'd be happy to talk more about.
In any case, to the extend that we can upstream the more generally useful bits, or at least share behavior so users will have a similar experience on both ecosystems, I'd be happy to help represent Debian. Probably not much will change there until after the upcoming release though, since the archive is currently frozen.
Cheers, -Barry
Fedora's equivalent to dirtbike is rewheel [0][1] and it would be great if these projects could be unified somehow, however that's a separate conversation for another time.
I'd be happy to take a closer look at dirtbike and maybe initiate some discussions when I get some free time.
[0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/plain/00189-add-rewheel-... [1] https://github.com/fedora-python/rewheel (kinda outdated)
Charalampos Stratakis Associate Software Engineer Python Maintenance Team, Red Hat
----- Original Message ----- From: "Barry Warsaw" barry@python.org To: python-devel@lists.fedoraproject.org Sent: Thursday, April 27, 2017 5:27:55 PM Subject: Re: What is your opinion on "sudo pip" fix for Fedora 27?
On Apr 27, 2017, at 04:52 PM, Nick Coghlan wrote:
From my perspective, the main goal of the change is to make it possible to fully recover from "sudo pip install <package>" by doing "sudo pip uninstall <package>", and the approach Debian took is sufficient to achieve that in almost all cases. So +1 from me for replicating something similar to Debian's approach in Fedora rather than blazing new trails by fiddling with `sys.prefix` directly.
All of Debian's deltas from upstream pip are here:
https://anonscm.debian.org/cgit/python-modules/packages/python-pip.git/tree/...
We also have a somewhat more elaborate machinery to unbundle packages using a tool called dirtbike to turn installed (at build time) packages into wheels. That's a separate issue I'd be happy to talk more about.
In any case, to the extend that we can upstream the more generally useful bits, or at least share behavior so users will have a similar experience on both ecosystems, I'd be happy to help represent Debian. Probably not much will change there until after the upcoming release though, since the archive is currently frozen.
Cheers, -Barry
_______________________________________________ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-leave@lists.fedoraproject.org
On Apr 27, 2017, at 12:01 PM, Charalampos Stratakis wrote:
Fedora's equivalent to dirtbike is rewheel [0][1] and it would be great if these projects could be unified somehow, however that's a separate conversation for another time.
I'd be happy to take a closer look at dirtbike and maybe initiate some discussions when I get some free time.
+1
-Barry
On 28 April 2017 at 02:01, Charalampos Stratakis cstratak@redhat.com wrote:
Fedora's equivalent to dirtbike is rewheel [0][1] and it would be great if these projects could be unified somehow, however that's a separate conversation for another time.
I'd be happy to take a closer look at dirtbike and maybe initiate some discussions when I get some free time.
My recollection from the last time I looked at this was that where rewheel assumes that the Python level PEP 376 package metadata will be available, dirtbike instead queried the operating system level metadata.
So while they're technically solving the same problem (injecting a system level Python package or bindings into a virtual environment), they took radically different paths to get there.
The biggest benefit I see to the dirtbike approach is that it has the potential to work with Python bindings that aren't built with the standard Python packaging tools, and hence don't have any PEP 376 metadata.
Conversely, the biggest benefit I see to the rewheel approach is that it lets system packages more clearly indicate whether or not they want to support rewheel by including or omitting the PEP 376 metadata.
Cheers, Nick.
On Apr 28, 2017, at 11:50 AM, Nick Coghlan wrote:
My recollection from the last time I looked at this was that where rewheel assumes that the Python level PEP 376 package metadata will be available, dirtbike instead queried the operating system level metadata.
dirtbike consults a list of "strategies" of which the PEP 376 metadata strategy is the top priority. If that fails, then it does fall back to dpkg-based strategies.
Why might the PEP 376 strategy fail? There are cases in Debian where we split upstream source packages into multiple binary packages. So for example, the pkg_resources binary package is separate from setuptools, so if we want to rewheel pkg_resources, there won't be PEP 376 metadata associated with it.
All of the existing fall back strategies exist to handle real-world corner cases. The intent (if not the reality, 'cause it hasn't been done yet ;) is that dirtbike is extensible for systems where different fall backs would be necessary. But if the PEP 376 metadata can be found, and it provides what's needed, that would be good enough.
(Some of the testing framework is probably Debian-centric, but I think that's also tractable.)
The biggest benefit I see to the dirtbike approach is that it has the potential to work with Python bindings that aren't built with the standard Python packaging tools, and hence don't have any PEP 376 metadata.
Yep, and it should be adaptable to finding that non-standard metadata by consulting any alternative system databases.
Conversely, the biggest benefit I see to the rewheel approach is that it lets system packages more clearly indicate whether or not they want to support rewheel by including or omitting the PEP 376 metadata.
I should also mention that according to Debian policy, we're tightly focused on rewheeling just the small number of packages to make pip and virtualenv/venv work. Not that it couldn't be used for other things, just that only those dependencies are allowed to ship .whl files in debs.
https://github.com/paulproteus/dirtbike
Cheers, -Barry
On 29 April 2017 at 01:51, Barry Warsaw barry@python.org wrote:
On Apr 28, 2017, at 11:50 AM, Nick Coghlan wrote:
My recollection from the last time I looked at this was that where rewheel assumes that the Python level PEP 376 package metadata will be available, dirtbike instead queried the operating system level metadata.
dirtbike consults a list of "strategies" of which the PEP 376 metadata strategy is the top priority. If that fails, then it does fall back to dpkg-based strategies.
Oh, nice - either it didn't do that yet when I last looked at it (which was literally years ago), or else I simply missed that it tried to read the PEP 376 metadata before falling back to dpkg.
In that case, the most promising direction for future consolidation is likely to be figuring out a way to reimplement rewheel as an interface shim on top of dirtbike.
Cheers, Nick.
On 26 April 2017 at 19:19, Michal Cyprian mcyprian@redhat.com wrote:
The other possibility is to limit the pip install location change to distutils and pip [2]. This is the "safer" option, but does not cover all corner cases. For example, Python software built locally using cmake or similar tools will be installed into /usr/lib and can conflict with system tools. Debian chose to implement similar solution.
Re-reading https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe after the discussion on the main devel list, I have a few further comments on this point:
1. I think the tone of the change proposal should be softened to say it's about make "sudo pip install" *safer*, rather than about making it absolutely safe - you can still break your system if you're not careful, the layout change just makes it easier to recover if you do (at least in most cases) 2. The question of installation directories other than site-packages (mostly notably /usr/bin) should be explicitly covered, and either also moved under /usr/local, or else a rationale given for not moving them yet 3. The proposal should be more explicitly scoped by specifying some sequences of operations that you want to make safe, such as:
$ sudo pip3 install contextlib2 && sudo pip3 uninstall contextlib2 $ sudo pipsi install ansible && sudo pipsi uninstall ansible
(Getting the latter example to be safe would require changing the install directory to /usr/local in a way that "pipsi" detects)
Cheers, Nick.
python-devel@lists.fedoraproject.org
-
Barry Warsaw -
Charalampos Stratakis -
Michal Cyprian -
Nick Coghlan