On 2 May 2017 at 09:36, Nico Kadel-Garcia nkadel@gmail.com wrote:

On Mon, May 1, 2017 at 9:14 AM, Nick Coghlan ncoghlan@gmail.com wrote:

...

On 1 May 2017 at 22:47, Nico Kadel-Garcia nkadel@gmail.com wrote:

...

On Sun, Apr 30, 2017 at 10:30 PM, Nick Coghlan ncoghlan@gmail.com wrote:

...

If the intended benefit of this change remains unclear, it may help to focus on a specific concrete case, which would be that the following operations should be completely indistinguishable at the system level from not having done anything at all (except in the sudo logs):

$ sudo pip3 install contextlib2
$ sudo pip3 uninstall contextlib2

And this successfully ignores *all* the dependencies which contextlib2 may add at installation time.

There's a reason I chose contextlib2 as my example rather than something more complex like ansible - I'm the maintainer of contextlib2, and I know it doesn't have any runtime dependencies (and likely never will, since it's a stdlib module backport).

Heh: it seems as if you are cheating somewhat by choosing one of the easiest components. I'm concerned that casual use of a new "pip3 install" localized utility will replicate the problems I described fairly quickly, and in ways that someone not as experienced with dependency violations will have difficulty cleaning up. I'm particularly thinking of "awscli" and the various Sphinx version dependencies of its dependency tree.

Nico, this suggests you're still missing the essential point of the proposal.

Status quo: people run "sudo pip3 install <whatever>" and *corrupt their system (or SCL) packages*. Remediation in the case of system breakage will be some form of clearing out the pip installed components, and reinstalling the dnf/yum managed ones.

Proposed change: people run "sudo pip3 install <whatever>" and only `/usr/local` (or the equivalent `/opt/...` directory for SCLs) gets modified. This will often "just work" (when there's no conflict with system packages), and even when it doesn't, remediation only involves removing the pip installed packages - the dnf/yum managed ones should still be in their original pristine state.

Saying "This only mitigates the potential problems, it doesn't completely eliminate them" doesn't make sense as a response to a harm mitigation measure.

That said, I re-read the original change proposal page, and that *does* give the impression that it completely solves the problem (since even the title uses words like "safe" rather than the more measured "safer"), so I've suggested on the Python SIG list that it be reworded to better explain the scope and potential benefits.

...

Given the proposal at hand though, writing a `remove-pip-installed-modules` cleaner utility becomes a lot simpler that it is today, since it just needs to clear out any Python packages it finds in /usr/local (based on the Python level installation records), rather than needing to interact with the RPM database, figure out which system packages may have potentially been corrupted and reinstall them.

That makes sense, at least to me. I continue to urge you to keep it *out* of the default compiled in equivalents of PYTHONPATH. Perhaps there needs to be something like a "pip3.local", to activate such such modules in a better segregated space?

That's what per-user and venv installs are for, as well as SCLs and containers at the OS level.

One thing I would like to start doing is more strongly encouraging the use of `pipsi` ("pip script installer") when installing things like awscli directly from PyPI for personal or system wide use, since that automatically takes care of creating an application specific virtual environment for the tool's dependencies.

However, there are still some issues with that around availability (pipsi isn't in the Python SCLs or EPEL at this point), as well as around readily upgrading components in the per-application venvs.

Cheers, Nick.