On Dec 10, 2013 6:26 AM, "Bohuslav Kabrda" bkabrda@redhat.com wrote:

----- Original Message -----

...

On 11/28/2013 12:42 AM, Bohuslav Kabrda wrote:

...

I hope I covered all the important points. Basically, we can make

this work

...

...

in a way acceptable for upstream, if we package setupttols and pip as wheels. It'll require some extra effort, but I think it's worth it. Thoughts? Anyone has better/simpler ideas?

From an upstream point of view, so long as test.test_ensurepip and test.test_venv still work, things should generally be OK.

I quite like the idea of checking for the consistency of the RECORD files between the system pip and the one in the virtualenv (as well as using RECORD as a guide to what to copy into a fresh venv). If you get that working, I'd be interested in a Python 3.5 venv and/or ensurepip patch to do that by default, and only bootstrap from the embedded wheel if there was no system pip available.

Cheers, Nick.

Actually, there seems to be a much simpler way of doing this in Fedora

(and any distro more generally):

• setuptools and pip RPMs will carry the wheel inside them and drop it

into ensurepip/_bundled

• the wheels will be rebuilt during every RPM build everytime *after

patching*, so they will carry security patches etc.

• we will use the RPM release as the "build tag" mentioned in PEP 427

[1], so that when we e.g. fix a security bug but don't bump the version, "ensurepip --upgrade" will still see that the wheel has to be reinstalled (otherwise it'd say think the version is already there and wouldn't reinstall)

So the only thing we will need to implement will be autodiscovery of the

wheels, since they will change names independently on python3 package, but I think we can do that :) From upstream point of view this shouldn't break anything, but it'd also probably not have any benefit. Would you still accept such patch?

Note that this misses one thing mentioned in my mail: local changes.

Because scripting languages are readable and modifiable, sysadmins will sometimes fix bugs that affect them by deploying just the changed .py files (rather than rebuilding an rpm package). Configuration management (puppet, Ansible, chef, etc) makes it easy to deploy and track hotfixes this way. If wheels are cached at rpm buildtime then this workflow will cease to function correctly for virtualenv and these bundled libraries. Since the bundling is an implementation detail that sysadmins will have no reason to know about until they want to hotfix whichever libraries are bundled, this is likely to take them a while to figure out.

Creating the wheels from the files on the system keeps the number of copies of files to a minimum. That, in turn, makes it easier for everyone involved to discover which files are the ones that do something.

-Toshio

Regards, Slavek.

[1] http://www.python.org/dev/peps/pep-0427/#id11 _______________________________________________ python-devel mailing list python-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel

On 12/11/2013 01:28 AM, Bohuslav Kabrda wrote:

> Actually, there seems to be a much simpler way of doing this in Fedora (and any distro more generally):
> - setuptools and pip RPMs will carry the wheel inside them and drop it into ensurepip/_bundled
> - the wheels will be rebuilt during every RPM build everytime *after patching*, so they will carry security patches etc.
> - we will use the RPM release as the "build tag" mentioned in PEP 427 [1], so that when we e.g. fix a security bug but don't bump the version, "ensurepip --upgrade" will still see that the wheel has to be reinstalled (otherwise it'd say think the version is already there and wouldn't reinstall)
> So the only thing we will need to implement will be autodiscovery of the wheels, since they will change names independently on python3 package, but I think we can do that :) From upstream point of view this shouldn't break anything, but it'd also probably not have any benefit. Would you still accept such patch?
>

Note that this misses one thing mentioned in my mail: local changes.

Because scripting languages are readable and modifiable, sysadmins
will sometimes fix bugs that affect them by deploying just the
changed .py files (rather than rebuilding an rpm package). 
Configuration management (puppet, Ansible, chef, etc) makes it easy
to deploy and track hotfixes this way.  If wheels are cached at rpm
buildtime then this workflow will cease to function correctly for
virtualenv and these bundled libraries.  Since the bundling is an
implementation detail that sysadmins will have no reason to know
about until they want to hotfix whichever libraries are bundled,
this is likely to take them a while to figure out.

The upstream solution has this problem, too. In fact, this approach will be much closer to the upstream solution in the sense that it uses prebuilt wheels to install/update virtualenv. The wheels will just come from different packages than python3 itself. IMO it'd be sufficient to place a README file into the ensurepip/_bundled dir that would explain this. Actually, diverging from upstream in this way is an argument for not doing the "files copying" solution now.

Well, we only went for the full bundling solution upstream as an expedient hack that we *knew* we could get working in time for Python 3.4. I'd actually prefer it if, given a system install of pip (and setuptools), ensurepip bootstrapped those into new environments rather than the bundled ones.

At the moment, if you do "pip install --upgrade pip" in your system Python (regardless of platform), future virtual environments will still get the old pip until the next CPython maintenance release. Now, in theory, there will be new CPython releases shortly after each new pip release, but going to a new CPython maintenance release is a higher impact change than just updating pip, so we have to expect substantial lags between pip updates and CPython updates in many environments.

While trying to create the patch that would rebuild the wheels from system and install them in venv, I bumped into few ugly problems, which is why I came up with this "wheels in RPMs" solution in the first place. I'd like to do this for 3.4 in Fedora and then start working on the "file copying" solution that would be acceptable for upstream Python 3.5. When this is done upstream, we will move to there with 3.5.

Alas, it sounds like the pip folks were right when they said "we're not sure that can actually work at this point" (something I hadn't thought about previously: at the very least, the install time shebang line rewriting and wrapper script generation will break. There's also the problem of *which* scripts get installed, since a system pip for Python 3 likely won't have the bare "pip" command, but we *do* want that in a virtual environment).

Creating the wheels from the files on the system keeps the number of
copies of files to a minimum.  That, in turn, makes it easier for
everyone involved to discover which files are the ones that do
something.

So, here's a crazy thought: what if, rather than copying the installed files directly into the virtual environment, we reverse engineered a wheel archive *dynamically* from the system install and then installed from that? That would avoid the problems with trying to bypass pip's script generation, while still bootstrapping new virtual environments based on the installed versions of pip and setuptools.

Cheers, Nick.

On 12/11/2013 04:52 PM, Bohuslav Kabrda wrote:

----- Original Message -----

...

...

While trying to create the patch that would rebuild the wheels from system and install them in venv, I bumped into few ugly problems, which is why I came up with this "wheels in RPMs" solution in the first place. I'd like to do this for 3.4 in Fedora and then start working on the "file copying" solution that would be acceptable for upstream Python 3.5. When this is done upstream, we will move to there with 3.5.

Alas, it sounds like the pip folks were right when they said "we're not sure that can actually work at this point" (something I hadn't thought about previously: at the very least, the install time shebang line rewriting and wrapper script generation will break. There's also the problem of *which* scripts get installed, since a system pip for Python 3 likely won't have the bare "pip" command, but we *do* want that in a virtual environment).

...

Creating the wheels from the files on the system keeps the number of
copies of files to a minimum.  That, in turn, makes it easier for
everyone involved to discover which files are the ones that do
something.

So, here's a crazy thought: what if, rather than copying the installed files directly into the virtual environment, we reverse engineered a wheel archive *dynamically* from the system install and then installed from that? That would avoid the problems with trying to bypass pip's script generation, while still bootstrapping new virtual environments based on the installed versions of pip and setuptools.

There is a slight problem here: In a stable Fedora release, we typically won't change pip's version, but rather backport security patches and the version will stay the same. So if you reconstruct the wheel from that, running "ensurepip --upgrade" in virtualenvs won't actually do anything, since pip will see the same version in virtualenv as is in the system and do nothing. There are few possible solutions to this:

• if version in system is the same as in virtualenv, check file checksums and if some differ, reinstall
• if version in system is the same as in virtualenv, uninstall pip in virtualenv and then reinstall the one from the system
• (ugly, not upstream acceptable) do "rpm -q python3-pip" and use the Fedora release as a "build tag" for reconstructed wheel (similar to what I proposed in the "bundled wheels" solution)

If upstream (you :)) accepts any of these types of behaviour (the second one seems the easiest and should work fine) in the PEP, I'll work on that. Quite literally this time, we would reinvent the wheel :)

Ah, true, this is the thing we're planning to fix by formally adding local version numbering as part of the upstream data model in metadata 2.0 (that is, pip, setuptools, etc would actually be *required* to cope properly if the version metadata contained a number like "1.5-2" rather than just "1.5", so that patched downstream rebuilds can have different version numbers in the metadata than vanilla upstream versions). (This is already in the draft metadata spec, but I haven't quite worked through all the implications for version pinning yet)

You may even want to try that out (patching the pip and setuptools version numbers to include a release suffix) and see if it already works, since the scheme we're making official in metadata 2.0 is based on the way pkg_resources.parse_version() already behaves:

...

...

pkg_resources.parse_version("1.5") < pkg_resources.parse_version("1.5-1") < pkg_resources.parse_version("1.5-2")

True

Failing that, I think the uninstall/reinstall trick (preferably combined with some kind of marker so it only happens when there *is* an upgrade to be installed) sounds like a reasonable workaround until we get metadata 2.0 formalised upstream (which will hopefully happen before pip 1.6 is released mid next year).

Cheers, Nick.

On 12/12/2013 01:18 AM, Bohuslav Kabrda wrote:

Well yeah, my point is that there is no upstream-acceptable way other than checking the file hashes by ensurepip, is there? If I wouldn't want to check file hashes, I'd have to query RPM for release - or is there some other way you're thinking of?

I think doing it initially as a downstream only change where you query RPM will work for now (perhaps by patching the way pip handles the case where ENSUREPIP_OPTIONS is set?).

By the time this approach is posted upstream, then PEP 426/440 will hopefully by Final and we can just use the metadata version field directly rather than needing to grab the release increment from the RPM repo. (I think this situation provides a good practical use case for why it's important to standardise this feature upstream, too).

Cheers, Nick.

On 12/12/2013 11:09 PM, Bohuslav Kabrda wrote:

----- Original Message -----

...

On 12/12/2013 01:18 AM, Bohuslav Kabrda wrote:

...

Well yeah, my point is that there is no upstream-acceptable way other than checking the file hashes by ensurepip, is there? If I wouldn't want to check file hashes, I'd have to query RPM for release - or is there some other way you're thinking of?

I think doing it initially as a downstream only change where you query RPM will work for now (perhaps by patching the way pip handles the case where ENSUREPIP_OPTIONS is set?).

By the time this approach is posted upstream, then PEP 426/440 will hopefully by Final and we can just use the metadata version field directly rather than needing to grab the release increment from the RPM repo. (I think this situation provides a good practical use case for why it's important to standardise this feature upstream, too).

So, getting back to Toshio's concern about sysadmins who just update'n'upload files, the workflow for them would be "update files, bump build tag and then upload", right?

What I mean is, this still has two solutions:

• checking the build tag (seems to be very simple to do)
• check the file hashes

Both solutions are IMO acceptable upstream (when we can actually do build tags), but my question is: Which one is more likely to be accepted? I'm for "checking the build tag", even if it means the extra step for sysadmins who will need to bump the build tag when doing changes. (We may need to tell them to not bump the build tag "major number", but add something to it like "1" => "1.1", since we want distro package with "2" win over sysadmin's change.)

Does that make sense? What I'd love to hear is which of the two approaches is more likely to get accepted upstream, so that I can concentrate on that one approach.

I can't speak for the pip devs, but I think either would be acceptable from an ensurepip point of view. That means the decision comes down to how we want to deal with the "patch source in place" admins creating their snowflake servers that can't easily be rebuilt from scratch:

- checking file hashes would mean that directly patching the system pip not only affects future virtual environments, but also "python -m ensurepip --upgrade" in existing virtual environments - relying on the build tag would mean that directly patching the system pip would affect future virtual environments, but *not* "python -m ensurepip --upgrade" in existing virtual environments

I think the latter limitation is fine, and more in line with the direction we're heading upstream (with the "local" suffix on build numbers), so unless Toshio thinks we absolutely *have* to provide the former behaviour, +1 for the build tag approach from me.

Cheers, Nick.