On Tue, Nov 05, 2013 at 07:29:35AM -0500, Bohuslav Kabrda wrote:

Hi all, I started working on updating python3 to 3.4.0. So far, this is only in a private branch "python3.4" in dist-git, since I'll need to get a Change approved for Fedora 21. I already created the Change [1], but I was told that it will get on FESCo schedule when Fedora 21 schedule and future is more clear, so for now I'll just work on Python 3.4.0.

Hmmm.... Putting on a fesco hat, I would be more than willing to approve changes for F21 now.... We've already approved Python3 as Default which has both F21 and F22 goals.

I think that the F21 Schedule and Product desciptions are somewhat orthogonal to some features including this. The things I'm looking at that imply that are:

* Something that either is non-specific to one of our products * Something that is an upgrade of packages already in Fedora

So in my view -- submit the Change! :-)

When Python 3.4.0 reaches beta1 (which means feature freeze, which also means that the bytecode format should no longer change) - November 24 [2]

• I plan to create a repo in Copr so that everyone has access to built

RPMs, and also I want to start rebuilding as many packages as possible to see how they work with Python 3.4.

<nod> The rebuilding make me want to get this into the main repos early.

As for the current state of Python 3.4 in the dist-git branch:

• It still seems a bit unclear what upstream will do with the sha3 implementation (although they will probably keep it), so I didn't rebase the fips patch yet, there is still plenty of time for that.

I think Christian outlined what they are going to do with sha3 yesterday (but yeah, it depends on what is happening with the sha3 standard so there's a bit of uncertainty there still).

• pep453 [3] (pip bootstraping) hasn't been implemented yet. The pep also contains recommendation for distro packagers, so here's what we should do.

-- Make a circular dependency between python3 and python3-pip (ugly, will require bootstraping with every new Python minor version).

Do you mean minor version as in the "4" in 3.4 or as in the micro level ("0" in 3.4.0)?

We can get around bootstrapping at minor version revs if pip is only used at runtime, not at buildtime.

We can still get around bootstrapping at micro version if we are careful never to include pip's code in the package. Only figure out where to find pip's code.

-- Definitely delete bundled CA certificates and use system certificates instead.

+1

-- Maybe remove the bundled pip (although this goes against the pep) and use python3-pip. E.g. everything would work as expected, but under the wraps, python3-pip package would be used. So when doing security updates, we'd just fix python3-pip. It is however true that if Fedora's pip would be different from the upstream bundled one, users may experience some behaviour differencies. I'd like to hear your opinions on how we should approach this.

We should definitely do this. pip itself bundles code that we've had to unbundle locally -- maintaining that morass of software again (it's also bundled inside of virtualenv...) is going to lead to a lot of work anytime an update needs to happen.

However, talking with ncoghlan it may not be simple. The case that makes things tricky is pyvenv. Upstream python wants to protect venv's from changes to code on the system. ie: if a user installs a backwards incompatible pip on the system, that should not affect the pip inside the existing venvs. Equally, if there are security issues found with pip, those changes would not be propogated to the pip inside the venvs.

ncoghlan proposed that we create wheels in the pip package (and all the packages that pip bundles that we have to unbundle) and then when ensurepip installs into a venv, it copies those wheels in there.

I'm wondering if we could do a little better. Instead of keeping around two copies of all this software on the system, if we can reconstruct a wheel zip (or possibly just copy the files in) from the wheel metadata and files on the system. I know that sysadmins and developers will hotfix software on the system for issues that they care about. Having those changes propogate to newly installed venvs seems like a better plan than having the venvs get the old copies that were built at rpm build time.

-Toshio

Update: - DONE: I've just created a copr repo for Python 3.4 [1] and built beta1 - IN PROGRESS: I'm now working on unbundling of pip and setuptools wheels, as mentioned in thread [2]. Progress: -- DONE: created python-wheels package for Fedora (I'll build it in copr ASAP) -- DONE: added logic to python-setuptools and python-pip so that they can be built as wheels (I'll build in copr ASAP) -- TODO: Now that I have an idea how to build wheel RPMs, I need to: --- wait to see if my patches to pip that enable this get accepted [3], [4] (and perhaps communicate with upstream to make them better) --- once the patches are accepted, I'd like to talk to FPC about allowing building wheel RPMs (most likely from Fedora 21 on) and how to do that --- once that is discussed, build setuptools and pip as wheels for F21 (this is really necessary, so even if Fedora as a whole distro doesn't accept wheels usage yet, I'll try to negotiate exception for these two packages) -- TODO: in the meanwhile, I'll start working on the actual ensurepip patch and I'll propose it upstream - TODO: also in the meanwhile, I'll be updating Python 3.4 to the newest prereleases, as they are released by upstream

If anyone wants to help, you can: - help push my patches to pip upstream - help with creating a proposal about building wheel RPMs for FPC (have a look at the SRPMs in the copr repo - once I've built them - to see how I'm currently doing it) - help with creating the ensurepip patch (if you want to work on any of these, please let everyone know, so that we don't duplicate our efforts)

Thanks a lot, Slavek.