python-devel November 2013

python-devel@lists.fedoraproject.org

4 participants
5 discussions

Python 3.4, ensurepip and wheels
by Bohuslav Kabrda 12 Dec '13

12 Dec '13

Hi all, so I started to work on packaging Python 3.4 beta 1 and I wanted to create ensurepip patch also acceptable for upstream - Toshio and Nick talked about that at [1]. I tried to analyze the situation and here is what I have. First some facts: - with ensurepip, Python 3.4 bundles setuptools and pip wheel archives - when ensurepip is run, it installs (or upgrades) currently installed pip from the bundled wheels A way to solve this "properly" in a distro (as mentioned in [1]) would be: - throw away bundled setuptools and pip wheels when building Python - making python3 require python-pip - patching ensurepip (this is the fun part): -- if ensurepip is run in venv: --- we will need to reconstruct the wheels and install them into venv; or just grab the RECORD files and use them as a list of files we need to copy into venv --- to reconstruct the wheels from installed RPMs, it'd be a good idea to package python-{setuptools,pip} as wheels (with the ".dist-info" directory and all the wheel goodies inside) so that we have the RECORD files --- to package setuptools and pip as wheels, they'll have to BR: python3-pip (I saw Toshio's notes from flock and I agree with the part about how Fedora should use wheels [2]) -- if ensurepip is run systemwide: --- if "--upgrade" is used, should this run "yum update python3-pip"? Now, when new X.Y version Python is introduced, we will have to: - bootstrap the new python3 interpreter to not require python3-pip, since python3-pip will require python(abi)=X.(Y-1) - then we will need to build setuptools and pip *not as wheels*, because we still won't have pip that would work with our new Python X.Y - that means boostrapping them too and building them using the current "setup.py install" approach - when we will have the new pip, we will be able to reenable python3 requires for python3-pip - and we will also be able to rebuild setuptools and pip *as wheels* There are also some more minor issues, that are worth mentioning: - We typically won't update pip after Fedora is released - if there are security issues, we will backport patches and bump the release. The problem with this is that "ensurepip --upgrade" is not able to see Fedora releases and wouldn't work in venv, leaving it vulnerable. I think we have three options here: -- Either we use some Fedora-specific RPM magic, which won't be acceptable for upstream -- Or we will use a nice feature of wheel RECORD files - they keep file hashes, so we can check venv pip against system pip and see if something has changed when "ensurepip --upgrade" is run -- Or we can just copy whole pip again into venv, regardless of what is currently there (doesn't seem very nice..., but it's simple and in the end it works as the above method) - We can't copy the *.pyc and *.pyo files from system to venv, because they have file locations hardcoded in them, that are used in tracebacks. We can solve this by just copying *.py files and running py_compile in our ensurepip. I hope I covered all the important points. Basically, we can make this work in a way acceptable for upstream, if we package setupttols and pip as wheels. It'll require some extra effort, but I think it's worth it. Thoughts? Anyone has better/simpler ideas? Thanks a lot, Slavek. [1] https://lists.fedoraproject.org/pipermail/python-devel/2013-November/000535… [2] https://fedoraproject.org/wiki/User:Toshio/Flock2013_Python_Guidelines#Whee…

3 11

Packaging Python 3.4.0
by Bohuslav Kabrda 29 Nov '13

29 Nov '13

Hi all, I started working on updating python3 to 3.4.0. So far, this is only in a private branch "python3.4" in dist-git, since I'll need to get a Change approved for Fedora 21. I already created the Change [1], but I was told that it will get on FESCo schedule when Fedora 21 schedule and future is more clear, so for now I'll just work on Python 3.4.0. When Python 3.4.0 reaches beta1 (which means feature freeze, which also means that the bytecode format should no longer change) - November 24 [2] - I plan to create a repo in Copr so that everyone has access to built RPMs, and also I want to start rebuilding as many packages as possible to see how they work with Python 3.4. As for the current state of Python 3.4 in the dist-git branch: - It still seems a bit unclear what upstream will do with the sha3 implementation (although they will probably keep it), so I didn't rebase the fips patch yet, there is still plenty of time for that. - pep453 [3] (pip bootstraping) hasn't been implemented yet. The pep also contains recommendation for distro packagers, so here's what we should do. -- Make a circular dependency between python3 and python3-pip (ugly, will require bootstraping with every new Python minor version). -- Definitely delete bundled CA certificates and use system certificates instead. -- Maybe remove the bundled pip (although this goes against the pep) and use python3-pip. E.g. everything would work as expected, but under the wraps, python3-pip package would be used. So when doing security updates, we'd just fix python3-pip. It is however true that if Fedora's pip would be different from the upstream bundled one, users may experience some behaviour differencies. I'd like to hear your opinions on how we should approach this. - I'll be continuously working in the dist-git branch, so if you try it and something doesn't work, I'm just working on it or I plan to :) Patches are welcome. Thanks, Slavek. [1] https://fedoraproject.org/wiki/Changes/Python_3.4 [2] http://www.python.org/dev/peps/pep-0429/ [3] http://www.python.org/dev/peps/pep-0453/

3 5

Changes for F22 -- Python 3 as Default
by Toshio Kuratomi 25 Nov '13

25 Nov '13

The Change page for python3 as default has a pretty empty entry for policies and guidelines. Just, "discuss with FPC"; no list of what to actually change. We should start working on fleshing that out. Here's one that should be fairly easy: * https://fedoraproject.org/wiki/Packaging:Python#Guidelines : Currently reads: If the executables provide the same functionality independent of whether they are run on top of Python 2 or Python 3, then only one version of the executable should be packaged. Currently it will be the python 2 implementation, but once the Python 3 implementation is proven to work, the executable can be retired from the python 2 build and enabled in the python 3 package. Be sure to test the new implementation. Transitioning from python2 to python3 is left to individual package maintainers except for packages in Fedora's critical path. For these, we want to port to python3 versions in the same Fedora release if possible. Change that to: If the executables provide the same functionality independent of whether they are run on top of Python 2 or Python 3, then only one version of the executable should be packaged. In F22 and later, this should be the Python 3 version. In F21 and earlier this is left to the individual package maintainers '''except''' for packages in Fedora's critical path. Those should remain on the Python 2 version so that we don't drag in both python2 and python3 stacks on a minimal system. I think this change could go into FPC's queue to approve now because the differences are all per-version so it does not causes problems for packages which are not yet thinking about F22. @churchyard, as part of this we should probably come up with another list of packages that do not have python3 versions for the packages in the critical path. (btw, I'm much happier with the text of the latest round of bug reports, thanks). (Note though -- I think fesco still likes us to pass anything we're going to mass file bugs for to them (or to the devel mailing list?) to review first because mass filings can affect a lot of people. Final note: There were three changes to the python guidelines approved by FPC in this ticket: https://fedorahosted.org/fpc/ticket/327#comment:9 Unfortunately, both bkabrda and I have been pulled away to work on SCL drafts so the three changes have not been written up. If someone would be willing to write up those changes as a new version of the guidelines page:: http://fedoraproject.org/wiki/Packaging:Python I'd be able to make time to review that and merge it into the Guidelines. Thanks, -Toshio

3 3

python-setuptools update in F20
by Toshio Kuratomi 07 Nov '13

07 Nov '13

A new setuptools upstream release (1.2) was made last week that adds support for subversion 1.7 and 1.8. Subversion 1.7 shipped in Fedora 19 so the subversion support in setuptools is currently broken in both F19 and F20. Unless there's objections I'm definitely going to update setuptools in F20. I'm quite hesitant about updating in F19 due to potential incompatibilities. In particular, pip, buildout, and virtualenv would likely all need to update (these were things that we found had minimum versions to work with the F20 setuptools package as per: https://fedoraproject.org/wiki/Changes/Python_setuptools_0.7 ) Backwards incompatible changes were introduced between 0.9.8 and 1.0: https://pypi.python.org/pypi/setuptools#id145 but these changes are unlikely to affect any code that's currently in the wild. Please speak now if you don't want this update to hit F20 or do want the update to hit F19. My plan is to update the python-setuptools package in F20 to 1.3 near the end of this week. -Toshio

1 1

Removing python-setuptools-devel backwards compat
by Toshio Kuratomi 04 Nov '13

04 Nov '13

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel November 2013