python-devel May 2010

python-devel@lists.fedoraproject.org

6 participants
7 discussions

Fedora Python features
by David Malcolm 28 May '10

28 May '10

I've added a list of Python-related features to our wiki page here: https://fedoraproject.org/wiki/SIGs/Python#Python_Features Is anyone else working on Python-related Fedora features? In particular, I've added the three that I'm looking at (so far...): * the debug builds that I've mentioned on this list [1] * upgrading python from 2.6 to 2.7 (and rebuilding the world) [2], which I think we'll want to do soon: 2.7 beta 2 is out, with the final release scheduled by upstream for 2010-07-03, well within F-14's schedule. * upgrading python3 from 3.1 to 3.2 [3]. This is more uncertain, as the upstream schedule doesn't line up so well with ours. We could potentially package an alpha release I guess. python3 isn't on the critical path, and the number of packages needing a rebuild is _much_ smaller than for Python 2, but I'd want to talk to upstream about it. By comparison, Ubuntu's next release has a slightly later feature freeze than ours, and is planning to ship a beta of 3.2 ([4] and [5]). Thoughts? Dave [1] https://fedoraproject.org/wiki/Features/DebugPythonStacks [2] https://fedoraproject.org/wiki/Features/Python_2.7 [3] https://fedoraproject.org/wiki/Features/Python_3.2 [4] https://wiki.ubuntu.com/MaverickMeerkat/TechnicalOverview/Python [5] http://mail.python.org/pipermail/python-dev/2010-May/100165.html

2 3

Debug Python stacks revisited - experimental build in Rawhide, targetting Fedora 14
by David Malcolm 26 May '10

26 May '10

(This is a scaled-back version of a proposal I sent to this list a couple of months ago [1]) There are various configuration flags that can be used when building Python. Currently we have a configuration aimed at the typical use-case: as much optimization as reasonable. However, upstream Python supports a number of useful debug options which use more RAM and CPU cycles, but make it easier to track down bugs [2] Typically these are of use to people working on Python C extensions, for example, for tracking down awkward reference-counting mistakes. I've had at least three developers whose opinion I value very highly ask me for these (for example John Palmieri is currently working on the PyGI stack, and is running into difficult reference-counting issues). Indeed, Debian and Ubuntu have had these alternate builds available for a couple of years now. [3] I've looked through Debian's patch [4], and come up with a somewhat modified version that does mostly the same thing, though (I think) somewhat better fitting our build process. The python.spec now configures and builds, and installs the python sources twice, once with the regular optimized settings, and again with debug settings. (in most cases the files are identical between the two installs, and for the files that are different, they get separate paths) I've been testing with this on my machine and it works fine; I've also been able to successfully use distutils to build extension modules So I've decided to try this in Rawhide for F-14; the latest build is here: http://koji.fedoraproject.org/koji/buildinfo?buildID=174357 The relevant CVS commit is here: http://cvs.fedoraproject.org/viewvc/rpms/python/devel/python.spec?r1=1.184&… http://cvs.fedoraproject.org/viewvc/rpms/python/devel/python-2.6.5-debug-bu… and the specfile comment contains more detailed implementation notes. The builds are set up so that they can share the same .py and .pyc files - they have the same bytecode format. However, they are incompatible at the machine-code level: the extra debug-checking options change the layout of Python objects in memory, so the configurations have different shared library ABIs. A compiled C extension built for one will not work with the other. The key to keeping the different module ABIs separate is that module "foo.so" for the standard optimized build will instead be "foo_d.so" i.e. gaining a "_d" suffix to the filename, and this is what the "import" routine will look for. This convention is from the Debian patch, and ultimately comes from the way the Windows build is set up in the upstream build process. Similarly, the optimized libpython2.6.so.1.0 now has a libpython2.6_d.so.1.0 cousin for the debug build: all of the extension modules are linked against the appropriate libpython, and there's a /usr/include/python2.6-debug directory, parallel with the /usr/include/python2.6 directory. There's a new "sys.pydebug" boolean to distinguish the two configurations, and the distutils module uses this to supply the appropriate header paths ,and linker flags when building C extension modules. Finally, the debug build's python binary is /usr/bin/python2.6-debug, hardlinked as /usr/bin/python-debug (as opposed to /usr/bin/python2.6 and /usr/bin/python) It's easy to spot the debug build: the interactive mode tells you the total reference count of all live Python objects after each command: [david@surprise devel]$ python-debug Python 2.6.5 (r265:79063, May 19 2010, 18:20:14) [GCC 4.4.3 20100422 (Red Hat 4.4.3-18)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> print "hello world" hello world [28748 refs] >>> [28748 refs] [15041 refs] So the debug build shares _most_ of the files with the regular build (.py/.pyc/.pyo files; directories; support data; documentation); the only differences are the ELF files (binaries/shared libraries), and infrastructure relating to configuration (Include files, Makefile, python-config => python-debug-config, etc) that are different. I've tested building the "coverage" module against both runtimes, and it works; it installs shared .py/.pyc files and a pair of tracer.so/tracer_d.so files. I tried a few different ways of packaging the debug configuration: I considered (a) adding it to the python-devel subpackage, or (b) to the python-debuginfo subpackage (Debian adds it to their python-dbg packages, which are kind of the equivalent of our -debuginfo rpms), alternatively (c) building out a "debug" subpackage for each of the subpackages within the python specfile, doubling the number of subpackages The approach I favor (option (d), I guess), is to have a single "python-debug" subpackage, holding everything to do with the debug configuration: equivalent to all of the subpackaes from the regular configuration, and requiring them all (since they leverage the shared .py files, for instance). My reasoning here is that this feature is aimed at advanced Python developers, and if you want some of it you probably want all of it - so just one subpackage for simplicitly - but you don't need it for regular builds or debugging, so it's seems better to keep separate from the -devel and -debuginfo subpackages. This is a scaled-back version of my earlier proposal (in which I proposed entirely parallel stacks, and varying the unicode settings) This is far simpler. In particular, the optimized build should be unaffected: all of the paths and the ELF metadata for the standard build should be unchanged compared to how they were before adding the debug configuration. I would like to build out some of our compiled extension modules so that we can add -debug subpackages, in an analogous way to the core python package, but I think it should purely be a voluntary thing: I don't want to burden people packaging Python modules with additional work. Having said that, if you do find yourself debugging a nasty reference counting issue inside an extension module, you'll need a debug build of every C extension module that your reproducer script uses, so the more the better. For reference, Ubuntu do this for all of the Python code in a typical GNOME desktop [3]. We should figure out sane RPM conventions for packaging these (sorry: yes I want to change the python packaging guidelines again, hopefully less invasive than the Python 3 change though) I'm tracking all of this work here: https://fedoraproject.org/wiki/DaveMalcolm/DebugPythonStacks I hope for it to be a Fedora 14 feature. It's debatable whether it should be a feature: this is an area where we're somewhat behind other distributions, so not so good from a marketing perspective - but a good thing to get fixed. I plan to work next on doing the same for our python3 src.rpm. I need to try to get this upstream in some form as well. Hope this seems sane - thoughts? (thanks for reading this far; I know this email is too long) Dave [1] http://lists.fedoraproject.org/pipermail/python-devel/2010-March/000213.html [2] http://svn.python.org/projects/python/trunk/Misc/SpecialBuilds.txt [3] https://wiki.ubuntu.com/PyDbgBuilds [4] http://patch-tracker.debian.org/patch/series/view/python2.6/2.6.5-2/debug-b… and http://patch-tracker.debian.org/patch/series/view/python2.6/2.6.5-2/pydebug…

2 3

FYI: python-sphinx-1.0 coming to rawhide soon
by Toshio Kuratomi 25 May '10

25 May '10

This is just a heads up -- python-sphinx-1.0b1 has arrived. hircus or I will be updating rawhide to the new version soon -- which may cause your documentation to stop building. There are some incompatible changes in this update, notably to the markup used for c functions and modules. There is a compatibility plugin that you can enable to take care of some of the issues or you can port to the new "domains" syntax. we probably will not be pushing this back to older versions of Fedora but we'll have to see what the response is. If we need to push out a compat package for EPEL5 (python-sphinx1-1.0) then we'll have the possibility of doing the same for older Fedora. -Toshio

1 0

Yet another Python runtime: pynie
by David Malcolm 19 May '10

19 May '10

[CCing parrot-owner(a)fedoraproject.org] Pynie is yet another implementation of Python, this time on top of the Parrot virtual machine. I've created a src.rpm and opened a package review request for Pynie here: https://bugzilla.redhat.com/show_bug.cgi?id=593069 Anyone feel like reviewing it? Hope this is helpful Dave

2 1

TurboGears 2.1 beta in updates-testing
by Luke Macken 14 May '10

14 May '10

Hey all, Last week I updated the entire TurboGears2/Pylons stack in rawhide, F13, F12, and EL-5. Everything should currently be in in updates-testing, so any feedback or karma you can provide would be appreciated. I know there is a TG2 hackfest this Sunday, where we are trying to chip away at the small blocker list for the TG2.1 release. I'm also still trying to figure out a couple of bugs in Moksha & fedoracommunity due to the latest stack, but I'm hoping to have those ironed out before or shortly after F13. I also updated the TurboGears resources on the wiki. http://fedoraproject.org/wiki/TurboGears2 luke

2 1

Some more Python 3: (was Re: rawhide report: 20100507 changes)
by David Malcolm 07 May '10

07 May '10

On Fri, 2010-05-07 at 12:25 +0000, Rawhide Report wrote: Three more python 3 subpackages in today's "rawhide" heading for F14 - I've gone ahead and added these to the wiki here: https://fedoraproject.org/wiki/Features/Python3F13#Python_3_already_in_Fedo… Are we missing anything? [snip] > libsemanage-2.0.45-4.fc14 > ------------------------- > * Tue Apr 27 2010 David Malcolm <dmalcolm(a)redhat.com> - 2.0.45-4 > - add python3 subpackage [snip] > python-beaker-1.5.3-2.fc14 > -------------------------- > * Thu May 06 2010 Luke Macken <lmacken(a)redhat.com> - 1.5.3-2 > - Add a python3 subpackage > > > python-mako-0.3.2-2.fc14 > ------------------------ > * Tue May 04 2010 David Malcolm <dmalcolm(a)redhat.com> - 0.3.2-2 > - add python3 subpackage [snip]

2 3

Trying to package "pypy" for F13 onwards
by David Malcolm 04 May '10

04 May '10

pypy is an alternate implementation of Python [1], implemented in Python itself, together with a set of tools for compiling that implementation down to C code. The resulting implementation can, in theory, be faster than the regular "CPython" implementation we know as /usr/bin/python I've managed to package up pypy for Fedora; I've opened up a review request for it here: https://bugzilla.redhat.com/show_bug.cgi?id=588941 Anyone feel like reviewing it? You can download F-13 packages from the scratch build here: http://koji.fedoraproject.org/koji/taskinfo?taskID=2147291 Upstream pypy is working on implementing the C extension module API, so in theory we could try recompiling our python extension modules against pypy at some future point. I don't yet know if that's going to be feasible in the F14 timescale, or if we're looking at further in the future. Hope this is useful Dave [1] http://pypy.org/

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel May 2010