python-devel January 2017

python-devel@lists.fedoraproject.org

7 participants
8 discussions

Python 3.6, Fedora, and the "C" locale
by Nick Coghlan 24 Feb '17

24 Feb '17

Hi folks, One of the minor irritations with running Python 3 applications in Fedora containers is that those containers still default to the "C" local by default - you have to force "C.UTF-8" or "en_US.UTF-8" in order to tell Python that it should use UTF-8 for communicating with system interfaces. However, since 3.5, the CPython start-up sequence has treated claims from the OS that it should use ASCII more skeptically, and enabled the "surrogateescape" error handling by default on the standard streams. Along similar lines, what do folks think of the idea of patching Python 3.6 in Fedora to assume UTF-8 if it's told that it should use ASCII to communicate with the OS? That changes the failure mode from "interface problems happen any time you fail to configure your locale correctly" to "interface problems happen any time you fail to configure your locale correctly *and* your default locale uses an encoding other than UTF-8". That latter case then further breaks down into the following two situations: - you use an ASCII-incompatible encoding like Shift-JIS or GB-18030, in which case the old POSIX default of ASCII was already broken, and the surrogateescape workaround in 3.5 probably didn't help much - you use an ASCII-compatible encoding like koi8-r, which means the surrogateescape workaround in 3.5 probably helped a bit, but libraries like `click` would still have complained and refused to run So I think doing this would be a nice usability improvement for users of the system Python 3 installation on Fedora. I also have an upstream motive for suggesting this, though: if we do this in the more constrained environment of "Fedora users" and it doesn't break the world, then that will help build a case for making it the default upstream behaviour in Python 3.7. Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

7 19

Re: Packages FTBFS with Python 3.6
by Zbigniew Jędrzejewski-Szmek 18 Jan '17

18 Jan '17

On Tue, Jan 17, 2017 at 08:29:16AM -0700, Jerry James wrote: > On Tue, Jan 17, 2017 at 7:08 AM, Zbigniew Jędrzejewski-Szmek > <zbyszek(a)in.waw.pl> wrote: > > On Mon, Jan 16, 2017 at 10:27:19PM -0800, Adam Williamson wrote: > >> On Tue, 2017-01-17 at 05:33 +0000, Zbigniew Jędrzejewski-Szmek wrote: > >> > Oh, Jerry beat me to it by taking a different route and disabling most > >> > test. That should work too, but I think it hides real problems with > >> > the python3 code... > >> > >> Yeah. Please don't just disable tests. Tests exist for a reason. > > > > We need to disable *some* tests, at least if we hope to build this > > quickly. It's just a question whether to disable more or less of > > them. > > > > Most tests fail because of sympy/pyglet/xvfb/opengl interaction > > (#904851), not from sympy error per se — the same tests run fine on a > > "real" display. > > Oh, sorry if I stepped on somebody's toes. i didn't realize others > were trying to fix sympy, too. I guess I should join the python SIG > mailing list, but I'm already on far too many mailing lists... No, no, there were no toes to step on. I didn't announce anywhere that I'm working on this. And anyway, I think our work is complementary: I didn't know about the patch you pulled in, but otoh, I managed to fix some of the tests by passing options to xvfb. I'll try to push some of my changes too, let's see if that builds. > But I don't understand what you mean about me disabling tests. I > haven't disabled any tests, much less "most tests". i did add a BR on > pyglets, then discovered that that CAUSES test failures due to missing > GLX support, and so I removed the pyglets BR again. Maybe a BR on > mesa is needed as well as pyglets? I don't know. > > Anyway, I checked in an experimental patch in an attempt at fixing the > test failures, and now there is a different test failure. :-( So > either my experimental patch isn't good enough, or it works perfectly > and has unmasked another problem. I probably won't have time to > figure that out for a couple of days, so if somebody else is > interested in looking at it, here is the latest build failure: > > https://koji.fedoraproject.org/koji/buildinfo?buildID=828867 Zbyszek

1 1

Re: Packages FTBFS with Python 3.6
by Zbigniew Jędrzejewski-Szmek 17 Jan '17

17 Jan '17

On Mon, Jan 16, 2017 at 10:27:19PM -0800, Adam Williamson wrote: > On Tue, 2017-01-17 at 05:33 +0000, Zbigniew Jędrzejewski-Szmek wrote: > > On Tue, Jan 17, 2017 at 05:29:35AM +0000, Zbigniew Jędrzejewski-Szmek wrote: > > > On Wed, Jan 11, 2017 at 04:28:18AM +0000, Zbigniew Jędrzejewski-Szmek wrote: > > > > > python3-sympy-0:1.0-3.fc25.noarch > > > > > > > > Pfff, it still fails in the test suite :( > > > > > > I'm chipping away at this one. There are some failing python2 tests > > > which I had to disable, but it seems that things are mostly with python2. > > > But it seems that the python3 code is plain broken (bad non-absolute > > > imports). > > > > Oh, Jerry beat me to it by taking a different route and disabling most > > test. That should work too, but I think it hides real problems with > > the python3 code... > > Yeah. Please don't just disable tests. Tests exist for a reason. We need to disable *some* tests, at least if we hope to build this quickly. It's just a question whether to disable more or less of them. Most tests fail because of sympy/pyglet/xvfb/opengl interaction (#904851), not from sympy error per se — the same tests run fine on a "real" display. Zbyszek

1 0

Re: Packages FTBFS with Python 3.6
by Zbigniew Jędrzejewski-Szmek 16 Jan '17

16 Jan '17

On Thu, Jan 05, 2017 at 02:45:37PM -0800, Adam Williamson wrote: > On Wed, 2016-12-21 at 00:18 +0100, Miro Hrončok wrote: > > python-sphinxcontrib-programoutput > > This one was abandoned by its upstream years ago. He actually entirely > deleted the repository from github, though someone else put up a copy > of it recently (they didn't pick up maintenance, just put up a > 'historical copy'). > > It's only used by certbot and python-acme (which is actually a part of > certbot that gets its own releases on pypi and which we treat as a > separate source package). > > Both have stopped using it upstream now, so we can quite trivially > backport those fixes to our certbot and python-acme packages, and > retire sphinxcontrib-programoutput. > > Zbyszek is the owner of the package...do you agree with this plan, > Zbyszek? I'm fixing up certbot and python-acme, but you'd have to > retire programoutput. Hi Adam, I'm just back from the winter break. It's always great to see other people take care of my packages, thanks! You're right that the original repo is gone, but I think that this extension remains to be useful. There's a bunch of forks on github, so hopefully somebody will pick up maintenance. If that doesn't happen, or if a good replacement appears, I'll drop the package. But for now I prefer to let it live. I fixed the test suite, it's building in rawhide. Zbyszek PS. What's the status of the python3.6 rebuild? Can somebody post a list of work-needing packages?

1 4

To use or not to use: packaged flask
by Alec Leamas 13 Jan '17

13 Jan '17

Hi out there! I'm dipping my toes in flask, completely newbie. Doing so, I see a lot of fedora flask packages, but no-one anywhere recommends using these - it's all about pypi. I "think" I prefer the packaged version, partly because I'm using another package with native code (which, as I understand it, isn't that easy to make a linux pypi package of). However, I'm stuck at the very beginning: $ ./run_flask ... ImportError: No module named 'flask.cli' So, my question: is it possible to run a flask application using the packaged components similar to run_flask above, in any way? If so, how? Cheers! --alec

3 4

newbie: Linking python C extensions blues
by Alec Leamas 13 Jan '17

13 Jan '17

Still struggling with my first package. Don't know if this belong to this list (let me know if not) Anyway, I package the extension and make a 'pip install' which builds it. Linker command is: gcc -pthread -shared -Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld build/temp.linux-x86_64-3.5/lirc/_client.o -L/usr/lib64 -llirc_client -lpython3.5m -o build/lib.linux-x86_64-3.5/_client.cpython-35m-x86_64-linux-gnu.so Still, this module has unresolved references. This can be seen using ldd. When invoked on a correctly linked variant there is a line ldd ../lib/.libs/_client.so liblirc_client.so.0 => /home/mk/tmp/lirc/........... However, the variant created by setuptools misses this line, and the corresponding symbols are unresolved. Still, it's linked using -llirc_client in the linker command above. Any clue out there? Why isn't my _client.so linked to liblirc_client as it should? Cheers! --alec

1 1

Packages FTBFS with Python 3.6
by Miro Hrončok 07 Jan '17

07 Jan '17

Hi all, We've recently tried to rebuild all Python packages with Python 3.6. However, we currently have bunch of packages that simply fail to build. As the list contains >200 packages, it would be very helpful, if you (package maintainers) could help us solve the issues, as we cannot go one by one to investigate the issue. I've attached the list of failed packages (failed.txt). You can search Koji to find out, what went wrong. Sometimes, it's just missing dependency. That dependency should be on this list as well. If it is not, maybe we lost it, please tell us if that's the case. Maybe the dependency is circular and something needs to be bootstrapped. If you need provenpackager powers, let me know. Everything currently happens in a side tag. I will notify you when the side tag gets merged. To build your package in mock, I've attached a configuration file for it (fedora-rawhide-x86_64-p36.cfg). Some useful commands: fedpkg build --target=f26-python3 # build in Koji from git fedpkg build --target=f26-python3 --scratch # scratch build in Koji from git koji build f26-python3 --scratch <srpm> # scratch build in Koji from SRPM fedpkg srpm && mock -r fedora-rawhide-x86_64-p36 <srpm> # mock build Thanks for your help. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

3 4

Python 3.6 and blocking-by-default os.urandom()
by Nick Coghlan 04 Jan '17

04 Jan '17

Hi folks, The long discussions upstream regarding the default behaviour of Python's os.urandom() API in Python 3.6 have come to an end, with Guido opting to make os.urandom() block implicitly waiting for the system RNG to be ready, even on Linux (where /dev/urandom doesn't do that). While I can see his point from a cross-platform language runtime design perspective, I'm still not sure it's the right answer for the Fedora system Python, as it doesn't play nice with ABRT, and goes against Ted T'so's decision to keep /dev/urandom non-blocking at the kernel level (risking confusion for folks that are Linux developers & operators first, and Pythonistas second). If "os.urandom() was called before system RNG is ready" throws an uncaught BlockingIOError, then we'll get a nice ABRT-friendly Python stack trace for people to debug (and perhaps figure out their VM entropy pool isn't being seeded properly, or their ARM hardware design needs a better source of entropy). By contrast, if it just blocks, then folks will be faced with a system hang, which they'll need to trace back to CPython being blocked on a kernel getrandom() call, and then infer from that that the system RNG isn't ready (for whatever reason), and they should probably do something about that. Patching this behaviour should be relatively straightforward - Python 3.5 currently makes this call in non-blocking mode and falls back to reading /dev/urandom directly in that case, so we'd just be forward porting the same logic to 3.6 and raising an exception instead of falling back to the file descriptor. Accordingly, what I propose we do is as follows: 1. Raise the concern in the F26 system-wide change proposal for migrating to Python 3.6 2. Apply the patch when the 3.6 beta releases are added to Fedora Rawhide 3. Decide whether or not to keep the patch based on ABRT results and other feedback on the Rawhide builds. Note that if the feedback on Rawhide shows that the change is helping people to find and diagnose VMs and hardware with improperly seeded entropy pools, that's a *good* thing: this proposed change is replacing Python 3.5's "/dev/urandom and os.urandom() may silently return statistically less-than-fully-random random numbers if the kernel entropy pool isn't seeded properly" with "os.urandom() will fail noisily in those cases, so you can either switch to the random module, or fix your entropy pool seeding". Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 9

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel January 2017