Hi folks,
One of the minor irritations with running Python 3 applications in Fedora containers is that those containers still default to the "C" local by default - you have to force "C.UTF-8" or "en_US.UTF-8" in order to tell Python that it should use UTF-8 for communicating with system interfaces.
However, since 3.5, the CPython start-up sequence has treated claims from the OS that it should use ASCII more skeptically, and enabled the "surrogateescape" error handling by default on the standard streams.
Along similar lines, what do folks think of the idea of patching Python 3.6 in Fedora to assume UTF-8 if it's told that it should use ASCII to communicate with the OS? That changes the failure mode from "interface problems happen any time you fail to configure your locale correctly" to "interface problems happen any time you fail to configure your locale correctly *and* your default locale uses an encoding other than UTF-8".
That latter case then further breaks down into the following two situations:
- you use an ASCII-incompatible encoding like Shift-JIS or GB-18030, in which case the old POSIX default of ASCII was already broken, and the surrogateescape workaround in 3.5 probably didn't help much
- you use an ASCII-compatible encoding like koi8-r, which means the surrogateescape workaround in 3.5 probably helped a bit, but libraries like `click` would still have complained and refused to run
So I think doing this would be a nice usability improvement for users of the system Python 3 installation on Fedora.
I also have an upstream motive for suggesting this, though: if we do this in the more constrained environment of "Fedora users" and it doesn't break the world, then that will help build a case for making it the default upstream behaviour in Python 3.7.
Regards, Nick.
On Sat, Dec 10, 2016 at 11:56:44PM +1000, Nick Coghlan wrote:
Along similar lines, what do folks think of the idea of patching Python 3.6 in Fedora to assume UTF-8 if it's told that it should use ASCII to communicate with the OS?
+1
Non-utf8 environments are nowadays a rarity, OTOH misconfigured installations which do support utf8 but are just missing an env var are rather common (e.g. mock).
Zbyszek
On Sat, Dec 10, 2016 at 3:40 PM, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Sat, Dec 10, 2016 at 11:56:44PM +1000, Nick Coghlan wrote:
Along similar lines, what do folks think of the idea of patching Python 3.6 in Fedora to assume UTF-8 if it's told that it should use ASCII to communicate with the OS?
+1
Non-utf8 environments are nowadays a rarity, OTOH misconfigured installations which do support utf8 but are just missing an env var are rather common (e.g. mock).
Why aren't we fixing Fedora Cloud/Atomic and the container images to be C.UTF-8 instead of just plain C, then?
On Sat, Dec 10, 2016 at 05:41:45PM -0500, Neal Gompa wrote:
On Sat, Dec 10, 2016 at 3:40 PM, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Sat, Dec 10, 2016 at 11:56:44PM +1000, Nick Coghlan wrote:
Along similar lines, what do folks think of the idea of patching Python 3.6 in Fedora to assume UTF-8 if it's told that it should use ASCII to communicate with the OS?
+1
Non-utf8 environments are nowadays a rarity, OTOH misconfigured installations which do support utf8 but are just missing an env var are rather common (e.g. mock).
Why aren't we fixing Fedora Cloud/Atomic and the container images to be C.UTF-8 instead of just plain C, then?
It's a game of whack-a-mole. You can always fix that place you just noticed where it's missing, but then a few days later it's in another place. Not saying that we should be initializing the locale properly, but rather than we can do both independently.
Zbyszek
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl schrieb am Sa., 10. Dez. 2016 um 23:50 Uhr:
On Sat, Dec 10, 2016 at 05:41:45PM -0500, Neal Gompa wrote:
On Sat, Dec 10, 2016 at 3:40 PM, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Sat, Dec 10, 2016 at 11:56:44PM +1000, Nick Coghlan wrote:
Along similar lines, what do folks think of the idea of patching Python 3.6 in Fedora to assume UTF-8 if it's told that it should use ASCII to communicate with the OS?
+1
Non-utf8 environments are nowadays a rarity, OTOH misconfigured installations which do support utf8 but are just missing an env var are rather common (e.g. mock).
Why aren't we fixing Fedora Cloud/Atomic and the container images to be C.UTF-8 instead of just plain C, then?
It's a game of whack-a-mole. You can always fix that place you just noticed where it's missing, but then a few days later it's in another place. Not saying that we should be initializing the locale properly, but rather than we can do both independently.
To change the default encoding for python was proposed a while ago [1], but was finally dropped again, as upstream didn't agree to this change. Did anything changed here from upstream python?
Best, Thomas
[1] https://fedoraproject.org/wiki/Features/PythonEncodingUsesSystemLocale
On Sun, Dec 11, 2016 at 11:24:54AM +0000, Thomas Spura wrote:
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl schrieb am Sa., 10. Dez. 2016 um 23:50 Uhr:
On Sat, Dec 10, 2016 at 05:41:45PM -0500, Neal Gompa wrote:
On Sat, Dec 10, 2016 at 3:40 PM, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Sat, Dec 10, 2016 at 11:56:44PM +1000, Nick Coghlan wrote:
Along similar lines, what do folks think of the idea of patching Python 3.6 in Fedora to assume UTF-8 if it's told that it should use ASCII to communicate with the OS?
+1
Non-utf8 environments are nowadays a rarity, OTOH misconfigured installations which do support utf8 but are just missing an env var are rather common (e.g. mock).
Why aren't we fixing Fedora Cloud/Atomic and the container images to be C.UTF-8 instead of just plain C, then?
It's a game of whack-a-mole. You can always fix that place you just noticed where it's missing, but then a few days later it's in another place. Not saying that we should be initializing the locale properly, but rather than we can do both independently.
To change the default encoding for python was proposed a while ago [1], but was finally dropped again, as upstream didn't agree to this change. Did anything changed here from upstream python?
Best, Thomas
[1] https://fedoraproject.org/wiki/Features/PythonEncodingUsesSystemLocale
That was 2010, now it's 2016, going on 2017. That was Python 2, now it's Python 3.6 ;) But seriously, with Python 2 such changes were much more risky: Unicode was bolted on, and it was (is) very easy to cause things to fail unpredictably when Unicode was added to the mix. But with Python 3, Unicode is the basis upon which everything is built, and things more often start to fail when (unexpectedly) input is suddenly not Unicode. I think the considerations are a bit different for Python upstream, which has a lot of Windows users and utf-16, and for Linux, where utf-8 is pervasive.
To make things less hand-wavy: there are concrete examples where this would be known to help: containers as mentioned in the original proposal, and mock (in my experience tests most often require setting an utf-8 environment to work). Do we know of examples where the proposed change would make things worse?
Zbyszek
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl schrieb am Mo., 12. Dez. 2016, 02:59:
On Sun, Dec 11, 2016 at 11:24:54AM +0000, Thomas Spura wrote:
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl schrieb am Sa., 10. Dez. 2016 um 23:50 Uhr:
On Sat, Dec 10, 2016 at 05:41:45PM -0500, Neal Gompa wrote:
On Sat, Dec 10, 2016 at 3:40 PM, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Sat, Dec 10, 2016 at 11:56:44PM +1000, Nick Coghlan wrote:
Along similar lines, what do folks think of the idea of patching Python 3.6 in Fedora to assume UTF-8 if it's told that it should
use
ASCII to communicate with the OS?
+1
Non-utf8 environments are nowadays a rarity, OTOH misconfigured installations which do support utf8 but are just missing an env var are rather common (e.g. mock).
Why aren't we fixing Fedora Cloud/Atomic and the container images to be C.UTF-8 instead of just plain C, then?
It's a game of whack-a-mole. You can always fix that place you just noticed where it's missing, but then a few days later it's in another place. Not saying that we should be initializing the locale properly, but rather than we can do both independently.
To change the default encoding for python was proposed a while ago [1],
but
was finally dropped again, as upstream didn't agree to this change. Did anything changed here from upstream python?
Best, Thomas
[1]
https://fedoraproject.org/wiki/Features/PythonEncodingUsesSystemLocale
That was 2010, now it's 2016, going on 2017. That was Python 2, now it's Python 3.6 ;) But seriously, with Python 2 such changes were much more risky: Unicode was bolted on, and it was (is) very easy to cause things to fail unpredictably when Unicode was added to the mix. But with Python 3, Unicode is the basis upon which everything is built, and things more often start to fail when (unexpectedly) input is suddenly not Unicode. I think the considerations are a bit different for Python upstream, which has a lot of Windows users and utf-16, and for Linux, where utf-8 is pervasive.
Don't get me wrong. I'm +1 for unicode everywhere :)
To make things less hand-wavy: there are concrete examples where this would be known to help: containers as mentioned in the original proposal, and mock (in my experience tests most often require setting an utf-8 environment to work). Do we know of examples where the proposed change would make things worse?
I don't and didn't when this change was dropped a few years ago... It would certainly help to discuss this also upstream and if there is no objection, implement Nick's proposal.
Best, Thomas
On 12 December 2016 at 16:18, Thomas Spura tomspur@fedoraproject.org wrote:
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl schrieb am Mo., 12. Dez. 2016, 02:59:
To make things less hand-wavy: there are concrete examples where this would be known to help: containers as mentioned in the original proposal, and mock (in my experience tests most often require setting an utf-8 environment to work). Do we know of examples where the proposed change would make things worse?
I don't and didn't when this change was dropped a few years ago... It would certainly help to discuss this also upstream and if there is no objection, implement Nick's proposal.
I don't anticipate any major concerns with downstream redistributors adding this behaviour, as the main thing that makes us nervous about globally changing the default upstream is the sheer variety of Linux distros out there, and the fact that folks are inclined to take their Linux integration bugs straight to bugs.python.org rather than first trying the issue tracker for their particular distro.
By contrast, for the Fedora system Python, we only need to worry about doing what's right for *Fedora 26+*, which is a much more constrained environment than "arbitrary versions of arbitrary Linux distributions running arbitrary init systems".
Specifically:
- we know that the C.UTF-8 locale should always be available - we know that system applications don't intentionally use the C locale - we know that some popular Python libraries will explicitly fail if you try to use the C locale with Python 3 - we know that anyone trying to use the C locale with a non-UTF-8 default locale already has other problems - we know that (containers aside) systemd is pretty good about setting the locale appropriately based on /etc/locale.conf
That lets us pursue a simpler version of the idea that can ignore some failure modes (like "C.UTF-8" being unavailable).
As far as where we might add that check, I'd suggest the entry point for the `python3` binary itself, rather than in the shared library: https://hg.python.org/cpython/file/3.6/Programs/python.c#l46
At various points during start-up CPython uses 'setlocale(LC_ALL, "")' to force itself back into the default locale for the process based on its environment, so the locale injection would need to look something like:
char *loc = setlocale(LC_CTYPE, NULL); if (loc != NULL && strcmp(loc, "C") == 0) { setenv("LC_ALL", "C.UTF-8", 1); setenv("LANG", "C.UTF-8", 1); }
(Plus error checking and the warning on stderr that this was being done)
I guess the next step would be for me to try this in a local clone, and see what happens when running "LANG=C ./python -m regrtest" as well as when running click.
Cheers, Nick.
On Mon, Dec 12, 2016 at 1:39 AM, Nick Coghlan ncoghlan@gmail.com wrote:
I don't anticipate any major concerns with downstream redistributors adding this behaviour, as the main thing that makes us nervous about globally changing the default upstream is the sheer variety of Linux distros out there, and the fact that folks are inclined to take their Linux integration bugs straight to bugs.python.org rather than first trying the issue tracker for their particular distro.
My one concern is precisely this variety. For instance, if I get a report that my application is raising a UnicodeError on RHEL7 when run under cron (which uses the C locale) I might then try to replicate the error on Fedora using the same LC_ALL=C locale. With this change I would fail to reproduce the error.
This is a variation on arguments about why individual sites should not change the default encoding via sitecustomize.py. The changes tend to make python applications non-portable. I don't think it is as severe because we're still able to broadly classify things as "Fedora Python" vs "Upstream Python" (instead of "Python running at My Business" vs "Python running on the rest of the world" but it still is problematic.
OTOH, if this is a stepping stone and proving ground for getting it into upstream Python then we just get this change a little early... that's IMHO, a good thing.
Perhaps what's needed is a locale on Fedora that allows people to select an ascii encoding for python which does not coincide with the C locale. This should satisfy the case you mention that *most* of the time the C locale is not a conscious desire to select the ascii encoding but also, as I'm pointing out, the need to select an ascii-only encoding for debugging cross-platform scripts and applications.
[..]
As far as where we might add that check, I'd suggest the entry point for the `python3` binary itself, rather than in the shared library: https://hg.python.org/cpython/file/3.6/Programs/python.c#l46
I think the library is the appropriate place. Otherwise you end up with a python application failing when run under mod_wsgi[*]_ which you can't debug using the command line interpreter.
.. [*]: or in libreoffice or any other application that links to libpython for scripting
-Toshio
On 15 December 2016 at 21:17, Toshio Kuratomi a.badger@gmail.com wrote:
On Mon, Dec 12, 2016 at 1:39 AM, Nick Coghlan ncoghlan@gmail.com wrote:
I don't anticipate any major concerns with downstream redistributors adding this behaviour, as the main thing that makes us nervous about globally changing the default upstream is the sheer variety of Linux distros out there, and the fact that folks are inclined to take their Linux integration bugs straight to bugs.python.org rather than first trying the issue tracker for their particular distro.
My one concern is precisely this variety. For instance, if I get a report that my application is raising a UnicodeError on RHEL7 when run under cron (which uses the C locale) I might then try to replicate the error on Fedora using the same LC_ALL=C locale. With this change I would fail to reproduce the error.
But with the current patch you *would* get a visible warning on stderr saying:
Python detected LC_CTYPE=C. Setting LC_ALL & LANG to C.UTF-8.
This is a variation on arguments about why individual sites should not change the default encoding via sitecustomize.py. The changes tend to make python applications non-portable. I don't think it is as severe because we're still able to broadly classify things as "Fedora Python" vs "Upstream Python" (instead of "Python running at My Business" vs "Python running on the rest of the world" but it still is problematic.
Agreed, and my original idea upstream included an environment variable override to account for that case: http://bugs.python.org/issue28180#msg282964
I just forgot about that bit while writing the initial patch :(
As documented at https://docs.python.org/3/using/cmdline.html#environment-variables the normal convention for Python environment variable toggles is "A non-empty string setting enables it", so the name I'd suggest here is PYTHONALLOWCLOCALE.
The error message would then change to:
Python detected LC_CTYPE=C, forcing LC_ALL & LANG to C.UTF-8 (set PYTHONALLOWCLOCALE to disable this behaviour)
and if the environment variable is already set:
Python detected LC_CTYPE=C, but PYTHONALLOWCLOCALE is set. Some applications may not work correctly.
Does that approach seem more reasonable than unilateral locale coercion with no off switch?
OTOH, if this is a stepping stone and proving ground for getting it into upstream Python then we just get this change a little early... that's IMHO, a good thing.
Yeah, my goal is to standardise this upstream for 3.7, but I expect folks to be more willing to make it the default behaviour on *nix systems if at least some distros are willing to try it out in their releases of 3.6 first.
Perhaps what's needed is a locale on Fedora that allows people to select an ascii encoding for python which does not coincide with the C locale. This should satisfy the case you mention that *most* of the time the C locale is not a conscious desire to select the ascii encoding but also, as I'm pointing out, the need to select an ascii-only encoding for debugging cross-platform scripts and applications.
As in an explicit "LANG=C.ASCII"? While I agree that would work, it's probably more complexity than is needed vs a dedicated off switch for the locale coercion.
On the other hand, if *glibc* were to some day start natively interpreting "no locale set" or an unqualified "C" locale as "C.UTF-8", then I agree a "C.ASCII" locale to explicitly opt in to the old behaviour would make sense.
[..]
As far as where we might add that check, I'd suggest the entry point for the `python3` binary itself, rather than in the shared library: https://hg.python.org/cpython/file/3.6/Programs/python.c#l46
I think the library is the appropriate place. Otherwise you end up with a python application failing when run under mod_wsgi[*]_ which you can't debug using the command line interpreter.
There's one pragmatic problem with that, and one that's a question of appropriate division of responsibilities in terms of understanding the runtime's context of use.
The pragmatic problem is that the main CPython binary calls https://docs.python.org/3/c-api/sys.html#c.Py_DecodeLocale to convert the command line arguments from char* to wchar_t* before it calls Py_Main, which means we have to override the locale *before* we hand over control to the dynamically linked library. Otherwise we end up in exactly the same situation that click complains about: by the time we find out there's a problem with the locale, some work has already been done using the wrong setting.
The architectural problem is that when you embed CPython, it really is one of the embedding application's responsibilities to configure the locale such that the interpreter plays nice with the rest of the application. It's one thing to second guess the shell from directly inside a C-level main() function when we know POSIX makes some really old ASCII-centric assumptions and that developers are prone to writing "LANG=C" rather than "LANG=C.UTF-8" to turn off their locale settings, but something else entirely to second guess a GUI application like Blender (where arbitrary amounts of code may have already run before the CPython runtime gets initialised) or an application platform with its own environment management system like Apache httpd.
Cheers, Nick.
jinx ;-)
On Thu, Dec 15, 2016 at 4:38 AM, Nick Coghlan ncoghlan@gmail.com wrote:
On 15 December 2016 at 21:17, Toshio Kuratomi a.badger@gmail.com wrote:
My one concern is precisely this variety. For instance, if I get a report that my application is raising a UnicodeError on RHEL7 when run under cron (which uses the C locale) I might then try to replicate the error on Fedora using the same LC_ALL=C locale. With this change I would fail to reproduce the error.
But with the current patch you *would* get a visible warning on stderr saying:
Python detected LC_CTYPE=C. Setting LC_ALL & LANG to C.UTF-8.
warning is not enough IMHO, but....
Agreed, and my original idea upstream included an environment variable override to account for that case:
I do think this is sufficient. Debugging already requires setting an env var (LC_*=C) so setting a second one in addition is not a big deal. The internet will have outdated information on debugging for a few years and then people will figure out the new invocation and adapt.
I think the library is the appropriate place. Otherwise you end up with a python application failing when run under mod_wsgi[*]_ which you can't debug using the command line interpreter.
There's one pragmatic problem with that, and one that's a question of appropriate division of responsibilities in terms of understanding the runtime's context of use.
The pragmatic problem is that the main CPython binary calls https://docs.python.org/3/c-api/sys.html#c.Py_DecodeLocale to convert the command line arguments from char* to wchar_t* before it calls Py_Main, which means we have to override the locale *before* we hand over control to the dynamically linked library. Otherwise we end up in exactly the same situation that click complains about: by the time we find out there's a problem with the locale, some work has already been done using the wrong setting.
I thought about this one and decided it isn't really a problem. Just make the check in both places. The CPython binary is choosing/needs to preprocess the arguments before calling Py_Main so it needs to check and set locale to do its job. libpython needs to make the same check before it knows it can do its job successfully. (Actually, this looks like an API choice on libpython's part. The arguments are decoded from raw bytes but they aren't assigned any semantic meaning. If libpython handled the decoding as well, then this wouldn't be a concern).
The architectural problem is that when you embed CPython, it really is one of the embedding application's responsibilities to configure the locale such that the interpreter plays nice with the rest of the application. It's one thing to second guess the shell from directly inside a C-level main() function when we know POSIX makes some really old ASCII-centric assumptions and that developers are prone to writing "LANG=C" rather than "LANG=C.UTF-8" to turn off their locale settings, but something else entirely to second guess a GUI application like Blender (where arbitrary amounts of code may have already run before the CPython runtime gets initialised) or an application platform with its own environment management system like Apache httpd.
yeah, this one is much tougher, although I disagree on the reason it's a problem. I do not think it's necessarily the embedding application's responsibility to make sure the embedded interpreter.can run but the nature of the environment variables being process-global means that the library can't set them without affecting the application as a whole. That's a big no-no..
I'd almost say that internalizing the click behviour could be the correct design here. Have the library check that it has a locale with non-ascii capabilities and fail if it doesn't would be helpful. That would quickly point to differences in behaviours running under a mod_wsgi vs /usr/bin/python, for instance, prompting the user to fix the mod_wsgi deployment in advance. OTOH, users don't run into the problem all the time (it depends on the data being processed and how it is handled) so it seems heavy handed to do it this way (I suppose by the same argument I'd have to say that click is doing it wrong to force users to address ascii-only locales...)
The costs here are very steep in both directions... so I don't see any good ways to address it yet. The best I can offer so far is for the library to check and warn if an ascii-only locale is used. That way someone who encounters a UnicodeError in code deployed under mod_wsgi is shown how to debug this when they run it under /usr/bin/python. So something like this from the library:
libpython detected LC_CTYPE=C.Some encoding errors may occur.. Use.PYTHONALLOWCLOCALE=1 LC_CTYPE=C /usr/bin/python if debugging this under /usr/bin/python.
-Toshio
On 15 December 2016 at 23:19, Toshio Kuratomi a.badger@gmail.com wrote:
The costs here are very steep in both directions... so I don't see any good ways to address it yet. The best I can offer so far is for the library to check and warn if an ascii-only locale is used. That way someone who encounters a UnicodeError in code deployed under mod_wsgi is shown how to debug this when they run it under /usr/bin/python. So something like this from the library:
libpython detected LC_CTYPE=C.Some encoding errors may occur.. Use.PYTHONALLOWCLOCALE=1 LC_CTYPE=C /usr/bin/python if debugging this under /usr/bin/python.
OK, I've posted a summary at https://bugzilla.redhat.com/show_bug.cgi?id=1404918#c5 with some proposed wording for the error messages.
For the embedded runtime warning, I'm suggesting "Some libraries and operating system interfaces may not work correctly." but could be pretty easily talked into changing that to "Some text encoding related errors may occur." (as even if a library bails out the way click does, it will probably mention ASCII in the error message)
For the PYTHONALLOWCLOCALE case I'm suggesting "Some libraries, applications and operating system interfaces may not work correctly." as that's literally the answer to "Why isn't the C locale allowed by default anymore?".
Cheers, Nick.
On Sun, Dec 11, 2016 at 3:24 AM, Thomas Spura tomspur@fedoraproject.org wrote:
To change the default encoding for python was proposed a while ago [1], but was finally dropped again, as upstream didn't agree to this change. Did anything changed here from upstream python?
Best, Thomas
[1] https://fedoraproject.org/wiki/Features/PythonEncodingUsesSystemLocale
The Feature to change the encoding in Python2 was problematic because of its implementation. It lead to some pretty nasty problems such as breaking dictionaries. See my blog post about that: https://anonbadger.wordpress.com/2015/06/16/why-sys-setdefaultencoding-will-...
So the difference here is that the proposal is to change the locale. That won't mess with the low-level handling of data structures that messing with sys.setdefaultencoding would.
One of the things that makes changing the locale easier is that Fedora now supports a C.UTF8 locale. So Fedora now ships with a UTF8 capable fallback. If we'd tried to implement this precise change before the nearest equivalent of a universal utf8-capable locale would have been something like en_US.UTF8 which a significant minority would not have found as universal as we would have liked.
-Toshio
On 11 December 2016 at 08:41, Neal Gompa ngompa13@gmail.com wrote:
On Sat, Dec 10, 2016 at 3:40 PM, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Sat, Dec 10, 2016 at 11:56:44PM +1000, Nick Coghlan wrote:
Along similar lines, what do folks think of the idea of patching Python 3.6 in Fedora to assume UTF-8 if it's told that it should use ASCII to communicate with the OS?
+1
Non-utf8 environments are nowadays a rarity, OTOH misconfigured installations which do support utf8 but are just missing an env var are rather common (e.g. mock).
Why aren't we fixing Fedora Cloud/Atomic and the container images to be C.UTF-8 instead of just plain C, then?
This isn't an either/or - we can do both. The advantage of also making a change in the Python 3.x binary is that it should be able to do the right thing even for cases like:
================ $ LANG=C python3 -c 'import click; click.command()(lambda:None)()' Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/core.py", line 716, in __call__ return self.main(*args, **kwargs) File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/core.py", line 675, in main _verify_python3_env() File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/_unicodefun.py", line 119, in _verify_python3_env 'mitigation steps.' + extra) RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Either run this under Python 2 or consult http://click.pocoo.org/python3/ for mitigation steps.
This system supports the C.UTF-8 locale which is recommended. You might be able to resolve your issue by exporting the following environment variables:
export LC_ALL=C.UTF-8 export LANG=C.UTF-8 ================
There's a reason Armin Ronacher made click apps abort outright in that case - it really isn't recoverable from the application framework level, as CPython will have already initialised various things incorrectly by the time any click components get a chance to run.
However, as the distro provider, we can be both more opinionated than the upstream CPython devs as to the correct behaviour here *and* intervene earlier in the interpreter bootstrapping process than library and framework developers can (i.e. before anything gets decoded incorrectly).
While there have been a few different issues about this upstream over the years, http://bugs.python.org/issue28180 is the most recent one (from a user pointing out that sys.getfilesystemencoding() gives them the unhelpful answer of "ASCII" when running under Docker). I posted my thoughts on what I think we should do for 3.7 at: http://bugs.python.org/issue28180#msg282964
Cheers, Nick.
On 12/10/2016 02:56 PM, Nick Coghlan wrote:
Hi folks,
One of the minor irritations with running Python 3 applications in Fedora containers is that those containers still default to the "C" local by default - you have to force "C.UTF-8" or "en_US.UTF-8" in order to tell Python that it should use UTF-8 for communicating with system interfaces.
However, since 3.5, the CPython start-up sequence has treated claims from the OS that it should use ASCII more skeptically, and enabled the "surrogateescape" error handling by default on the standard streams.
Along similar lines, what do folks think of the idea of patching Python 3.6 in Fedora to assume UTF-8 if it's told that it should use ASCII to communicate with the OS? That changes the failure mode from "interface problems happen any time you fail to configure your locale correctly" to "interface problems happen any time you fail to configure your locale correctly *and* your default locale uses an encoding other than UTF-8".
That latter case then further breaks down into the following two situations:
- you use an ASCII-incompatible encoding like Shift-JIS or GB-18030,
in which case the old POSIX default of ASCII was already broken, and the surrogateescape workaround in 3.5 probably didn't help much
- you use an ASCII-compatible encoding like koi8-r, which means the
surrogateescape workaround in 3.5 probably helped a bit, but libraries like `click` would still have complained and refused to run
So I think doing this would be a nice usability improvement for users of the system Python 3 installation on Fedora.
I also have an upstream motive for suggesting this, though: if we do this in the more constrained environment of "Fedora users" and it doesn't break the world, then that will help build a case for making it the default upstream behaviour in Python 3.7.
Regards, Nick.
+1 Handling this is on my (and thus python-maint's) TODO list, but python 3.6, "sudo pip", and PEP 354 have higher priority for us currently. If anyone wants to go ahead, make a patch and put it on Bugzilla, or draft a Fedora Change page, please go ahead!
On 12 December 2016 at 20:37, Petr Viktorin pviktori@redhat.com wrote:
On 12/10/2016 02:56 PM, Nick Coghlan wrote:
I also have an upstream motive for suggesting this, though: if we do this in the more constrained environment of "Fedora users" and it doesn't break the world, then that will help build a case for making it the default upstream behaviour in Python 3.7.
+1 Handling this is on my (and thus python-maint's) TODO list, but python 3.6, "sudo pip", and PEP 354 have higher priority for us currently. If anyone wants to go ahead, make a patch and put it on Bugzilla, or draft a Fedora Change page, please go ahead!
I opened an issue and attached a patch at https://bugzilla.redhat.com/show_bug.cgi?id=1404918
I also cross-linked it with the most recent upstream issue at http://bugs.python.org/issue28180
Cheers, Nick.
On Wed, Dec 14, 2016 at 10:40 PM, Nick Coghlan ncoghlan@gmail.com wrote:
I opened an issue and attached a patch at https://bugzilla.redhat.com/show_bug.cgi?id=1404918
I also cross-linked it with the most recent upstream issue at http://bugs.python.org/issue28180
Comments on the patch:
* In: http://bugs.python.org/issue28180#msg282964 you mentioned adding a "PYTHONKEEPASCIILOCALE environment variable to turn that behaviour off". but this patch doesn't implement thta. I think that will be necessary to address the cross-platform debuggability concern I had eearlier in the thread.
* I'm not 100% certain that LC_CTYPE is the best thing to check. People will set LC_CTYPE in conjunction with LC_COLLATE to C to get a predictable sort order.(CTYPE is needed because bytes can be interpreted as different characters and those differences can affect sort order.). Changing this will mean that their command line sort order (ls, sort, etc) could differ from python's sort order. I haven't thought this fully through or a better way to check for our actual meaning, though, and perhaps python already uses LC_CTYPE in ways that would differ from other unix tools?
* Thinking about whether this belongs in the library or the interpreter some more I'm seeing some hefty cons in both directions. already noted that the con for doing it in the interpreter is that we get out of sync with other things linking to libpython, therefore making debugging harder. The con for doing it in the library is that the environment is global to the process. So mucking with it in the library would affect the entire application, not just the portion that's interpreting python. Not sure what to do here....
-Toshio
On 15 December 2016 at 22:39, Toshio Kuratomi a.badger@gmail.com wrote:
- I'm not 100% certain that LC_CTYPE is the best thing to check.
People will set LC_CTYPE in conjunction with LC_COLLATE to C to get a predictable sort order.(CTYPE is needed because bytes can be interpreted as different characters and those differences can affect sort order.). Changing this will mean that their command line sort order (ls, sort, etc) could differ from python's sort order. I haven't thought this fully through or a better way to check for our actual meaning, though, and perhaps python already uses LC_CTYPE in ways that would differ from other unix tools?
Yep, LC_CTYPE is the source of all the pain, as it's what controls the encoding used for everything that CPython needs to decode before it gets its own codec machinery bootstrapped. Victor Stinner made a couple of attempts at overriding it later in the interpreter bootstrapping process (e.g. based on environment variables) after the codec system was fully up and running, and the problem you end up with is that:
1. Within CPython it's easy to lose track of how you decoded system provided text like sys.argv, sys.warnoptions, sys._xoptions, sys.executable, os.env, etc, so "fixing" incorrectly decoded values later is fragile 2. Even if you *do* get all the details right within CPython, you may be in trouble again as soon as you call out to a third party C/C++ library, especially GUI toolkits like Tcl/Tk, Qt or Gtk that have a lot of locale dependent behaviour
His conclusion was that letting the locale-as-seen-by-CPython diverge from that seen by the rest of the process simply doesn't work, and I don't have any reason to second guess that conclusion.
However, I'll also note that any tooling written in Rust or Go already makes the "UTF-8 everywhere" assumption at the level of the language design, so the proposed change would just move tools written in Python 3 into the same category as those written in those languages (unless you set PYTHONALLOWCLOCALE to request the old behaviour).
- Thinking about whether this belongs in the library or the
interpreter some more I'm seeing some hefty cons in both directions. already noted that the con for doing it in the interpreter is that we get out of sync with other things linking to libpython, therefore making debugging harder.
Note that CPython already offers a range of "preconfiguration" APIs that allow applications embedding the runtime to override otherwise environment based configuration settings. In particular https://docs.python.org/3/c-api/init.html#c.Py_SetStandardStreamEncoding was added specifically so Blender could just tell the Python 3 runtime "configure the standard streams like *this*", rather than having to persuade CPython to guess the right answer.
So the fact embedded runtimes can give you different results from what you get at the command prompt isn't a *new* problem.
[Copying-and-pasting some of your comments from the other subthread to consolidate the two discussions]
I'd almost say that internalizing the click behviour could be the correct design here. Have the library check that it has a locale with non-ascii capabilities and fail if it doesn't would be helpful. That would quickly point to differences in behaviours running under a mod_wsgi vs /usr/bin/python, for instance, prompting the user to fix the mod_wsgi deployment in advance.
While I don't like the idea of locale *coercion* inside the library, I'd be fine with emitting a proper Python level warning inside Py_Initialize after we get the warnings machinery up and running
OTOH, users don't run into the problem all the time (it depends on the data being processed and how it is handled) so it seems heavy handed to do it this way
I think erroring out would be unduly harsh, but a warning seems reasonable given the availability of C.UTF-8.
(I suppose by the same argument I'd have to say that click is doing it wrong to force users to address ascii-only locales...)
click is younger than Python 3, so Armin did make some initial attempts to get it working in the C locale on both 2 & 3. However, he eventually gave the latter up as unsupportable and the error makes it clear that "I don't need to support ASCII based locales on Python 3" is a key constraint in deciding whether or not to adopt click.
Cheers, Nick.
Bumping this thread.
Draft of the self contained change: https://fedoraproject.org/wiki/User:Cstratak/python3.6_c.utf-8_locale
Some of the sections are reworded from PEP 538 [0] as most (if not all) of the motivation section for upstream, also applies for Fedora.
[0] https://www.python.org/dev/peps/pep-0538/
Feedback will be greatly appreciated
Regards,
On 15 February 2017 at 02:42, Charalampos Stratakis cstratak@redhat.com wrote:
Bumping this thread.
Draft of the self contained change: https://fedoraproject.org/wiki/User:Cstratak/python3.6_c.utf-8_locale
Some of the sections are reworded from PEP 538 [0] as most (if not all) of the motivation section for upstream, also applies for Fedora.
Sorry for the lack of response! I basically stopped reading email during PyCon Pune and the trip back to Australia :)
Aside from a typo in the variable name (which I fixed), this looks great to me, and I see from https://fedoraproject.org/wiki/Category:ChangeAnnounced that it's already been announced on fedora-devel.
Cheers, Nick.
python-devel@lists.fedoraproject.org
-
Charalampos Stratakis -
Neal Gompa -
Nick Coghlan -
Petr Viktorin -
Thomas Spura -
Toshio Kuratomi -
Zbigniew Jędrzejewski-Szmek