On Thu, Jun 01, 2017 at 11:26:55AM -0700, Toshio Kuratomi wrote:

I think it is better if the build system sets LANG=C.UTF-8, not the python specific macros.

Yeah, setting it through python macros would mean that e.g. any python program invoked during the build would see the obsolete non-utf8 encoding. Seems better to just set LANG=C.UTF-8 globally.

Zbyszek

On Thu, Jun 1, 2017 at 8:56 AM, Miro Hrončok mhroncok@redhat.com wrote:

...

Hi Pythonistas.

Regarding our Python 3 C.UTF-8 locale coercing [1], aka PEP 538 [2].

As you probably know, we build RPM packages with the C locale. So everytime we use python3 in the spec file, the coercing message is shown. This can be more problematic than just spamming the build logs, see for example the related rpmlint bug [3][4].

Our macros, such as %{python3_sitelib}, %{python3_version} etc. all call python3 and generate the warning. Should we "fix" our macros to set the LANG to C.UTF-8?

If we change the %{__python3} macro entirely, we might get rid of all of those warnings and we will workaround the fact that we build RPM packages with the C locale. On the other hand the packager would not be able to set a desired locale because it will always be overwritten:

# The crazy test suite needs Czech locale
LANG=cs_CZ.utf8 %{__python3} -m pytest

Will become:

LANG=cs_CZ.utf8 LANG=C.utf8 /usr/bin/python3 -m pytest

So I would not do that.

But we can change all other macros in /usr/lib/rpm/macros.d/macros.python3 to set the UTF-8 locale. Would that be wise? Desired?

Any thoughts?

[1] https://fedoraproject.org/wiki/Changes/python3_c.utf-8_locale [2] https://www.python.org/dev/peps/pep-0538/ [3] https://bugzilla.redhat.com/show_bug.cgi?id=1457786 [4] https://bugzilla.redhat.com/show_bug.cgi?id=1436345

-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-leave@lists.fedoraproject.org

---

python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-leave@lists.fedoraproject.org

On Thu, Jun 01, 2017 at 11:02:25PM +0200, Miro Hrončok wrote:

On 1.6.2017 20:49, Zbigniew Jędrzejewski-Szmek wrote:

...

On Thu, Jun 01, 2017 at 11:26:55AM -0700, Toshio Kuratomi wrote:

...

I think it is better if the build system sets LANG=C.UTF-8, not the python specific macros.

Yeah, setting it through python macros would mean that e.g. any python program invoked during the build would see the obsolete non-utf8 encoding. Seems better to just set LANG=C.UTF-8 globally.

Who could make that happen? Should I propose this at devel@?

It seems that mock uses $LANG to configure $LANG and all $LC_* variables in the build environment. But when mock is invoked with LANG=C, it ignores that and uses C.UTF-8 instead, because of the "Python 3 C.UTF-8 locale" change in python. So with new-enough python, we'd get C.UTF-8 anyway on the builders, once the builders are upgraded to F26. But then you'd still get a warning on every mock invocation.

It seems that right solution, that would work while the builders are still not F26, would be to configure their locale to C.UTF-8. It'd just be a matter of dropping LANG=C.UTF-8 into /etc/locale.conf. I think a ticket on https://pagure.io/fedora-infrastructure would be the way to request the change.

Zbyszek

On 2 June 2017 at 18:46, Nick Coghlan ncoghlan@redhat.com wrote:

On Fri, Jun 2, 2017 at 1:15 PM, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:

...

It seems that right solution, that would work while the builders are still not F26, would be to configure their locale to C.UTF-8. It'd just be a matter of dropping LANG=C.UTF-8 into /etc/locale.conf. I think a ticket on https://pagure.io/fedora-infrastructure would be the way to request the change.

While this seems like the right next step to me, if it isn't feasible for some reason, we may need to reconsider emitting the warning at the Python level, and instead just silently change LC_CTYPE to the more sensible setting.

A follow-up on this: I just made the change upstream to turn these warnings off by default, and add a "PYTHONCOERCECLOCALE=warn" runtime flag to let folks opt in to the warnings as a tool for debugging potential locale coercion and locale compatibility problems. We were simply getting far too many false positives where code that was previously working fine (because it was genuinely only needing to handle 7-bit ASCII inputs and outputs) was failing solely due to the new warnings, not because anything was actually broken.

There are also some significant improvements to the test suite to make sure that it's properly covering the expected behaviour when there *aren't* any coercion target locales available. While that's not applicable to Fedora directly, it *will* apply to any future Python 3.6 SCL, as well as to any python36 packages put together for EPEL.

Tracker issue: https://bugs.python.org/issue30565 Squashed commit: https://github.com/python/cpython/commit/eb81795d7d3a8c898fa89a376d63fc3bbfb...

Cheers, Nick.