python-devel December 2016

python-devel@lists.fedoraproject.org

20 participants
15 discussions

Making sudo pip Safe
by Michal Cyprian 16 Dec '16

16 Dec '16

Hello, there is a long-standing problem that `sudo pip install` cannot be safely used in Fedora. Many users don't know about this and break python packages on theirs systems. Packages installed using this command can conflict and overwrite Python rpm packages. This is a major problem and we have seen several systems broken by people using "sudo pip". Unfortunately, telling people to not use it will not work: "sudo pip" appears in documentation of too many projects online. We plan to solve this in Fedora 26. A more precise description of the problem follows. Current behavior of python packages installation tools in Fedora: 1) sudo dnf install python3-foo # root installs foo from rpm to /usr/lib/pythonX.Y/site-packages 2) sudo pip3 install foo # root installs foo from PyPI to /usr/lib/pythonX.Y/site-packages 3) sudo pip3 install -t /usr/local/lib/pythonX.Y/site-packages foo # root install package to selected location 4) pip3 install --user foo # user install foo from PyPI to ~/.local/lib/python3.5/site-packages Using the --user option with pip would be the ideal solution. However, it is reportedly broken in some versions of Ubuntu, so it is hard to convince software authors to recommend it. Packages installed using `sudo pip` (2) under /usr can be overwritten or removed by dnf. `dnf install python3-foo` fails if foo was `sudo pip installed` before etc. This problem has been reported many times. [6] Another issue is that packages installed under usr/local (3) cannot be imported unless PYTHONPATH or sys.path is set explicitly. The default install location of pip/distutils-installed packages depends on the value of the sys.prefix variable [4]. There were several discussions on Bugzilla [1] and the pypa-dev mailing list [2]. Interesting solutions were conceived at the CPython level: - addition of sys.local_prefix [2] - simplification of CPython startup sequence [4] Unfortunately none of them were realized and both solutions require many changes of Python and Python Standard Library. We discussed all the possible solutions with colleagues from the Python maint team. It would be great to fix this upstream, but it won't happen in a reasonable time frame. We realized that System Python [3], which was announced in Fedora 24, can help us reach the goal. We came up with the following solution: - sys.prefix of the python3 executable will be set to /usr/local - sys.prefix of system-python will be set to /usr - all rpm python packages will be installed using system-python (value of rpm macro %{__python3} will be changed to point to system-python) - The path /usr/lib/.../site_packages will be included in sys.path of /usr/bin/python3 Note that no packages will be affected by other work going on around system-python, namely that packages can declare that they are only using a smaller subset of the Python stdlib. That option is still opt-in. Behavior of install methods after mentioned changes: 1) sudo dnf install foo # root installs foo from rpm to /usr/lib/pythonX.Y/site-packages 2) sudo pip3 install foo # root installs foo from PyPI to /usr/local/lib/pythonX.Y/site-packages - pip never touches /usr/lib/python3.5/site-packages - /usr/bin/python3 can import modules from - Python Standard Library - site-packages under /usr/local (priority) - site-packages under /usr - system-python (i.e. what all programs installed via DNF will use) is limited to site-packages under /usr, so DNF-installed software is unaffected by anything installed with pip [1] https://bugzilla.redhat.com/show_bug.cgi?id=662034 [2] https://groups.google.com/forum/#!topic/pypa-dev/r6qsAmJl9t0 [3] https://fedoraproject.org/wiki/Changes/System_Python [4] https://docs.python.org/3/library/sys.html#sys.prefix [5] https://www.python.org/dev/peps/pep-0432/ [6] For example: * https://bugzilla.redhat.com/show_bug.cgi?id=1396158 * https://bugzilla.redhat.com/show_bug.cgi?id=1397575 * https://bugzilla.redhat.com/show_bug.cgi?id=1400377 * More: https://www.google.cz/search?q="sudo+pip"+site%3Abugzilla.redhat.com And there are many anecdotal stories of frustrated users who didn't report Michal Cyprian

11 22

Python EGG guidelines need updating
by Orion Poplawski 10 Dec '16

10 Dec '16

Python folks - The FPC needs your help with cleaning up these guidelines: https://fedoraproject.org/wiki/Packaging:Python_Eggs I have a draft here: https://fedoraproject.org/wiki/PackagingDrafts:Python_Eggs that makes use of some new %py{2,3}_{build,install}_egg macros. However, I think a lot more cleanup needs to be done: - Just remove Why eggs through [edit] Providing Egg Metadata Using Setuptools as that is just the standard course now? - Make __requires__='MODULE[VERSIONINFO]' the recommended way to require a version as setuptools seems to officially support this? - Drop Egg "Features" to avoid - Need updated links, or just drop them? Please comment here and/or in https://fedorahosted.org/fpc/ticket/663 Thank you! -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion(a)nwra.com Boulder, CO 80301 http://www.nwra.com

2 1

Python SIG on-boarding issue
by Dhanesh B. Sabane 08 Dec '16

08 Dec '16

CommOps is currently working on the Python SIG On-Boarding ticket [1] and one of the tasks we have identified is re-writing the Python SIG wiki page to make it more beginner-friendly. I was assigned with this particular task which is now complete [2] [3]. We received some feedback from mhroncok today during the CommOps meeting and following points were discussed: 1. Splitting Python SIG in two FAS groups: 1.1 *python-sig* for newcomers and interested members who hang out and help with Python on Fedora 1.2 *python-packagers* for proven members of the community / packagers who will also receive security-related bugs. Promising members from *python-sig* group can be promoted to *python-packagers* given that they have contributed to the package's git. 2. Python Features section on the wiki page needs some love. 3. Targeting specific information for newcomers to include in their self-introductions that would be helpful for SIG members to know where someone fits in the Python community and what their expertise is. We would like to hear your feedback on the draft and also on the aforementioned points. Everyone is welcome to leave a comment on the ticket[1]. [1] https://pagure.io/fedora-commops/issue/84 [2] https://fedoraproject.org/wiki/User:Dhanesh95/SIGs/Python [3] https://fedoraproject.org/wiki/User:Dhanesh95/SIGs/Python/Join --- Dhanesh B. Sabane

6 11

[RFC] RPM's Python dependency generator
by Igor Gnatenko 04 Dec '16

04 Dec '16

Hi, in short, it reads egg metadata and can generate Provides (which we already do now), Requires (which I want to talk about) and Recommends (which I don't care atm). Let's take simple package -- aiohttp. https://bugzilla.redhat.com/show_bug.cgi?id=1381750 As you can see, since some version multidict requirement was bumped to >= 2.0 and async_timeout requirement was added. Currently we specify all requirements during initial package and usually without versions which is breaking after some time. So, let's try it (I will skip unrelated parts). Before: python(abi) = 3.5 python3-chardet python3-multidict After: python(abi) = 3.5 python3.5dist(async-timeout) python3.5dist(chardet) python3.5dist(multidict) >= 2.0 Without more complicated packages (MNE, nipy, nilearn, moss) it's getting much more harder since I have there 10+ deps. We can't really track all changes in upstream code, so if we will enable dependency generator, it will do this work for us. Note that we can't just enable it in RPM, because it will break a lot of packages due to: * missing requires in egg metadata * extra requires in egg metadata (e.g. windows-modules) I would propose plan: 1. Create macro which will enable/disable depgen (e.g %python_enable_depgen) 2. Start enabling depgen and porting things (somehow reuse portingdb.xyz probably?) and submitting patches upstream 3. In 1-2 releases I hope we can use it for major amount of packages 4. Enable depgen by default in RPM, add disabling depgen for remaining packages Neal, you can share how Mageia did that as well and feel free to comment this ;) Thoughts? -- -Igor Gnatenko

7 10

PortingDB is temporarily moved to a new address
by Tomas Orsava 02 Dec '16

02 Dec '16

Hi! PortingDB—a tool providing an overview of the Python 3 porting progress for Fedora packages—is temporarily unavailable through it's regular address [0] due to some DNS issues. You can access PortingDB here in the meantime: http://portingdb-encukou.rhcloud.com/ We apologize for the inconvenience. Tomas Orsava [0] http://portingdb.xyz/

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel December 2016