python-devel March 2020

python-devel@lists.fedoraproject.org

5 participants
5 discussions

Self-introduction
by Fabian Affolter 06 May '20

06 May '20

Hi all, I don't want to repeat myself, please take a look at my wiki page [1]. Why I want to join? I have one too many Python package and I started to add the python-sig as admin. But it goes both ways, if the SIG has access to my packages I want to be part of the SIG. My name is in the member list [2] for quite a while but I guess that doesn't come with the right permissions. Thanks for adding me to python-sig. If you have any further questions please replay to this message or write me in private. These days I'm no longer very active on IRC because of my day job. Kind regard, Fabian [1] https://fedoraproject.org/wiki/User:Fab [2] https://fedoraproject.org/wiki/SIGs/Python/Members_list

3 3

Current plan: Build python3, python3-libs etc. from python39 SRPM on F33+
by Miro Hrončok 06 Apr '20

06 Apr '20

Hello Pythonistas. (I've CC'ed the devel list for further exposure. But let's discuss this on python-devel list please to avoid noise.) We would like ro rename the "python3" component (SRPM) to "python39" to make maintaining various Python versions in various Fedora versions easier. The names of binary RPMs would be unchanged; you still do `dnf install python3`. The change should only affect packagers of Python itself; not users or packagers of software written in Python. The only user-visible or general-packager-visible change is a different component (SRPM) name. Details follow. Currently, we build the Python interpreters this way: - the "main" Python 3 interpreter component is called python3 - it builds a couple of subpackages, like python3-libs or python3-devel - the "alternate" Python 3 versions components are called python3X (python37, python39...) For example, Fedora 31 has: - python36 - python3 (the "main" Python, which is 3.7.x) - python38 - python39 (python37 is retired in f31, but present in Fedora versions where python3 != 3.7) This makes updating the "main" Python fairly complicated. To upgrade from 3.7 to 3.8 in f32, we had to: 1) retire python38 2) upgrade python3 from 3.7 to 3.8 (this makes python3 obsolete python38) 3) re-introduce (unretire) python37 Such an update comes every few years: - https://fedoraproject.org/wiki/Changes/Python3.7 (f29) - https://fedoraproject.org/wiki/Changes/Python3.8 (f32) - https://fedoraproject.org/wiki/Changes/Python3.9 (f33) What's even more complicated is backporting patches across Fedora releases. Assume there is an important upgrade/fix for Python 3.7 we wish to apply for all Fedoras including stable. That currently means we need to apply the fix in: - master branch of python37 - f32 branch of python37 - f31 branch of python3 (!) - f30 branch of python3 While cherry-picking or merging patches between branches of one git repo is quite straight-forward (the only major obstacle is the release and %chaneglog), doing this across multiple git repos (and spec filenames) is very tedious and error-prone (albeit possible). With the 3.9 update approaching in Fedora 33, we would end up with the following scheme when updating Python 3.8: - master branch of python38 - f32 branch of python3 (!) - f31 branch of python38 (!) - f30 branch of python38 Apart from technological obstacles, this causes nontrivial cognitive overhead. So far, I've been carrying that overhead myself, but others (who don't work on the Fedora Python packages daily) always need my assistance with this, because it's overly complicated and hard to understand without drawing pictures. For this to be easier for us, while keeping the change minimal for others, we have decided to retire the python3 component and only continue maintaining the interpreters in "python3X" components. The names of "binary" packages remain unchanged. Before: SRPM: python3 builds: - python3 - python-unversioned-command - python3-{libs,devel,tkinter,idle,test,debug} - python3-debug{info,source} After: SRPM: python39 builds: - python3 (unchanged) - python-unversioned-command (unchanged) - python3-{libs,devel,tkinter,idle,test,debug} (unchanged) - python39-debug{info,source} (usually not installed by name) To avoid further confusion, the "python3-libs" etc. packages will provide "python39-libs" as well. We plan to do this together with the Python 3.9 upgrade. I'll update the change page after some feedback happens here. The initial implementation is in: https://src.fedoraproject.org/rpms/python39/pull-request/32 Here are other approaches we have considered: Why not rename the packages to python39-libs etc. (and only provide python3-libs)? This would be inconsistent with the way other Python 3 packages are called (e.g. python39-setuptools). We consider that confusing. The goal here is to make our work easier while introducing minimal changes to our users and contributors. Why not rename all the Python packages to python39-setuptools etc.? This would create a mess upon distro boundary Python upgrades. All python39-foo packages would need to obsolete all python38-foo packages. Even if we add the obsoletes metadata to all the packages, it would prevent users from having the two parallel stacks of Python installed on the same OS. (We don't have such stacks in Fedora, but we intend to continue to make this possible in third party repos, downstreams of Fedora or other systems built on top or Fedora). See also the "minimal changes" point above. Why not package the alternate stacks as modular streams and keep everything called just python3- (or even python-)? 1) The Python stacks are naturally parallel-installable. This would not be possible if they were different streams of one module. 2) Vital system tools (dnf, anaconda, packaging tools) use Python, so the "main" Python can't be in a module. (We could solve this by a separate "platform Python" just for these system tools, but that makes everything more complex and harder to work with -- the opposite of what this change is about.) See also the "minimal changes" point above. Why are you breaking the world once again? Sorry if this breaks anything for you. Could you be as specific as possible? We anticipate no changes in Fedora packages (other than python3/python39). -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

4 8

RFC: Python minimization in Fedora
by Miro Hrončok 28 Mar '20

28 Mar '20

Hello Fedora! In Python Maint, we sat down and we came up with several ideas how to minimize the filesystem footprint of Python. Unfortunately, the result is horribly long, sorry about that. Please, share your feedback, additional solutions, comments etc. Version with formatting and pictures is available at: https://github.com/hroncok/python-minimization/blob/master/document.md Enclosing here for better in-line responses: # Python minimization in Fedora > While Fedora is well suited for traditional physical/virtual workstations and servers, it is often overlooked for use cases beyond traditional installs. > > Some modern types of deployments — such as IoT and containers — are quite sensitive to size. For IoT that's usually slow data connections (for updates/management) and for cloud and containers it’s the massive scale. -- the preamble of the [Fedora Minimization Objective](https://docs.fedoraproject.org/en-US/minimization/) One of the biggest things in Fedora is Python. Because [Fedora loves Python](https://fedoralovespython.org/) and because the package manager for Fedora packages -- dnf -- happens to be written in Python, the Python interpreter and its standard library comes pre-installed on many (if not all) Fedora systems and is often not possible to remove it without destroying the system completely or making it unmanageable. Python comes with [Batteries Included](https://en.wikipedia.org/wiki/Batteries_Included) -- the standard library is quite big. While pleasant for the programmers, this comes with a large filesystem footprint not entirely desired in Fedora. In this document, we will analyze the footprint and offer several minimization solutions/ideas with their challenges, pros (MiB saved) and cons. It is a list of ideas; **we're not promising to do any of this**. **Goal:** 1. Significantly lower the filesystem footprint of the mandatory Python installation in Fedora. **Non-goals:** 1. We don't aim to lower the filesystem footprint of all Python installations in Fedora -- the default may remain big, if there is an opt-out mechanism. 2. We don't aim to lower the filesystem footprint of all Fedora Python RPM packages, just the `python3` package and its subpackages -- the interpreter and the standard library. However, if any non-goal becomes a side effect of the solution of our goal, good. **Constraints:** 1. Do not break Python users' expectations. As an example, we don't strip Python standard library to the bare minimum and still call it Python. 2. Do not break Fedora users' expectations. As an example, we don't break the ability to hot patch Python files on a live system by default. 3. Do not break Fedora packagers' expectations. As an example, we don't [require "system tools" to use a custom Python entrypoint](https://fedoraproject.org/wiki/Changes/System_Python), such as `/usr/libexec/platform-python` or `/usr/libexec/system-python`. 4. Do not significantly increase the filesystem footprint of the default Python installation. As an example, we don't package [two separate versions (and stacks) of Python](https://fedoraproject.org/wiki/Changes/Platform_Python_Stack) -- one minimal for dnf (or Ansible) and another "normal" for the users. 5. Do not diverge from upstream significantly (but we can drive upstream change). As an example, we don't reinvent the import machinery of Python downstream only, but we might do it in upstream and even [use Fedora to pioneer the change](https://fedoraproject.org/wiki/Changes/python3_c.utf-8_locale). The listed constraints are not absolute. We will mention in each solution, whether we feel that some constraints are violated, but that doesn't mean we shall outright discard the solution. ## How large is Python, actually tl;dr Python 3.8.1 in Fedora has 111 MiB (approximately 77 3.5" floppy disks), but we only **install 37.5 MiB by default** (26 floppy disks). ![77 3.5" floppy disks](https://github.com/hroncok/python-minimization/raw/master/77-floppy-… *77 3.5" floppy disks, courtesy of Dana Walker. Imagine one of them is faulty.* (All numbers are real installed disk sizes based on the `python38` package installed on Fedora 31, x86_64. The split into subpackages is based on the `python3` package from Fedora 32. Slight differences between Fedora 31 and 32 or between various architectures are irrelevant here, we aim for a long term minimization. See the [source of the numbers][source].) In Fedora we split the Python interpreter into various RPM subpackages, some of them are optional. This is what you get all the time: - `python3` contains `/usr/bin/python3` and friends; has 21 KiB. - `python3-libs` contains `/usr/lib64/libpython3.8.so.1.0` and the majority of the standard library, is required by `python3`; has 37.5 MiB. And this is what you get optionally: - `python3-devel` contains the "development files" and makes it possible to compile extension modules, or build RPM packages with Python modules; has 4.5 MiB. - `python3-tkinter` contains the `tkinter` module and several others depending on it (e.g. `turtle`), it is *Recommended* (not *Required*) by `python3` when the *tk* framework is installed, to avoid an unnecessary dependency on *tk* and *X*; has 2 MiB. - `python3-idle` contains the [Python's Integrated Development and Learning Environment](https://docs.python.org/3/library/idle.html), an application, depends on `tkinter` and is not recommended nor required by anything; has 4.2 MiB. - `python3-test` has the `test` module (the selftest suite of Python) and tests contained in other modules (e.g. `lib2to3.tests`), most users don't need this package, it is the biggest part of Python; has 62.8 MiB. - `python-unversioned-command` contains the `/usr/bin/python` symbolic link; has close to 0 Bytes. For the sake of this document, we will mostly focus on the `python3-libs` package, as it contains the wast majority of the bytes we want to get rid of from minimal Fedora installations. We will mostly focus on the standard library, not `/usr/lib64/libpython3.8.so.1.0` -- that file has copious 3.7 MiB, but it contains the Python interpreter itself and minimizing that is out of scope here -- we have bigger fish to fry. ## 2-dimensional classification of the standard library files When we look closely on the files in the standard library, we can classify them by 2 important dimensions: Python modules and file types. ### Python modules The Python 3.8 standard library has 276 different top-level modules, the biggest two being `test` and `idlelib`, both already not part of `python3-libs`. If we factor out modules and submodules removed from `python3-libs`, the ten larges remaining modules are: 1. `encodings`: 2.5 MiB 1. `pydoc_data`: 1.8 MiB 1. `distutils`: 1.8 MiB 1. `asyncio`: 1.4 MiB 1. `email`: 1.1 MiB 1. `unicodedata`: 1.0 MiB 1. `xml`: 1010 KiB 1. `lib2to3`: 993 KiB 1. `multiprocessing`: 925 KiB 1. `unittest`: 750 KiB Some modules here are interesting because they contain mostly data (`encodings`, `pydoc_data`, `unicodedata`), or because they are obviously developer oriented and very rarely used on runtime (`distutils`, `lib2to3`, `unittest`). A special case is the `ensurepip` module -- it has only 34.4 KiB, but it *Requires* unbundled `python-pip-wheel` (1.18 MiB) and `python-setuptools-wheel` (348 KiB) - that puts it between (3) and (4) in the above statistics with 1.56 MiB in total. ### File types (and bytecode caches) The orthogonal dimension is the file type. Python standard library contains directories with both "extension modules" (written in C (usually) and compiled to `*.cpython-38-x86_64-linux-gnu.so` shared object file) and "pure Python" modules (written in Python and saved as `*.py` source file). Each pure Python module comes in 4 files: - `module.py` -- the source - `__pycache__/module.cpython-38.pyc` -- regular (not optimized) bytecode cache - `__pycache__/module.cpython-38.opt-1.pyc` -- optimized bytecode cache (level 1) - `__pycache__/module.cpython-38.opt-2.pyc` -- optimized bytecode cache (level 2) Each of these files has a different purpose (explained below) and each of the files is wasting precious storage space. In total, the different file types in `/usr/lib64/python3.8/` take (without 3rd party packages): - `.py`: 26.4 MiB - `.pyc`: 22.0 MiB - `.opt-1.pyc`: 22.0 MiB - `.opt-2.pyc`: 19.8 MiB - `.so`: 5.3 MiB Files from `python3-libs` in `/usr/lib64/python3.8/` take: - `.py`: 9.8 MiB - `.pyc`: 6.7 MiB - `.opt-1.pyc`: 6.7 MiB - `.opt-2.pyc`: 5.2 MiB - `.so`: 4.9 MiB We see that the various filetypes of pure Python modules occupy significant amount of space when combined. But what are they for? #### .py source files Python is an interpreted language. As such, when you `import` a pure Python module, it is primarily loaded from the `.py` source. The source is parsed and loaded to Python bytecode, which is stored in memory and executed. To speed things up, the bytecode is cached to special files described below. When the cached bytecode already exists (and considered valid), the module is loaded from there, bypassing the source code. We currently package the source files and the bytecode cache files as well, but the source files are still needed. They are used in the following ways: - module discovery -- the bytecode cache files in `__pycache__` are not importable without the source files; - tracebacks -- when Python raises an uncaught exception, it is presented in a form of a *traceback* containing the original source code, loaded from the source files on demand; - custom administrator changes and hotfixes -- when editing the source files directly on disk, the bytecode cache is invalidated (at least by default) and will not be used until re-cached; - cache invalidation checks -- each time the bytecode is loaded from the cache, the source file is checked for mtime, so it has to exist (there are however [other optional cache invalidation modes](https://docs.python.org/3/reference/import.html#pyc-invalidation) -- checking checksum of the source file or not checking anything); - `__file__` -- some modules read the path of their own sources from the magic `__file__` variable and some logic around that might fail if the path is different (such as if the modules is loaded directly from a bytecode cache file). #### .pyc regular (not optimized) bytecode cache When a pure Python module gets imported for the first time after it has been modified (or first time ever), the bytecode cache is is created in `__pycache__/<modulename>.cpython-38.pyc` to be later used on subsequent imports. Why are the bytecode cache files created during the buildtime of the RPM `python3` package and shipped with the corresponding `.py` file? This is what would happen if the files were not shipped: 1. If a non-root user executes Python code, Python won't succeed saving the file, the bytecode cache will not be written and hence there will be no future benefits from having the cache in the first place -- startup will be slower. On each import, Python will attempt the write which might have further minor negative impact on performance. 2. If a root user with restricted SELinux context executes Python code, then write operation will fail and the audit log will be pumped with AVC violations. The result is (1) + lots of noise. 3. If a root user with unrestricted SELinux context runs Python code, Python is able to regenerate and store the `.pyc` files. They will then stay on disk after the package is removed (possibly updated to the next 3.X version) unless proper RPM level trickery is done (such as listing it as `%ghost`). #### .opt-?.pyc (optimized) bytecode caches Similarly to the previous point, the optimized bytecode cache files -- `__pycache__/<modulename>.cpython-38.opt-1.pyc` (or `...opt-2.pyc`) -- are created when Python is invoked with the `-O` (or `-OO`) flag. When run with the optimization flag, [`-O`](https://docs.python.org/3/using/cmdline.html#cmdoption-o): > Remove assert statements and any code conditional on the value of `__debug__`. When run with [`-OO`](https://docs.python.org/3/using/cmdline.html#cmdoption-oo): > Do `-O` and also discard docstrings. To clarify: This *is* the optimization. There is nothing more. In most common cases, you don't gain any significant performance boost, yet we must assume that there are Fedora users out there invoking Python in this way -- either because their code actually gains performance or because they were tempted by the word "optimization". The bytecode has asserts, `__debug__` conditionalized code and docstrings (with level 2) optimized away and hence is different and needs a different cache. If the cache files don't exist and the users invoke Python with `-O`/`-OO` (or other means, such as the [`PYTHONOPTIMIZE`](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONOPTIMIZE) environment variable), everything bad from the previous section would happen. ### Biggest modules in python3-libs, breakdown by file type module | .py | .pyc | .opt-1.pyc | .opt-2.pyc | other | total ----------------|-----------|-----------|------------|------------|-------------|--------- encodings | 1.4 MiB | 378.4 KiB | 377.9 KiB | 362.4 KiB | 24.0 KiB | 2.5 MiB pydoc_data | 656.1 KiB | 408.3 KiB | 408.3 KiB | 408.3 KiB | 8.0 KiB | 1.8 MiB distutils | 647.1 KiB | 421.3 KiB | 420.5 KiB | 321.1 KiB | 16.9 KiB | 1.8 MiB ensurepip | 7.6 KiB | 6.5 KiB | 6.5 KiB | 5.9 KiB | 8.0 KiB | 34.4 KiB<br>+ 1.52 MiB wheels asyncio | 441.2 KiB | 365.8 KiB | 363.6 KiB | 291.2 KiB | 8.0 KiB | 1.4 MiB email | 364.8 KiB | 283.1 KiB | 282.8 KiB | 188.7 KiB | 16.0 KiB | 1.1 MiB unicodedata | | | | | 1.0 MiB .so | 1.0 MiB xml | 288.5 KiB | 242.8 KiB | 241.8 KiB | 196.4 KiB | 40.0 KiB | 1009.5 KiB lib2to3 | 281.1 KiB | 237.3 KiB | 234.2 KiB | 185.9 KiB | 32.0 KiB | 993.4 KiB multiprocessing | 262.9 KiB | 222.7 KiB | 220.4 KiB | 203.1 KiB | 16.0 KiB | 925.1 KiB See the remaining lines in the [data source][source]. ## Possible solutions Now when we know what is on those 77 floppy disks, we can decide which ones need to go. ![77 3.5" floppy disks](https://github.com/hroncok/python-minimization/raw/master/77-floppy-… *77 3.5" floppy disks, courtesy of Harold Miller. Shall we ditch the beige ones or the blue?* ### Solution 0: Do nothing, keep the status quo The status quo installs the mandatory 37.5 MiB (out of 111 MiB) by default. This is achieved by splitting various test modules, IDLE and tkinter to separate optional subpackages. How does that stand? This solution technically discards 51 floppy disks, gets rid of 73.5 MiB, saves 66% of space. That is pretty good. However, it is the status quo and we will use it as base to compare other proposed solutions, hence for the sake of our measurements, this **saves 0 MiB / 0%**. All further percentage savings will be based on the current mandatory 37.5 MiB. The status quo however already **violates constraint (1)**: it breaks Python users' expectations. As a Python user, I expect the entire of the standard library to be installed, which is not the case. While Python comes pre-installed and ready to be used by developers and users alike, programs using the `tkinter` module will simply fail with a confusing `ModuleNotFoundError`. This has been the case forever and the situation is similar (or worse) with other Linux distributors of Python, such as Debian or openSUSE. Always installing `tkinter` would contradict the goal here, so we won't change that. ### Solution 1: Slim down the Python standard library One solution is to stop having such a big standard library. Python has existed for some time now and a lot of the standard library modules might no longer be relevant to the general audience. Our colleague Christian Heimes has proposed [PEP 594](https://www.python.org/dev/peps/pep-0594/) -- *Removing dead batteries from the standard library* for Python upstream. So far, it has not been approved and the discussion [turned out to be a heated one](http://pyfound.blogspot.com/2019/05/amber-brown-batteries-included-but…. It proposes to remove 30 modules from the standard library for various reasons, mostly because they have better replacements or because they are no longer as useful as they once were. If approved, this would **save 1.4 MiB / 3.7%** or a bit less (two removed classes are parts of bigger files and the calculations were simplified to assume the entire file is no longer there -- the difference is not significant). Not to violate the (5) constraint, this however **has to happen in upstream**, that means not sooner than in Python 3.10 (cca Fedora 35). This is not a kind of change that would benefit from pioneering in Fedora. We are not aware of a static analyzer that would recognize dependencies on standard library modules and there is no existing metadata for this. Just removing the modules in Fedora (or moving them to an optional subpackage) would only cause breakage and break Python users' (1) and Fedora packagers' expectations (3). ### Solution 2: Move developer oriented modules to python3-devel (or split the stdlib into pieces) Quite a handful of modules are clearly targeted at developers who code in Python and not at the users of the applications written in it. Here they are, largest first: 1. `pydoc_data`: Contains data for the `pydoc` module described below. 2. `distutils`: Used when distributing and installing Python packages trough `setup.py` files. Predecessor of `setuptools`. 3. `ensurepip`: Used to install `pip`, mostly to virtual environments via the `venv` module. 4. `lib2to3`: Used by the `2to3` tool to convert legacy code to Python 3. Also used on install time trough `setup.py` files. 5. `unittest`: A testing framework for unit tests. 6. `pydoc`: Generates developer documentation from docstrings. 7. `doctest`: Tests if documentation reflects the reality. 8. `venv`: Creates Python virtual environments. Moving all those modules to `python3-devel` (or `python3-libs-devel` etc.) could **save 6.1 MiB / 16%** and additional **1.5 MiB of wheels** (not calculated in the total amount we count percentages from). This would however **violate the (1) and (3) criterion**. Python users expect working `venv` and `unittest`. Fedora packagers would need to manually (remember: no metadata, no static analyzer) track runtime dependencies on such modules -- they actually happen, for example there are [modules depending on lib2to3](https://pypi.org/project/modernize/). Alternatively such thing would no longer be allowed to name itself Python. It would merely be a "minimal Python" with a separate entrypoint - and that **violates the (3) or (4) criterion** (depending on the actual implementation). Alternatively, this change would need to be driven upstream -- track dependencies on standard library modules and allow it to be shipped in parts. See also our draft [PEP 534](https://www.python.org/dev/peps/pep-0534/) -- *Improved Errors for Missing Standard Library Modules*. If implemented, this would allow us to split the library to several parts (either minimal + rest, or per module, or anything in between) and only make the actually used modules mandatory, saving an unknown amount of space (arguably quite large) and several external dependencies as well (such as `libsqlite3.so`, `libgdbm.so` etc.). We could basically do the `python3-tkinter` split at scale, via an upstream supported way. ### Solution 3: Compress large data-like modules Some pure Python modules, like `encodings` or `pydoc_data` contain mostly data. We could compress the data in the modules. For example `pydoc_data` is basically a dictionary with very long strings. Those strings are repeated in source as well as various bytecode cache files. We could store them as compressed bytestrings instead. Alternatively, we could leverage the Python's ability to import from a zip file and zip such modules. That prevents "hot patching" them on live system (constraint (2)), but if absolutely needed, they can be unzipped and edited. The need to live patch `encodings` or `pydoc_data` should not be very common. Not all modules can be zipped, extra caution would be needed. For example, `pydoc` currently reads a CSS file like this: ```python path_here = os.path.dirname(os.path.realpath(__file__)) css_path = os.path.join(path_here, url) with open(css_path) as fp: return ''.join(fp.readlines()) ``` Similar code would need to be ported to [importlib.resources](https://docs.python.org/3/library/importlib.html#modul… -- changes like this are very likely accepted by upstream, but still needed to be carefully found first. Either way, when carefully only zipping `encodings` and `pydoc_data`, we could **save 3.4 MiB / 9 %**. When compressing the strings inline, we anticipate similar or worse result. ### Solution 4: ZIP the entire standard library Stretching previous solution a bit further, we might want to zip the entire standard library (at least the pure Python parts). However, we are not sure whether this was anticipated by upstream and whether this does not in fact **violate constraint (5)**. Extra care would be needed. This would require a great deal of testing and thorough analysis of half a million of lines of code. Not only this will most likely break things, it will probably also **violate constraints (1) and (2)** (Python and Fedora users' expectations). It can also increase the startup time. To mitigate that, we might want to ship 2 RPM packages with the standard library -- one uncompressed and one zipped: - The `python3-libs` package would *Require* any of them (via virtual provides or boolean requires: `Requires: (python3-libs-modules or python3-libs-modules-zip)`). - The `python3-libs` package would **Recommend** the uncompressed package. - To avoid increasing the total filesystem footprint when both packages are installed, the packages might conflict with each other -- however that might be a bad user experience. Nevertheless, this might (in theory) **save 17.8 MiB / 47 %**. ### Solution 5: Stop shipping mandatory bytecode cache This solution sounds simple: We do no longer ship the bytecode cache mandatorily. Technically, we move the `.pyc` files to a subpackage of `python3-libs` (or three different subpackages, that is not important here). And we only *Recommend* them from `python3-libs` -- by default, the users get them, but for space critical Fedora flavors (such as container images) the maintainers can opt-out and so can the powerusers. This would **save 18.6 MiB / 50%** -- quite a lot. However, as said earlier, if the bytecode cache files are not there, Python attempts to create them upon first import. That can result in several problems, here we will try to propose how to workaround them. #### Problem 5.1: Slower starts without bytecode cache When a non-root user runs Python code, the bytecode cache is never created. This can result in potentially slower start of Python apps. However, that might be OK: The wast majority of Fedora users will get the *Recommended* bytecode cache and the rest will have a small slowdown. This does not violate users' expectations **if documented properly** - most users get the old behavior (the default remains fast, but big). Optionally, we might patch Python to warn in that case and suggest installing the appropriate subpackage. That would of course be a downstream only patch and would **violate constraint (5)**. Alternatively, the warning might suggest running a specific command as root to populate the cache -- that might (or might not) be acceptable upstream. Arguably it is not a very nice user experience, and also it only helps with limited bandwith, not limited storage space. #### Problem 5.2: Leftover bytecode cache files When a root user with unrestricted SELinux context runs Python code, the bytecode cache is created. As such, it would need to be marked as `%ghost` in the RPM package with the Python source, while it would exist as real file in the RPM package with the bytecode cache. Example pseudo-specfile snippet: ```spec %files libs # this package Recommends the 3 packages below .../module.py %dir .../__pycache__/ %ghost .../__pycache__/module.cpython-38.pyc %ghost .../__pycache__/module.cpython-38.opt-1.pyc %ghost .../__pycache__/module.cpython-38.opt-2.pyc %files libs-bytecode-cache # this package Requires the libs subpackage .../__pycache__/module.cpython-38.pyc %files libs-bytecode-cache-opt-1 # this package Requires the libs subpackage .../__pycache__/module.cpython-38.opt-1.pyc %files libs-bytecode-cache-opt-2 # this package Requires the libs subpackage .../__pycache__/module.cpython-38.opt-2.pyc ``` Our experiments show that if two packages co-own a file and one of them is marked as `%ghost`, everything works as expected: - manually created `.pyc` file is overridden by the packaged one without a conflict/error/warning/problem - manually created `.pyc` file is removed on package removal Hence, we anticipate this point as potentially non-problematic, however real testing with the `python3` package has not yet been done. #### Problem 5.3: SELinux denials When a root user with restricted SELinux context runs Python code, the bytecode cache is not created and the audit log is pumped with AVC violations. The result is the same as in 5.1 plus noise. As a workaround, we might work with the SELinux experts to allow the Python process to write the bytecode cache even in restricted context. This could be a **potential security problem** -- any malicious code written in Python would be able to store malicious bytecode in the cache -- all other invocations of Python would execute that bytecode instead of the proper one. As such, we *think* this **violates constraint (2)** -- Fedora users expect that SELinux keeps them safe. However, we don't really know what level of protection is expected here: This might require further discussions. As a solution to this problem, we might stop Python from attempting to write the bytecode cache in the first place. That would still preserve problem 5.1 (that is fine), but would also solve 5.2. However, we cannot just patch Python to stop writing bytecode cache, as that would violate constraints (1) and (5). We might however pioneer an upstream change, that skips writing bytecode cache if a certain marker is present in the `__pycache__` directory: ```spec %files libs .../module.py %dir .../__pycache__/ .../__pycache__/cpython-38.nowrite %files libs-bytecode-cache .../__pycache__/module.cpython-38.pyc ... other opt levels in this or other subpackages ... ``` (The name of the marker is just an example.) If present, all present bytecode cache would be read but there would be no attempts to write it. As a result, users would gain the cache benefit when they install the bytecode cache package(s) (recommended by the `python3-libs` package), but Python would not attempt to create the files. This is a reasonable compromise: - Default remains big and fast (cached). - Minimal is small and a bit slower. - No SELinux problems. *Note:* If we are to eventually adapt this solution (in either form) in all Python RPM packages to gain even more space, this would certainly need more RPM-level abstraction with macros and dark magic (like the debuginfo packages) -- we cannot anticipate all Fedora Python package maintainers to manually do this. However for now, we would only do it in `python3-libs` as written in the goal of this document. ### Solution 6: Stop shipping mandatory optimized bytecode cache This is essentially the same as previous solution except we would keep the non-optimized bytecode cache mandatory. That gives us several more options to workaround the caveats. This would **save 11.9 MiB / 32%**. #### Workaround 6.1: Fallback to less optimized bytecode cache We can patch Python to fallback to less optimized bytecode cache if the properly optimized bytecode cache does not exist or cannot be created. 1. opt-2 would fallback to opt-1 or non-optimized (in this order) 1. opt-1 would fallback to non-optimized 1. non-optimized would always be present This workaround would require a change of the current caching logic. Either there will be no attempt to write the new bytecache files if the less optimized bytecode cache exists, or Python would check if it can write the bytecode cache and only fallback to less optimized ones if it cannot write to the destination. This workaround however **violates Python users' expectations (1)**: It executes less optimized bytecode than the user has elected to. At the same time, this **violates (5)** if done downstream-only. Both can be **solved by doing this with upstream coordination** -- designing a PEP that describes this behavior into great detail, implement the behavior in Fedora and bring it upstream once ready. Impact on performance would need to be evaluated as well. #### Optimization level 2 is already broken It is important to note that optimization level 2 bytecode cache in Fedora is already partially "broken". In the times of Python 2 and 3.4 or less, both non-zero optimization levels shared the same bytecode cache paths. Hence the Fedora packages only shipped optimization level 1 `.pyo` files (`o` for optimized). Python 3.5 has altered the paths to make optimization 1 and 2 cache coexistable and the `python3` package was adapted to ship all 3 levels of optimization (0, 1 and 2), but all the other packages still only ship two (0 and 1) -- [`brp-python-bytecompile` and `%py_byte_compile`](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#manual-bytecompilation) both only compile for the two levels. That means all the problems with missing bytecode cache files are actually already happening with all Fedora's Python 3 RPM packages (except `python3-libs` and other `python3` subpackages themselves) when Python is executed with `-OO` or when `PYTHONOPTIMIZE` is set to 2+. This has been the case **since Fedora 24** and **nobody has ever reported it as a problem** -- hence we might just drop the optimization level 2 bytecode cache and consider the problems an unsupported corner case. That would **save 5.2 MiB / 14%**. Technically this is wrong, but pragmatically it works just fine. Alternatively, we might make a case upstream and deprecate and eventually remove `-00` because we don't use it -- however we are not sure if that is a good enough reason. With the marker file proposed in the previous solution, we can outright drop the optimization level 2 bytecode cache for good (or move it to a package that is not even *Recommended*, only *Suggested*). ### Solution 7: Stop shipping mandatory source files, ship .pyc instead Since the `.py` source files are not the ones that are imported by default, we might as well ship only the bytecode files mandatorily. To allow module discovery, we would need to rename and move the `.pyc` files from `__pycache__/module.cpython-38.pyc` to `../module.pyc`. When such file is located in `sys.path`, this is what happens: - When only `module.py` exists (status quo), everything works as described in the first sections of this document. - When both `module.py` and `module.pyc` exist, the `.pyc` is ignored and everything works as if it was not there (including the bytecode cache files in `__pycache__/*.pyc`). - When only `module.pyc` exists, the module is imported from that bytecode cache file regardless of the optimization level (bytecode cache files in `__pycache__/*.pyc` are ignored). When doing it this way (shipping only nonoptimized `.pyc`, not shipping source or additional bytecode caches (optimized), we would **save 21.7 MiB / 57.9 %**. Several things would **violate Python/Fedora users' expectations (1)(2)**: - Tracebacks would not contain lines of sources. - The source files would be gone -- not only users cannot edit them but they can no longer even read them. To mitigate that, we could have 2 RPM packages with the standard library (similarly to *Solution 4: ZIP the entire standard library*): 1. One with moved `.pyc` files only. 2. One with source `.py` files and `__pycache__` (possibly only recommended if combined with other solutions). In order to save ourselves from 2 conflicting subpackages, we might do it this way: 1. The moved `.pyc` files package is mandatory. 2. The other RPM package is recommended. This however **violates constraint (4)** -- default users would get two files with non-optimized bytecode cache. Unfortunately the files are in different directories, and hence we cannot hardlink them on the RPM level -- RPM only allows hardlinking files in the same directory to avoid cross filesystem hardlinks. If we symlink the files, Python currently does not follow them. If we get upstream support for following symbolic links, we might do something like this: ```spec %files libs # Recommends libs-source .../module.pyc %files libs-source # Requires libs .../module.py %dir .../__pycache__/ .../__pycache__/module.cpython-38.pyc # symbolic link to ../module.pyc .../__pycache__/module.cpython-38.opt-1.pyc .../__pycache__/module.cpython-38.opt-2.pyc ``` With the two optimized caches optionally `%ghost`ed if combined with other solutions. If we don't get upstream support for following symbolic links, we might ship the duplicate bytecode cache files and change them to a hardlink in RPM scriptlet / trigger (if they are on the same filesystem, which is very likely), however that only helps with limited storage space (and it requires the storage during installation), not limited bandwith. Alternatively, we might change the way the source and bytecode caches are prioritized on import time, with upstream coordination, to allow having the non-optimized `.pyc` file in just one location without losing the benefits of having the source files. Such as having an (optionally compressed) source file in a `__pysource__` directory and loading it when showing tracebacks. We could also explore this solution with only some modules (e.g. big data modules, described in *Solution 3: Compress large data-like modules*: `encodings`, `pydoc_data`). For such limited scope, we could simply only ship the one `.pyc` file (one optimization level without sources). ### Solution 8: Compress .pyc files We might propose an upstream change (pioneered in Fedora) to add an option to [compress the `.pyc` files](https://bugs.python.org/issue22789). We would add a "compressed" flag to the `.pyc` header, and we would change `importlib` to unzip the payload before unmarshalling (deserializing) the bytecode. This would potentially save **10.2 MiB / 27.2%**, but it might have negative impact on performance. The number is based on actually zipping each individual `.pyc` file, not on only compressing the content. ### Solution 9: Deduplicate bytecode cache Given the nature of the bytecode caches, the non-optimized, optimized level 1 and optimized level 2 `.pyc` files may or may not be identical. Consider the following Python module: ```python 1 ``` All three bytecode cache files would by identical. While with: ```python assert 1 ``` Only the two optimized cache files would be identical with each other. And this: ```python """Dummy module docstring""" 1 ``` Would produce two identical bytecode cache files but the opt-2 file would differ. Only modules like this would produce 3 different files: ```python """Dummy module docstring""" assert 1 ``` When we examine all the bytecode cache files currently shipped with `python3-libs` and compare them between the optimization levels, we get: - 607 modules have bytecode files - 454 identical optimization 0 and 1 pairs - 68 identical optimization 1 and 2 pairs - 62 identical optimization 0, 1 and 2 triads (already counted in both of the above) Since all of the bytecode caches are kept within the same folder, we can in fact hardlink them between each other and **save 4.0 MiB / 10.7 %**. Even if this would be [done automagically by the filesystem](https://btrfs.wiki.kernel.org/index.php/Deduplication), by doing it explicitly we also save the bandwidth -- the RPM packages are smaller. It is also important to realize that most of the standard library modules have docstrings (except empty `__init__.py` files), but only every fourth has `__debug__` conditionals or asserts. If we also go with a solution that removes the second optimization level bytecode cache and combine it with this one, we can deduplicate optimization level 1 bytecode cache for three quarters of the modules. When the bytecode cache is updated for some reason, e.g. because the source file was updated by an administrator, the cache file is recreated, effectively breaking the hardlink. As more files get updated this way, the size naturally increases, but this does not break users' expectations. As a nice benefit, we can automatically do this with all Fedora Python RPM packages without any cons (except for an insignificant slowdown when comparing the files during build) saving potentially large amounts of space. That's a lot of saved money in the cloud world. As a single data point for that general slim down: On my workstation I have 360 MiB of various Python 3.7 bytecode files in `/usr` and I can save 108 MiB. ### Solution 10: Stop shipping mandatory Python, rewrite dnf to Rust The main reason we need to ship Python everywhere is the package manager -- dnf. If we rewrite dnf to some non-Python, possibly compiled language such as Rust (or C if we are more traditional), we don't need to ship Python at all. This might sound crazy, but see for example [microdnf](https://github.com/rpm-software-management/microdnf) -- a minimal dnf for (mostly) Docker containers that uses libdnf and hence doesn't require Python. This solution **saves 37.5 MiB / 100%** of mandatory Python. It possibly also saves more space by reducing the amount of installed Python packages, but increases the size of dnf itself. We can most likely assume a compiled executable would have a lesser footprint than a handful of Python modules used by dnf -- this doesn't violate constraint (4): the combined footprint of (micro)dnf + Python won't be significantly larger than now. However, most importantly, this solution **violates constraint (2)**: Fedora users expect Python to be available, always. Missing Python could break stuff like Ansible based deployments. ## Conclusion You can see that some of the solutions offer significant slim-down with very little struggle, while other solutions may turn out to be to breaking. At the same time, various solutions can be combined. It is important to note that the solutions can contradict each other and the storage savings cannot be generally summed when combining them. As an example, we cannot deduplicate different optimization level bytecode cache files and ship them from different optional subpackages at the same time. For now, we plan to [start with bytecode cache deduplication](https://github.com/fedora-python/compileall2/issues/16), and we will let the Fedora community discuss our proposals. After all, there might be holes in them and the list is certainly not complete. ## Copyright This document is placed in the public domain or under the [CC0 1.0 Universal license](https://creativecommons.org/publicdomain/zero/1.0/), whichever is more permissive. The photos are [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). [source]: https://github.com/hroncok/python-minimization/blob/master/python-minimizat… -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

4 4

Help needed to get dependencies in EPEL 8 for pagure
by Neal Gompa 26 Mar '20

26 Mar '20

Hey all, I've been trying to get Pagure into EPEL 8 for a couple of months now so that we can upgrade our Pagure instances to RHEL 8[1]. Thankfully, most of Pagure's dependencies *are* now present in EPEL 8, so there's only a few that need to be added. The list of Pagure dependencies missing are below: * gitolite3: limb * python-jenkins: cottsay * python-binaryornot: pingou * python-celery: bowlofeggs, mrunge, pingou, ngompa * python-flask-wtf: pingou * python-wtforms: kumarpraveen, pjp, sundaram * python-pygit2: pwalter Celery is in a special position here, because it has a number of missing dependencies, too (which is why I haven't branched and built it yet): * python-kombu: mrunge, pingou, fab, pjp, sundaram * python-billiard: mrunge, pingou, fab, pjp, sundaram * python-amqp: eharney * python-vine: mrunge * python-case: mrunge A note here: RHEL 8 ships libgit2 0.26.8, so we need pygit2 0.26.x. This is already what we ship in the EPEL 7 branch, so that can just be branched into a new epel8 branch and built. Fortunately, I only need Python 3 variants of all these things, as Pagure is Python 3 in Fedora and I intend to keep it that way for EPEL 8. This is also a prerequisite for dropping Python 2 support in Pagure for 6.0, as we need all Pagure production servers to be able to move to Python 3 first (and our servers run RHEL instead of Fedora). Thank you all in advance! [1]: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapr… -- 真実はいつも一つ！/ Always, there's only one truth!

2 4

Re: Pagure for EL8 (EPEL8)
by Neal Gompa 25 Mar '20

25 Mar '20

On Mon, Nov 18, 2019 at 11:18 AM Kevin Fenzi <kevin(a)scrye.com> wrote: > > On Sat, Nov 16, 2019 at 05:37:16PM -0500, Neal Gompa wrote: > > I've done an early build locally to determine what's needed to make > > this possible. The following report from DNF indicates the missing > > packages that need to be added to EPEL 8 before I can introduce Pagure > > into EPEL8: > > > > Problem 1: conflicting requests > > - nothing provides python3-jenkins needed by pagure-ci-5.8-1.el8.noarch > > Problem 2: conflicting requests > > - nothing provides gitolite3 needed by pagure-5.8-1.el8.noarch > > - nothing provides python3.6dist(binaryornot) needed by > > pagure-5.8-1.el8.noarch > > - nothing provides python3.6dist(celery) needed by pagure-5.8-1.el8.noarch > > - nothing provides python3.6dist(flask-wtf) needed by pagure-5.8-1.el8.noarch > > - nothing provides python3.6dist(redis) needed by pagure-5.8-1.el8.noarch > > - nothing provides python3.6dist(straight.plugin) needed by > > pagure-5.8-1.el8.noarch > > - nothing provides python3.6dist(wtforms) needed by pagure-5.8-1.el8.noarch > > - nothing provides python3.6dist(pygit2) >= 0.26.0 needed by > > pagure-5.8-1.el8.noarch > > Problem 3: conflicting requests > > - nothing provides python3-trololio needed by pagure-ev-5.8-1.el8.noarch > > > > One of the reasons I'd like to have this done sooner rather than later > > is so that we can drop Python 2 support from Pagure with version 6.0. > > I think it's quite reasonable to say that version 6.0 isn't going to > > happen until we can get our Pagure servers running on EL8 using Python > > 3. > > > > So now, I need some help making this happen. I already own trololio, > > and I'm going to make that available in EPEL 8 ASAP. Can anyone help > > with some of the other dependencies here? > > Can you give a list with maintainers? I'm not sure off hand how many of > those are maintained by me/infra-sig, but any I can I would be happy to > add in. There's a few that are in testing I think already... > Sure, here's a list so far (package: maintainer): * gitolite3: limb * python-jenkins: cottsay * python-binaryornot: pingou * python-celery: bowlofeggs * python-flask-wtf: pingou * python-redis: kevin * python-straight-plugin: pingou * python-wtforms: kumarpraveen * python-pygit2: pwalter * python-trololio: ngompa I've already got trololio going: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-6c443851dd -- 真実はいつも一つ！/ Always, there's only one truth!

1 3

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel March 2020