Hi folks,
[Note: this may not make much sense to folks that haven't been closely following the Modular Server work for F27. If that's you, sorry - hopefully this will be more comprehensible by the time I officially propose it for F28, as the relevant terminology becomes more widely understood. In the meantime, check out https://docs.pagure.org/modularity/ to learn more]
I'm working on a draft proposal for a "Default Python" module (see https://fedora-python.readthedocs.io/en/latest/plans/default-python-module/ for details), and hit an interesting challenge when it comes to defining the containing images that we want to be able to build from our module definitions.
A quick summary of what I'm expecting we'll have by F28:
- a separate platform-python binary in the Platform module - python2 modules with a full 2.7 stream and a partial interpreter-only 2.6 stream - python3 modules with a full 3.6 stream and a TBD set of other streams - a default-python module to switch between no-default, python2-default and python3-default
With those modules defined, a minimal Fedora container image will only include platform-python, but we'd at least also have Python3-F28, and Python2-F28 images that included the respective user Python stack in addition to the platform Python runtime.
The part I'm struggling with is this: Python 3.7.0 is expected to be released in June 2018, while Fedora 28 is due out in May 2018. It would be *really nice* to be able to define a Fedora 28 based Python 3.7 container where "/usr/bin/python" and "/usr/bin/python3" were both references to "/usr/bin/python3.7".
The challenge with actually doing that lies in the "/usr/bin/python3" symlink: integrated userspace Python applications in F28 are going to expect that to still refer to the Python 3.6 stream.
One way I could potentially see this working:
* the normal non-conflicting setup is as follows:
* the python3 3.6 stream includes a "/usr/bin/python3" symlink * the other python3 3.x streams do *not* include such a symlink & hence don't conflict * the default-python python3-default stream does *not* include such a symlink & hence doesn't conflict
We then add some more streams to the default-python module that *do* include a "/usr/bin/python3" symlink: python3.5-default, python3.7-default, etc
The trick with these streams is that they would *conflict* with the regular python3 3.6 stream, due to the disagreement over what "/usr/bin/python" means. That dependency resolution conflict would then mean that have a non-default version of Python 3.x configured as the default when defining your container would *prevent* you from including any regular userspace Python components - you'd only be able to include the ones built specifically for that stream.
Does that approach sound sufficiently plausible to folks that I can use it to provide feedback to the folks working on the modularity tooling as to the capabilities we think we'll need?
Cheers, Nick.
On 08/18/2017 01:38 PM, Nick Coghlan wrote:
Hi folks,
[Note: this may not make much sense to folks that haven't been closely following the Modular Server work for F27. If that's you, sorry - hopefully this will be more comprehensible by the time I officially propose it for F28, as the relevant terminology becomes more widely understood. In the meantime, check out https://docs.pagure.org/modularity/ to learn more]
I'm working on a draft proposal for a "Default Python" module (see https://fedora-python.readthedocs.io/en/latest/plans/default-python-module/ for details), and hit an interesting challenge when it comes to defining the containing images that we want to be able to build from our module definitions.
A quick summary of what I'm expecting we'll have by F28:
- a separate platform-python binary in the Platform module
- python2 modules with a full 2.7 stream and a partial
interpreter-only 2.6 stream
- python3 modules with a full 3.6 stream and a TBD set of other streams
- a default-python module to switch between no-default,
python2-default and python3-default
With those modules defined, a minimal Fedora container image will only include platform-python, but we'd at least also have Python3-F28, and Python2-F28 images that included the respective user Python stack in addition to the platform Python runtime.
The part I'm struggling with is this: Python 3.7.0 is expected to be released in June 2018, while Fedora 28 is due out in May 2018. It would be *really nice* to be able to define a Fedora 28 based Python 3.7 container where "/usr/bin/python" and "/usr/bin/python3" were both references to "/usr/bin/python3.7".
The challenge with actually doing that lies in the "/usr/bin/python3" symlink: integrated userspace Python applications in F28 are going to expect that to still refer to the Python 3.6 stream.
One way I could potentially see this working:
the normal non-conflicting setup is as follows:
- the python3 3.6 stream includes a "/usr/bin/python3" symlink
- the other python3 3.x streams do *not* include such a symlink &
hence don't conflict
- the default-python python3-default stream does *not* include such
a symlink & hence doesn't conflict
We then add some more streams to the default-python module that *do* include a "/usr/bin/python3" symlink: python3.5-default, python3.7-default, etc
The trick with these streams is that they would *conflict* with the regular python3 3.6 stream, due to the disagreement over what "/usr/bin/python" means. That dependency resolution conflict would then mean that have a non-default version of Python 3.x configured as the default when defining your container would *prevent* you from including any regular userspace Python components - you'd only be able to include the ones built specifically for that stream.
Does that approach sound sufficiently plausible to folks that I can use it to provide feedback to the folks working on the modularity tooling as to the capabilities we think we'll need?
That sounds like it would work. But yes, please talk to the Modularity WG to see if modules can be made to work that way.
On 21 August 2017 at 19:46, Petr Viktorin pviktori@redhat.com wrote:
On 08/18/2017 01:38 PM, Nick Coghlan wrote:
Does that approach sound sufficiently plausible to folks that I can use it to provide feedback to the folks working on the modularity tooling as to the capabilities we think we'll need?
That sounds like it would work. But yes, please talk to the Modularity WG to see if modules can be made to work that way.
Aye, this draft proposal was essentially me figuring out what I think we're going to want/need for Python before I pitched the related feature ideas to the Modularity WG :)
The proposal then became something I could point to to say "This is the problem I'm trying to solve" while we discussed various possible ways of solving it.
I finally had a discussion with Langdon about it today, and he really isn't a fan of my idea of trying to enhance the modularity tooling to natively support parallel installation of streams - he'd strongly prefer that we stick to the idea of "one active stream per module per system" (at least for now), and then rely on separate SRPMs and/or the Software Collections parallel installation layout to handle use cases like tox.
(Note: I'll get the properly formatted proposal updated by tomorrow, so feel free to wait for that if the email version below is a bit hard to read)
My current thinking based on that discussion is that we're actually going to need a module set that looks like this for F28+:
* Platform Python (already planned for F27) - part of the Platform module - stream names match Fedora releases (f26, f27, etc) * Python Application Runtime Modules (already planned for F27) - one module for Python 2 and one for Python 3 - stream names match upstream Python releases (2.7, 3.6, etc) - Application Runtime Modules conflict with the corresponding Integrated Python Module - provide "/usr/bin/python2" and "/usr/bin/python3" respectively - also satisfy "python2-*" and "python3-*" RPM dependencies - only the Python 2 module satisfies "python-*" RPM dependencies (for now) * Integrated Python Modules - one module for Python 2 and one for Python 3 - stream names match Fedora releases (f26, f27, etc) - each depends on an *exact* version of the corresponding Python Application Runtime module * Default Python Module - provides /usr/bin/python (and nothing else) - stream names: no-default, python3-default, python2-default
The scope for F27 would specifically be Platform Python and the Python Application Runtime Modules, with the separate Integrated Python Modules being defined for F28+
At least for now, the Python XY stacks for Fedora and EPEL, as well as the Python SCLs, would be treated as something generated downstream from the app runtime modules, rather than as something that the modularity tooling would necessarily handle natively.
The general idea is that it would then be up to other modules to decide whether they specifically needed the Integrated Python Module with all the related system bindings (in which case they'd either depend on that module, or depend on another RPM that depended on it), or were prepared to cope with any installed version of Python 3 (in which case they'd just use normal RPM level "python3*" dependencies).
Right now, the difference between the Integrated Python Modules and the Python Application Runtime Modules isn't particularly obvious, but it hopefully becomes clearer once we consider questions like "Who will decide when to rebase to Python 3.7?" and "When will a particular stream stop being updated?".
For the Integrated Python Modules, those are Fedora level design decisions, as reflected in the stream names being based on Fedora versions. This means that for a system container, or a traditional mutable installation, you'd be able to continue to rely on a shared Python installation with all the operating system level bindings for things like the RPM database, without Fedora package maintainers having to provide prebuilt bindings for arbitrary Python versions. You'd only change streams when you changed base Fedora versions, and the update cycle for these streams would match the Fedora release cycle (i.e. each stream would receive updates for 13 months from the date of the corresponding Fedora release).
By contrast, for the Python App Runtime Modules, when to rebase is an application developer level decision, as reflected in the stream names being based directly on Python versions. This means that for an application container image, you'd be able to select an arbitrary Python version of your choice, but in doing so, you'd potentially be giving up ready access to pre-built bindings for various system APIs (if your choice of stream didn't match the dependencies declared in the Integrated Python Module). For these streams, the update cycle would match that of the relevant upstream Python branch (i.e. full maintenance updates until shortly after the next Python feature release, then security updates until 5 years after the original X.Y.0 release).
Cheers, Nick.
On 08/24/2017 10:13 AM, Nick Coghlan wrote:
On 21 August 2017 at 19:46, Petr Viktorin pviktori@redhat.com wrote:
On 08/18/2017 01:38 PM, Nick Coghlan wrote:
Does that approach sound sufficiently plausible to folks that I can use it to provide feedback to the folks working on the modularity tooling as to the capabilities we think we'll need?
That sounds like it would work. But yes, please talk to the Modularity WG to see if modules can be made to work that way.
Aye, this draft proposal was essentially me figuring out what I think we're going to want/need for Python before I pitched the related feature ideas to the Modularity WG :)
The proposal then became something I could point to to say "This is the problem I'm trying to solve" while we discussed various possible ways of solving it.
I finally had a discussion with Langdon about it today, and he really isn't a fan of my idea of trying to enhance the modularity tooling to natively support parallel installation of streams - he'd strongly prefer that we stick to the idea of "one active stream per module per system" (at least for now), and then rely on separate SRPMs and/or the Software Collections parallel installation layout to handle use cases like tox.
(Note: I'll get the properly formatted proposal updated by tomorrow, so feel free to wait for that if the email version below is a bit hard to read)
My current thinking based on that discussion is that we're actually going to need a module set that looks like this for F28+:
- Platform Python (already planned for F27)
- part of the Platform module
- stream names match Fedora releases (f26, f27, etc)
Stream names match the Platform module. We follow its policy here, even when it changes.
- Python Application Runtime Modules (already planned for F27)
- one module for Python 2 and one for Python 3
- stream names match upstream Python releases (2.7, 3.6, etc)
- Application Runtime Modules conflict with the corresponding > Integrated Python Module
- provide "/usr/bin/python2" and "/usr/bin/python3" respectively
- also satisfy "python2-*" and "python3-*" RPM dependencies
- only the Python 2 module satisfies "python-*" RPM dependencies (for now) > * Integrated Python Modules
- one module for Python 2 and one for Python 3
These need to be parallel installable, to support tox. So we need separate modules for each Python release.
- stream names match Fedora releases (f26, f27, etc)
- each depends on an *exact* version of the corresponding Python Application Runtime module
Huh? You just said Python Application Runtime Modules would conflict with this.
- Default Python Module
- provides /usr/bin/python (and nothing else)
- stream names: no-default, python3-default, python2-default
+1 I guess this would contain a "python" RPM, containing just a /usr/bin/python symlink, which would added to non-modular Fedora as well.
The scope for F27 would specifically be Platform Python and the Python Application Runtime Modules, with the separate Integrated Python Modules being defined for F28+
At least for now, the Python XY stacks for Fedora and EPEL, as well as the Python SCLs, would be treated as something generated downstream from the app runtime modules, rather than as something that the modularity tooling would necessarily handle natively.
The general idea is that it would then be up to other modules to decide whether they specifically needed the Integrated Python Module with all the related system bindings (in which case they'd either depend on that module, or depend on another RPM that depended on it), or were prepared to cope with any installed version of Python 3 (in which case they'd just use normal RPM level "python3*" dependencies).
Right now, the difference between the Integrated Python Modules and the Python Application Runtime Modules isn't particularly obvious, but it hopefully becomes clearer once we consider questions like "Who will decide when to rebase to Python 3.7?" and "When will a particular stream stop being updated?".
For the Integrated Python Modules, those are Fedora level design decisions, as reflected in the stream names being based on Fedora versions. This means that for a system container, or a traditional mutable installation, you'd be able to continue to rely on a shared Python installation with all the operating system level bindings for things like the RPM database, without Fedora package maintainers having to provide prebuilt bindings for arbitrary Python versions. You'd only change streams when you changed base Fedora versions, and the update cycle for these streams would match the Fedora release cycle (i.e. each stream would receive updates for 13 months from the date of the corresponding Fedora release).
By contrast, for the Python App Runtime Modules, when to rebase is an application developer level decision, as reflected in the stream names being based directly on Python versions. This means that for an application container image, you'd be able to select an arbitrary Python version of your choice, but in doing so, you'd potentially be giving up ready access to pre-built bindings for various system APIs (if your choice of stream didn't match the dependencies declared in the Integrated Python Module). For these streams, the update cycle would match that of the relevant upstream Python branch (i.e. full maintenance updates until shortly after the next Python feature release, then security updates until 5 years after the original X.Y.0 release).
Cheers, Nick.
On 24 August 2017 at 19:02, Petr Viktorin pviktori@redhat.com wrote:
On 08/24/2017 10:13 AM, Nick Coghlan wrote:
My current thinking based on that discussion is that we're actually going to need a module set that looks like this for F28+:
- Platform Python (already planned for F27)
- part of the Platform module
- stream names match Fedora releases (f26, f27, etc)
Stream names match the Platform module. We follow its policy here, even when it changes.
Oh, interesting, I had that backwards (I thought the planned F27 Python modules were the ones named after upstream Python releases).
That means the current module definitions would be closer to what I'm calling the "Integrated Python Modules".
- Python Application Runtime Modules (already planned for F27)
- one module for Python 2 and one for Python 3
- stream names match upstream Python releases (2.7, 3.6, etc)
- Application Runtime Modules conflict with the corresponding >
Integrated Python Module
- provide "/usr/bin/python2" and "/usr/bin/python3" respectively
- also satisfy "python2-*" and "python3-*" RPM dependencies
- only the Python 2 module satisfies "python-*" RPM dependencies (for
now) > * Integrated Python Modules
- one module for Python 2 and one for Python 3
These need to be parallel installable, to support tox. So we need separate modules for each Python release.
Aye, I agree that will be a good way to tackle the parallel installable versions that *don't* define "/usr/bin/python3". However, for containers designed to run Python applications (web or otherwise), we *will* want mutually conflicting streams that define their symlinks and RPM provides based only on the Python major version.
So lets make those extra modules their own distinct category:
* Python Interoperability Testing Modules - one module per Python X.Y release - only one stream per module - conflict with the corresponding Application Runtime stream - provide "/usr/bin/pythonXY" and "/usr/bin/pythonX.Y" - do NOT satisfy "python2-*" and "python3-*" RPM dependencies
And looking at their current implementation in Fedora, one notable difference between them and the Application Runtime streams is that these would just use their bundled pip and setuptools, rather than splitting those out the way the regular packages do.
- stream names match Fedora releases (f26, f27, etc)
- each depends on an *exact* version of the corresponding Python Application Runtime module
Huh? You just said Python Application Runtime Modules would conflict with this.
Sorry about that - when I started writing the email, my plan was to have them conflict with each other, but as I wrote that initial version up I went "Hang on, couldn't I use a stream dependency here and rely on *that* to trigger a conflict if you had an Integrated Python module installed and then tried to switch to an unsupported runtime stream, or vice-versa?"
- Default Python Module
- provides /usr/bin/python (and nothing else)
- stream names: no-default, python3-default, python2-default
+1 I guess this would contain a "python" RPM, containing just a /usr/bin/python symlink, which would added to non-modular Fedora as well.
Pretty much, yeah. The "no-default" stream would be a bit different, as in that case it would install a shell script that printed out a custom error message.
Cheers, Nick.
On 08/24/2017 12:22 PM, Nick Coghlan wrote:
On 24 August 2017 at 19:02, Petr Viktorin pviktori@redhat.com wrote:
On 08/24/2017 10:13 AM, Nick Coghlan wrote:
My current thinking based on that discussion is that we're actually going to need a module set that looks like this for F28+:
- Platform Python (already planned for F27)
- part of the Platform module
- stream names match Fedora releases (f26, f27, etc)
Stream names match the Platform module. We follow its policy here, even when it changes.
Oh, interesting, I had that backwards (I thought the planned F27 Python modules were the ones named after upstream Python releases).
That means the current module definitions would be closer to what I'm calling the "Integrated Python Modules".
My comment was for Platform Python :)
- Python Application Runtime Modules (already planned for F27)
- one module for Python 2 and one for Python 3
- stream names match upstream Python releases (2.7, 3.6, etc)
- Application Runtime Modules conflict with the corresponding >
Integrated Python Module - provide "/usr/bin/python2" and "/usr/bin/python3" respectively - also satisfy "python2-*" and "python3-*" RPM dependencies - only the Python 2 module satisfies "python-*" RPM dependencies (for now) > * Integrated Python Modules - one module for Python 2 and one for Python 3
These need to be parallel installable, to support tox. So we need separate modules for each Python release.
Aye, I agree that will be a good way to tackle the parallel installable versions that *don't* define "/usr/bin/python3". However, for containers designed to run Python applications (web or otherwise), we *will* want mutually conflicting streams that define their symlinks and RPM provides based only on the Python major version.
So lets make those extra modules their own distinct category:
- Python Interoperability Testing Modules
- one module per Python X.Y release
- only one stream per module
- conflict with the corresponding Application Runtime stream
- provide "/usr/bin/pythonXY" and "/usr/bin/pythonX.Y"
- do NOT satisfy "python2-*" and "python3-*" RPM dependencies
And looking at their current implementation in Fedora, one notable difference between them and the Application Runtime streams is that these would just use their bundled pip and setuptools, rather than splitting those out the way the regular packages do.
No, I don't want to support *full* parallel-installable stacks. Let's stick to what we currently support in Fedora, not more.
"Python Interoperability Testing Modules" get "x.y" streams and provide "x.y" binaries; "Python Application Runtime Modules" get one stream for python3 and one python2, and provide the python2/python3 binaries.
I don't think "Integrated Python Modules" are necessary in Fedora.
Also, the names are a mouthful; let's revise naming after we get the semantics down :)
- stream names match Fedora releases (f26, f27, etc) - each depends on an *exact* version of the corresponding Python Application Runtime moduleHuh? You just said Python Application Runtime Modules would conflict with this.
Sorry about that - when I started writing the email, my plan was to have them conflict with each other, but as I wrote that initial version up I went "Hang on, couldn't I use a stream dependency here and rely on *that* to trigger a conflict if you had an Integrated Python module installed and then tried to switch to an unsupported runtime stream, or vice-versa?"
- Default Python Module
- provides /usr/bin/python (and nothing else)
- stream names: no-default, python3-default, python2-default
+1 I guess this would contain a "python" RPM, containing just a /usr/bin/python symlink, which would added to non-modular Fedora as well.
Pretty much, yeah. The "no-default" stream would be a bit different, as in that case it would install a shell script that printed out a custom error message.
A shell script that would satisfy file dependencies on "/usr/bin/python" might not be the best idea.
On 24 August 2017 at 21:28, Petr Viktorin pviktori@redhat.com wrote:
On 08/24/2017 12:22 PM, Nick Coghlan wrote:
Stream names match the Platform module. We follow its policy here, even when it changes.
Oh, interesting, I had that backwards (I thought the planned F27 Python modules were the ones named after upstream Python releases).
That means the current module definitions would be closer to what I'm calling the "Integrated Python Modules".
My comment was for Platform Python :)
Oops, reading comprehension fail on my part :)
- Python Interoperability Testing Modules
- one module per Python X.Y release
- only one stream per module
- conflict with the corresponding Application Runtime stream
- provide "/usr/bin/pythonXY" and "/usr/bin/pythonX.Y"
- do NOT satisfy "python2-*" and "python3-*" RPM dependencies
And looking at their current implementation in Fedora, one notable difference between them and the Application Runtime streams is that these would just use their bundled pip and setuptools, rather than splitting those out the way the regular packages do.
No, I don't want to support *full* parallel-installable stacks. Let's stick to what we currently support in Fedora, not more.
"Python Interoperability Testing Modules" get "x.y" streams and provide "x.y" binaries;
These modules will probably only have one stream each, so I don't have a strong personal opinion on what that stream is called (although it's also possible they could each end up with two streams: one that actually installs the parallel installable version, and one that just depends on the corresponding application runtime stream).
"Python Application Runtime Modules" get one stream for python3 and one python2, and provide the python2/python3 binaries.
I think we can do better than that, and the relative timing of the F28 and Python 3.7 releases provides a good illustration of why I think we should aim to do so:
* F28 code freeze is in March, with the release in May * 3.7 release candidate is in May, with the release in June
Traditionally, that would have meant that we wouldn't get a Fedora based Python 3.7 container out the door until the release of Fedora 29 in October, and we'd never ship the F28+3.7 combination at all, even for folks that mainly just want the Python runtime, and then will use pip to install everything else they need.
However, I think modularity gives us access to a more flexible approach: we can set up the F28 System Profile to *default* to the 3.6 Python App Runtime stream, allowing the sequence of updates to be:
1. In Fedora 28, we split the Python 3 runtime module into two pieces: - the App Runtime, with a 3.6 stream - the Integrated Runtime, with an f28 stream that depends on AppRuntime:3.6 2. After 3.7 hits beta in January 2018, we add a "3.7" stream to the App Runtime 3. After 3.7 is released, we start building a new F28+3.7 app development container - in this configuration, most system packages that need Python 3 can't be installed 4. In Fedora 29, we add an f29 stream to the integrated runtime that depends on AppRuntime:3.7 - now it would be the F29+3.6 configuration that prevents use of system packages that need Py3
I don't think "Integrated Python Modules" are necessary in Fedora.
Whether they're necessary or not depends on whether we enable the F28+3.7 configuration: if we allow that, then we need a way to make it clear that in that configuration, the only Python-3-based system packages you can install are the ones that rely on /usr/libexec/platform-python instead of /usr/bin/python3 (since the others ran their integration testing against the default 3.6 stack).
Also, the names are a mouthful; let's revise naming after we get the semantics down :)
I'm reasonably happy with App Runtime (since I stole that directly from "OpenShift Application Runtimes", which is the downstream use case I'm interested in making easier to maintain), and I think "Integrated Python" or "Integrated Runtime" accurately conveys the significance of the proposed stream mapping between Fedora versions and Python versions ("F28 integration testing uses Python 3.6", "F29 integration testing uses Python 3.7", etc).
For the Testing Runtimes, we could likely just drop the "interoperability" qualifier, giving:
- Platform Python: the private, always installed, potentially non-standards-compliant, Python that dnf uses - Integrated Python: an optional standards-compliant Python used for development and automated scripting *of* Fedora - Application Python: used to run Python apps *on* Fedora (but may not be integrated fully, depending on version) - Testing Python: sufficient for cross-version compatibility testing, but not recommended for app deployments
- Default Python Module
I guess this would contain a "python" RPM, containing just a /usr/bin/python symlink, which would added to non-modular Fedora as well.
Pretty much, yeah. The "no-default" stream would be a bit different, as in that case it would install a shell script that printed out a custom error message.
A shell script that would satisfy file dependencies on "/usr/bin/python" might not be the best idea.
Ugh, you're right - I completely overlooked that problem when I dreamed up that particular hack. That means providing a nicer error message than the default "interpreter not found" one would depend on either:
- being able to "anti-register" a file in RPM and/or DNF (i.e. saying "I know I put a file there, but it's not a real implementation, so please don't actually use it to satisfy any dependencies") - providing some other way to customise the "interpreter not found" message that doesn't involve installing a stub interpreter implementation
While the first of those actually sounds vaguely plausible, I'm not sure I dislike the current error message enough to put in *that* much effort into trying to improve it.
Cheers, Nick.
On 08/25/2017 10:23 AM, Nick Coghlan wrote:
On 24 August 2017 at 21:28, Petr Viktorin pviktori@redhat.com wrote:
On 08/24/2017 12:22 PM, Nick Coghlan wrote:
Stream names match the Platform module. We follow its policy here, even when it changes.
Oh, interesting, I had that backwards (I thought the planned F27 Python modules were the ones named after upstream Python releases).
That means the current module definitions would be closer to what I'm calling the "Integrated Python Modules".
My comment was for Platform Python :)
Oops, reading comprehension fail on my part :)
- Python Interoperability Testing Modules
- one module per Python X.Y release
- only one stream per module
- conflict with the corresponding Application Runtime stream
- provide "/usr/bin/pythonXY" and "/usr/bin/pythonX.Y"
- do NOT satisfy "python2-*" and "python3-*" RPM dependencies
And looking at their current implementation in Fedora, one notable difference between them and the Application Runtime streams is that these would just use their bundled pip and setuptools, rather than splitting those out the way the regular packages do.
No, I don't want to support *full* parallel-installable stacks. Let's stick to what we currently support in Fedora, not more.
"Python Interoperability Testing Modules" get "x.y" streams and provide "x.y" binaries;
These modules will probably only have one stream each, so I don't have a strong personal opinion on what that stream is called (although it's also possible they could each end up with two streams: one that actually installs the parallel installable version, and one that just depends on the corresponding application runtime stream).
"Python Application Runtime Modules" get one stream for python3 and one python2, and provide the python2/python3 binaries.
I think we can do better than that, and the relative timing of the F28 and Python 3.7 releases provides a good illustration of why I think we should aim to do so:
- F28 code freeze is in March, with the release in May
- 3.7 release candidate is in May, with the release in June
Traditionally, that would have meant that we wouldn't get a Fedora based Python 3.7 container out the door until the release of Fedora 29 in October, and we'd never ship the F28+3.7 combination at all, even for folks that mainly just want the Python runtime, and then will use pip to install everything else they need.
Traditionally, that's very well possible: there'd be a "python37" package that provides a /usr/bin/python37. We do the same for old versions, and for python36 on f25. It works quite well.
If the intended use for 3.7 in f28 is to make a virtualenv and use pip to install things, that virtualenv can very well be made by calling /usr/bin/python37. I still see no reason for a stream that makes /usr/bin/python3 point to 3.7.
However, I think modularity gives us access to a more flexible approach: we can set up the F28 System Profile to *default* to the 3.6 Python App Runtime stream, allowing the sequence of updates to be:
- In Fedora 28, we split the Python 3 runtime module into two pieces:
- the App Runtime, with a 3.6 stream
- the Integrated Runtime, with an f28 stream that depends on AppRuntime:3.6
- After 3.7 hits beta in January 2018, we add a "3.7" stream to the App Runtime
- After 3.7 is released, we start building a new F28+3.7 app
development container
- in this configuration, most system packages that need Python 3 can't
be installed
How do you achieve this for package that depend on /usr/bin/python3?
- In Fedora 29, we add an f29 stream to the integrated runtime that
depends on AppRuntime:3.7
- now it would be the F29+3.6 configuration that prevents use of
system packages that need Py3
I don't think "Integrated Python Modules" are necessary in Fedora.
Whether they're necessary or not depends on whether we enable the F28+3.7 configuration: if we allow that, then we need a way to make it clear that in that configuration, the only Python-3-based system packages you can install are the ones that rely on /usr/libexec/platform-python instead of /usr/bin/python3 (since the others ran their integration testing against the default 3.6 stack).
The only thing that relies on /usr/libexec/platform-python is dnf. Nothing else.
Also, the names are a mouthful; let's revise naming after we get the semantics down :)
I'm reasonably happy with App Runtime (since I stole that directly from "OpenShift Application Runtimes", which is the downstream use case I'm interested in making easier to maintain), and I think "Integrated Python" or "Integrated Runtime" accurately conveys the significance of the proposed stream mapping between Fedora versions and Python versions ("F28 integration testing uses Python 3.6", "F29 integration testing uses Python 3.7", etc).
For the Testing Runtimes, we could likely just drop the "interoperability" qualifier, giving:
- Platform Python: the private, always installed, potentially
non-standards-compliant, Python that dnf uses
- Integrated Python: an optional standards-compliant Python used for
development and automated scripting *of* Fedora
- Application Python: used to run Python apps *on* Fedora (but may not
be integrated fully, depending on version)
- Testing Python: sufficient for cross-version compatibility testing,
but not recommended for app deployments
It seems that the Application and Testing Pythons differ only in "SLA"s and availablility of additional packages, is that right?
Also, it seems that the difference between Integrated Python and Application Python is that one provides /usr/bin/python3. Is that correct?
On 29 August 2017 at 19:33, Petr Viktorin pviktori@redhat.com wrote:
On 08/25/2017 10:23 AM, Nick Coghlan wrote:
Traditionally, that would have meant that we wouldn't get a Fedora based Python 3.7 container out the door until the release of Fedora 29 in October, and we'd never ship the F28+3.7 combination at all, even for folks that mainly just want the Python runtime, and then will use pip to install everything else they need.
Traditionally, that's very well possible: there'd be a "python37" package that provides a /usr/bin/python37. We do the same for old versions, and for python36 on f25. It works quite well.
If the intended use for 3.7 in f28 is to make a virtualenv and use pip to install things, that virtualenv can very well be made by calling /usr/bin/python37. I still see no reason for a stream that makes /usr/bin/python3 point to 3.7.
My aim is to be able to define a "Fedora 28 + Python 3.7" container image that meets the following criteria:
1. Everything that's installed by default is owned by an RPM (no injecting extra symlinks as part of the container build process) 2. Invocations of "python3" and "/usr/bin/python3" (whether interactively or via shebang) run Python 3.7 3. I'm able to do this with just the Fedora Modularity tooling - no extra downstream tools like scl-utils
However, I think modularity gives us access to a more flexible approach: we can set up the F28 System Profile to *default* to the 3.6 Python App Runtime stream, allowing the sequence of updates to be:
- In Fedora 28, we split the Python 3 runtime module into two pieces:
- the App Runtime, with a 3.6 stream
- the Integrated Runtime, with an f28 stream that depends on
AppRuntime:3.6 2. After 3.7 hits beta in January 2018, we add a "3.7" stream to the App Runtime 3. After 3.7 is released, we start building a new F28+3.7 app development container
- in this configuration, most system packages that need Python 3 can't
be installed
How do you achieve this for package that depend on /usr/bin/python3?
If that's all they depend on, I guess they'll still be installable. However, if they depend on a particular version of the Python ABI, then the 3.7 package won't provide it.
Also, the names are a mouthful; let's revise naming after we get the semantics down :)
I'm reasonably happy with App Runtime (since I stole that directly from "OpenShift Application Runtimes", which is the downstream use case I'm interested in making easier to maintain), and I think "Integrated Python" or "Integrated Runtime" accurately conveys the significance of the proposed stream mapping between Fedora versions and Python versions ("F28 integration testing uses Python 3.6", "F29 integration testing uses Python 3.7", etc).
For the Testing Runtimes, we could likely just drop the "interoperability" qualifier, giving:
- Platform Python: the private, always installed, potentially
non-standards-compliant, Python that dnf uses
- Integrated Python: an optional standards-compliant Python used for
development and automated scripting *of* Fedora
- Application Python: used to run Python apps *on* Fedora (but may not
be integrated fully, depending on version)
- Testing Python: sufficient for cross-version compatibility testing,
but not recommended for app deployments
It seems that the Application and Testing Pythons differ only in "SLA"s and availablility of additional packages, is that right?
Also, it seems that the difference between Integrated Python and Application Python is that one provides /usr/bin/python3. Is that correct?
Originally that's what I had, but as I wrote it up in more detail (see https://github.com/fedora-python/fedora-python/commit/e76aa19870627059c9b153...) I came to realise that it probably makes more sense to have a module level dependency from the Integrated Python branches to the corresponding Application Python branches and then have the "python3 symlink or not?" distinction exist between the Application module config and the Testing module configs.
That way, the "Integrated Python" module shouldn't actually need to pull in any new source trees - it just becomes a way of mapping from Fedora versions to their integrated Python versions in a machine-readable way such that other Fedora modules can easily declare either "build for all Fedora Integrated Python streams" (by defining a module dependency on "python: []") *or* "build for all CPython version streams" (by depending on "cpython: []").
While exactly how the stream expansions are going to work is still a bit fuzzy (and I assume that's one of the things that will be resolved at Flock this week), my understanding based on the last fedora-devel thread about it is that we'll want to actively discourage folks from doing the latter without a compelling reason, as that's the approach that can lead to a combinatorial explosion of component builds: every version of Python on every active version of Fedora. While there are some cases where we'll actually want to do that (e.g. for the python-ecosystem modules), there are a lot of Fedora components that only need to be usable on the Python build that's integrated directly with the OS.
Cheers, Nick.
python-devel@lists.fedoraproject.org
-
Nick Coghlan -
Petr Viktorin