Hello,
we are considering to add a %pycached macro to be used in the %files section:
https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/37
We'd like to receive feedback. We plan to add it to rawhide and backport it to all Fedoras + EPEL7+.
Usage:
%files ... %pycached %{python3_sitelib}/foo.py
This will list:
/usr/lib/python3.8/site-packages/foo.py /usr/lib/python3.8/site-packages/__pycache__/foo.cpython-38{,.opt-?}.pyc
Assuming the Python 3 version is 3.8. The bytecode files are globbed, their presence is not checked.
This will fail:
%pycached %{python3_sitelib}/foo
error: %pycached can only be used with paths explicitly ending with .py
And so will any of this:
%pycached %{python3_sitelib}/* %pycached %{python3_sitelib}/foo.* %pycached %{python3_sitelib}/foo.p? %pycached %{python3_sitelib}/foo.?y %pycached %{python3_sitelib}/foo.??
But this will work:
%pycached %{python3_sitelib}/foo*.py
And it will generate the following globs:
/usr/lib/python3.8/site-packages/foo*.py /usr/lib/python3.8/site-packages/__pycache__/foo*.cpython-38{,.opt-?}.pyc
When used with paths that include Python 3 version, it globs with the version:
%pycached /opt/python3.10/foo.py
Generates:
/opt/python3.10/foo.py /opt/python3.10/__pycache__/foo.cpython-310{,.opt-?}.pyc
While paths without version have less strict globs:
%pycached /custom/foo.py /custom/foo.py /custom/__pycache__/foo.cpython-3*{,.opt-?}.pyc
This will generate a warning in RPM build:
warning: File listed twice: /custom/__pycache__/foo.cpython-38.opt-1.pyc
However it ensures the optimized bytecode is there.
I am against this. The *.py files should go to -src subpackage and should not be installed on my computer, because if I am not mistaken, they are useless when the *.pyc files are installed.
Vít
Dne 12. 12. 19 v 15:36 Miro Hrončok napsal(a):
Hello,
we are considering to add a %pycached macro to be used in the %files section:
https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/37
We'd like to receive feedback. We plan to add it to rawhide and backport it to all Fedoras + EPEL7+.
Usage:
%files ... %pycached %{python3_sitelib}/foo.py
This will list:
/usr/lib/python3.8/site-packages/foo.py /usr/lib/python3.8/site-packages/__pycache__/foo.cpython-38{,.opt-?}.pyc
Assuming the Python 3 version is 3.8. The bytecode files are globbed, their presence is not checked.
This will fail:
%pycached %{python3_sitelib}/foo
error: %pycached can only be used with paths explicitly ending with .py
And so will any of this:
%pycached %{python3_sitelib}/* %pycached %{python3_sitelib}/foo.* %pycached %{python3_sitelib}/foo.p? %pycached %{python3_sitelib}/foo.?y %pycached %{python3_sitelib}/foo.??
But this will work:
%pycached %{python3_sitelib}/foo*.py
And it will generate the following globs:
/usr/lib/python3.8/site-packages/foo*.py /usr/lib/python3.8/site-packages/__pycache__/foo*.cpython-38{,.opt-?}.pyc
When used with paths that include Python 3 version, it globs with the version:
%pycached /opt/python3.10/foo.py
Generates:
/opt/python3.10/foo.py /opt/python3.10/__pycache__/foo.cpython-310{,.opt-?}.pyc
While paths without version have less strict globs:
%pycached /custom/foo.py /custom/foo.py /custom/__pycache__/foo.cpython-3*{,.opt-?}.pyc
This will generate a warning in RPM build:
warning: File listed twice: /custom/__pycache__/foo.cpython-38.opt-1.pyc
However it ensures the optimized bytecode is there.
On 12. 12. 19 16:36, Vít Ondruch wrote:
I am against this. The *.py files should go to -src subpackage and should not be installed on my computer, because if I am not mistaken, they are useless when the *.pyc files are installed.
The macro doesn't change the way we distribute files, it merely makes the status quo easier for packagers.
It doesn't in any way interfere to your proposal. Yet you are against this. That's not helpful feedback.
It's like saying: I am against this bike shad being painted blue, because people should not ride bikes to work.
Dne 12. 12. 19 v 16:42 Miro Hrončok napsal(a):
On 12. 12. 19 16:36, Vít Ondruch wrote:
I am against this. The *.py files should go to -src subpackage and should not be installed on my computer, because if I am not mistaken, they are useless when the *.pyc files are installed.
The macro doesn't change the way we distribute files, it merely makes the status quo easier for packagers.
It doesn't in any way interfere to your proposal. Yet you are against this. That's not helpful feedback.
I am against, because it makes any transition to -src subpackages harder by cementing the status quo. And I am trying to focus you on something more valuable IMO.
Vít
It's like saying: I am against this bike shad being painted blue, because people should not ride bikes to work.
On 12. 12. 19 16:46, Vít Ondruch wrote:
Dne 12. 12. 19 v 16:42 Miro Hrončok napsal(a):
On 12. 12. 19 16:36, Vít Ondruch wrote:
I am against this. The *.py files should go to -src subpackage and should not be installed on my computer, because if I am not mistaken, they are useless when the *.pyc files are installed.
The macro doesn't change the way we distribute files, it merely makes the status quo easier for packagers.
It doesn't in any way interfere to your proposal. Yet you are against this. That's not helpful feedback.
I am against, because it makes any transition to -src subpackages harder by cementing the status quo. And I am trying to focus you on something more valuable IMO.
Cementing how? In fact, it makes it easier to modify the macro later to do something different (for example excluding the source and keeping the bytecode only), not possible in what we currently have (hardcoded bytecode path globs in every spec that needs it).
Dne 12. 12. 19 v 16:36 Vít Ondruch napsal(a):
I am against this. The *.py files should go to -src subpackage and should not be installed on my computer, because if I am not mistaken, they are useless
s/useless/not used/, because somebody might admit that they are useful to explore the code and tinker with it.
Vít
when the *.pyc files are installed.
Vít
Dne 12. 12. 19 v 15:36 Miro Hrončok napsal(a):
Hello,
we are considering to add a %pycached macro to be used in the %files section:
https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/37
We'd like to receive feedback. We plan to add it to rawhide and backport it to all Fedoras + EPEL7+.
Usage:
%files ... %pycached %{python3_sitelib}/foo.py
This will list:
/usr/lib/python3.8/site-packages/foo.py /usr/lib/python3.8/site-packages/__pycache__/foo.cpython-38{,.opt-?}.pyc
Assuming the Python 3 version is 3.8. The bytecode files are globbed, their presence is not checked.
This will fail:
%pycached %{python3_sitelib}/foo
error: %pycached can only be used with paths explicitly ending with .py
And so will any of this:
%pycached %{python3_sitelib}/* %pycached %{python3_sitelib}/foo.* %pycached %{python3_sitelib}/foo.p? %pycached %{python3_sitelib}/foo.?y %pycached %{python3_sitelib}/foo.??
But this will work:
%pycached %{python3_sitelib}/foo*.py
And it will generate the following globs:
/usr/lib/python3.8/site-packages/foo*.py /usr/lib/python3.8/site-packages/__pycache__/foo*.cpython-38{,.opt-?}.pyc
When used with paths that include Python 3 version, it globs with the version:
%pycached /opt/python3.10/foo.py
Generates:
/opt/python3.10/foo.py /opt/python3.10/__pycache__/foo.cpython-310{,.opt-?}.pyc
While paths without version have less strict globs:
%pycached /custom/foo.py /custom/foo.py /custom/__pycache__/foo.cpython-3*{,.opt-?}.pyc
This will generate a warning in RPM build:
warning: File listed twice: /custom/__pycache__/foo.cpython-38.opt-1.pyc
However it ensures the optimized bytecode is there.
packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject....
I though about this a bit and to be more constructive, I think that you should probably move the file list creation out of %files section. Do it probably in %install section instead. That way, you would have more flexibility. You could have something like "%pyfile %{python3_sitelib}/foo.py" and it would do on background something like:
~~~ echo %{python3_sitelib}/foo.py > pyfiles.lst echo %{python3_sitelib}/foo.pyc > pycache.lst ~~~
Later you would just submit these file lists to the appropriate packages. This would be IMO more flexible.
Vít
Dne 12. 12. 19 v 15:36 Miro Hrončok napsal(a):
Hello,
we are considering to add a %pycached macro to be used in the %files section:
https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/37
We'd like to receive feedback. We plan to add it to rawhide and backport it to all Fedoras + EPEL7+.
Usage:
%files ... %pycached %{python3_sitelib}/foo.py
This will list:
/usr/lib/python3.8/site-packages/foo.py /usr/lib/python3.8/site-packages/__pycache__/foo.cpython-38{,.opt-?}.pyc
Assuming the Python 3 version is 3.8. The bytecode files are globbed, their presence is not checked.
This will fail:
%pycached %{python3_sitelib}/foo
error: %pycached can only be used with paths explicitly ending with .py
And so will any of this:
%pycached %{python3_sitelib}/* %pycached %{python3_sitelib}/foo.* %pycached %{python3_sitelib}/foo.p? %pycached %{python3_sitelib}/foo.?y %pycached %{python3_sitelib}/foo.??
But this will work:
%pycached %{python3_sitelib}/foo*.py
And it will generate the following globs:
/usr/lib/python3.8/site-packages/foo*.py /usr/lib/python3.8/site-packages/__pycache__/foo*.cpython-38{,.opt-?}.pyc
When used with paths that include Python 3 version, it globs with the version:
%pycached /opt/python3.10/foo.py
Generates:
/opt/python3.10/foo.py /opt/python3.10/__pycache__/foo.cpython-310{,.opt-?}.pyc
While paths without version have less strict globs:
%pycached /custom/foo.py /custom/foo.py /custom/__pycache__/foo.cpython-3*{,.opt-?}.pyc
This will generate a warning in RPM build:
warning: File listed twice: /custom/__pycache__/foo.cpython-38.opt-1.pyc
However it ensures the optimized bytecode is there.
On 2019-12-16 10:19, Vít Ondruch wrote:
I though about this a bit and to be more constructive, I think that you should probably move the file list creation out of %files section. Do it probably in %install section instead. That way, you would have more flexibility. You could have something like "%pyfile %{python3_sitelib}/foo.py" and it would do on background something like:
echo %{python3_sitelib}/foo.py > pyfiles.lst echo %{python3_sitelib}/foo.pyc > pycache.lst
Later you would just submit these file lists to the appropriate packages. This would be IMO more flexible.
Vít
I believe this is a good general direction to go in, but the devil is in the details. You're asking to change the way file lists are generated, which is a lot more than a macro to simplify current practice.
You seem to want to not have .py files from the system, and only install .pyc. The basic issue there is that the current way (.py+.pyc) is to install:
%{python3_sitelib}/foo.py %{python3_sitelib}/__pycache__/foo.cpython-38.*.pyc
... but to make a lone .pyc importable, it needs to be at:
%{python3_sitelib}/foo.pyc
... i.e. it's not only a matter of omitting the .py file, but you also need to rename the .pyc. That is solvable, of course, but it's just the start of the rabbit hole. There are enough other issues and details to be thought out that I'm not sure your %pyfile macro would be useful in the end. (We're actually brainstorming ideas to minimize Python's footprint internally, but right now is not a good time so start a big flaming discussion -- many of us are taking a break from computers near the end of the year. So we're holding the discussion off until next year.)
Back to the proposed %pycached macro: I can see how it looks like a step backwards, if your goal is minimization. But I believe it's orthogonal, a step sideways :) It's meant to make current spec files specify the intent rather than the implementation, so we're better able to tweak the implementation. (Not just in released Fedora or CPython, but also for things like experiments in COPR.)
Dne 12. 12. 19 v 15:36 Miro Hrončok napsal(a):
Hello,
we are considering to add a %pycached macro to be used in the %files section:
https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/37
We'd like to receive feedback. We plan to add it to rawhide and backport it to all Fedoras + EPEL7+.
Usage:
%files ... %pycached %{python3_sitelib}/foo.py
This will list:
/usr/lib/python3.8/site-packages/foo.py
/usr/lib/python3.8/site-packages/__pycache__/foo.cpython-38{,.opt-?}.pyc
Assuming the Python 3 version is 3.8. The bytecode files are globbed, their presence is not checked.
This will fail:
%pycached %{python3_sitelib}/foo
error: %pycached can only be used with paths explicitly ending with .py
And so will any of this:
%pycached %{python3_sitelib}/* %pycached %{python3_sitelib}/foo.* %pycached %{python3_sitelib}/foo.p? %pycached %{python3_sitelib}/foo.?y %pycached %{python3_sitelib}/foo.??
But this will work:
%pycached %{python3_sitelib}/foo*.py
And it will generate the following globs:
/usr/lib/python3.8/site-packages/foo*.py
/usr/lib/python3.8/site-packages/__pycache__/foo*.cpython-38{,.opt-?}.pyc
When used with paths that include Python 3 version, it globs with the version:
%pycached /opt/python3.10/foo.py
Generates:
/opt/python3.10/foo.py /opt/python3.10/__pycache__/foo.cpython-310{,.opt-?}.pyc
While paths without version have less strict globs:
%pycached /custom/foo.py /custom/foo.py /custom/__pycache__/foo.cpython-3*{,.opt-?}.pyc
This will generate a warning in RPM build:
warning: File listed twice: /custom/__pycache__/foo.cpython-38.opt-1.pyc
However it ensures the optimized bytecode is there.
Dne 16. 12. 19 v 11:22 Petr Viktorin napsal(a):
On 2019-12-16 10:19, Vít Ondruch wrote:
I though about this a bit and to be more constructive, I think that you should probably move the file list creation out of %files section. Do it probably in %install section instead. That way, you would have more flexibility. You could have something like "%pyfile %{python3_sitelib}/foo.py" and it would do on background something like:
echo %{python3_sitelib}/foo.py > pyfiles.lst echo %{python3_sitelib}/foo.pyc > pycache.lst
Later you would just submit these file lists to the appropriate packages. This would be IMO more flexible.
Vít
I believe this is a good general direction to go in, but the devil is in the details. You're asking to change the way file lists are generated, which is a lot more than a macro to simplify current practice.
I think I understand and agree with your concerns and you are right that I don't understand the details, because I was not aware about the placement of *.pyc files. But I disagree with that it "is a lot more than a macro to simplify current practice"
The Miro's proposal is to replace:
~~~
%files %{python3_sitelib}/foo.py %{python3_sitelib}/__pycache__/foo.cpython-38.*.pyc
~~~
by
~~~
%pycached %{python3_sitelib}/foo.py
~~~
which is IMO making any future progress harder. Instead, my proposal is to replace:
~~~
%install ... snip ...
%files %{python3_sitelib}/foo.py %{python3_sitelib}/__pycache__/foo.cpython-38.*.pyc
~~~
by
~~~
%install ... snip ... %pyfile %{python3_sitelib}/foo.py
%files -l pyfiles.lst
~~~
This does not add any complexity while it helps to abstract away the .py/.pyc. IOW it still removes the "pyc" lines, while it opens the door for the future.
Vít
You seem to want to not have .py files from the system, and only install .pyc. The basic issue there is that the current way (.py+.pyc) is to install:
%{python3_sitelib}/foo.py %{python3_sitelib}/__pycache__/foo.cpython-38.*.pyc
... but to make a lone .pyc importable, it needs to be at:
%{python3_sitelib}/foo.pyc
... i.e. it's not only a matter of omitting the .py file, but you also need to rename the .pyc. That is solvable, of course, but it's just the start of the rabbit hole. There are enough other issues and details to be thought out that I'm not sure your %pyfile macro would be useful in the end. (We're actually brainstorming ideas to minimize Python's footprint internally, but right now is not a good time so start a big flaming discussion -- many of us are taking a break from computers near the end of the year. So we're holding the discussion off until next year.)
Back to the proposed %pycached macro: I can see how it looks like a step backwards, if your goal is minimization. But I believe it's orthogonal, a step sideways :) It's meant to make current spec files specify the intent rather than the implementation, so we're better able to tweak the implementation. (Not just in released Fedora or CPython, but also for things like experiments in COPR.)
Dne 12. 12. 19 v 15:36 Miro Hrončok napsal(a):
Hello,
we are considering to add a %pycached macro to be used in the %files section:
https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/37
We'd like to receive feedback. We plan to add it to rawhide and backport it to all Fedoras + EPEL7+.
Usage:
%files ... %pycached %{python3_sitelib}/foo.py
This will list:
/usr/lib/python3.8/site-packages/foo.py /usr/lib/python3.8/site-packages/__pycache__/foo.cpython-38{,.opt-?}.pyc
Assuming the Python 3 version is 3.8. The bytecode files are globbed, their presence is not checked.
This will fail:
%pycached %{python3_sitelib}/foo
error: %pycached can only be used with paths explicitly ending with .py
And so will any of this:
%pycached %{python3_sitelib}/* %pycached %{python3_sitelib}/foo.* %pycached %{python3_sitelib}/foo.p? %pycached %{python3_sitelib}/foo.?y %pycached %{python3_sitelib}/foo.??
But this will work:
%pycached %{python3_sitelib}/foo*.py
And it will generate the following globs:
/usr/lib/python3.8/site-packages/foo*.py /usr/lib/python3.8/site-packages/__pycache__/foo*.cpython-38{,.opt-?}.pyc
When used with paths that include Python 3 version, it globs with the version:
%pycached /opt/python3.10/foo.py
Generates:
/opt/python3.10/foo.py /opt/python3.10/__pycache__/foo.cpython-310{,.opt-?}.pyc
While paths without version have less strict globs:
%pycached /custom/foo.py /custom/foo.py /custom/__pycache__/foo.cpython-3*{,.opt-?}.pyc
This will generate a warning in RPM build:
warning: File listed twice: /custom/__pycache__/foo.cpython-38.opt-1.pyc
However it ensures the optimized bytecode is there.
On 2019-12-16 12:22, Vít Ondruch wrote:
Dne 16. 12. 19 v 11:22 Petr Viktorin napsal(a):
On 2019-12-16 10:19, Vít Ondruch wrote:
I though about this a bit and to be more constructive, I think that you should probably move the file list creation out of %files section. Do it probably in %install section instead. That way, you would have more flexibility. You could have something like "%pyfile %{python3_sitelib}/foo.py" and it would do on background something like:
echo %{python3_sitelib}/foo.py > pyfiles.lst echo %{python3_sitelib}/foo.pyc > pycache.lst
Later you would just submit these file lists to the appropriate packages. This would be IMO more flexible.
Vít
I believe this is a good general direction to go in, but the devil is in the details. You're asking to change the way file lists are generated, which is a lot more than a macro to simplify current practice.
I think I understand and agree with your concerns and you are right that I don't understand the details, because I was not aware about the placement of *.pyc files. But I disagree with that it "is a lot more than a macro to simplify current practice"
The Miro's proposal is to replace:
%files %{python3_sitelib}/foo.py %{python3_sitelib}/__pycache__/foo.cpython-38.*.pyc
by
%pycached %{python3_sitelib}/foo.py
which is IMO making any future progress harder. Instead, my proposal is to replace:
%install ... snip ... %files %{python3_sitelib}/foo.py %{python3_sitelib}/__pycache__/foo.cpython-38.*.pyc
by
%install ... snip ... %pyfile %{python3_sitelib}/foo.py %files -l pyfiles.lst
This does not add any complexity while it helps to abstract away the .py/.pyc. IOW it still removes the "pyc" lines, while it opens the door for the future.
I'm afraid your proposal doesn't open it wide enough: to support py-less installs, it will have to be changed (or removed) anyway. I also disagree disagree about the extra complexity, if only in the packager's mental model: "%pycached" expands to the given file and its cache; "%pyfile" writes these to a file and then lets you use "%files -l" to read that file back in.
Also, "%pyfile" would introduce extra issues. For example, how would one split several Python modules across several subpackages?
On Mon, Dec 16, 2019 at 12:46 PM Petr Viktorin pviktori@redhat.com wrote:
On 2019-12-16 12:22, Vít Ondruch wrote:
Dne 16. 12. 19 v 11:22 Petr Viktorin napsal(a):
On 2019-12-16 10:19, Vít Ondruch wrote:
I though about this a bit and to be more constructive, I think that you should probably move the file list creation out of %files section. Do it probably in %install section instead. That way, you would have more flexibility. You could have something like "%pyfile %{python3_sitelib}/foo.py" and it would do on background something like:
echo %{python3_sitelib}/foo.py > pyfiles.lst echo %{python3_sitelib}/foo.pyc > pycache.lst
Later you would just submit these file lists to the appropriate packages. This would be IMO more flexible.
Vít
I believe this is a good general direction to go in, but the devil is in the details. You're asking to change the way file lists are generated, which is a lot more than a macro to simplify current practice.
I think I understand and agree with your concerns and you are right that I don't understand the details, because I was not aware about the placement of *.pyc files. But I disagree with that it "is a lot more than a macro to simplify current practice"
The Miro's proposal is to replace:
%files %{python3_sitelib}/foo.py %{python3_sitelib}/__pycache__/foo.cpython-38.*.pyc
by
%pycached %{python3_sitelib}/foo.py
which is IMO making any future progress harder. Instead, my proposal is to replace:
%install ... snip ... %files %{python3_sitelib}/foo.py %{python3_sitelib}/__pycache__/foo.cpython-38.*.pyc
by
%install ... snip ... %pyfile %{python3_sitelib}/foo.py %files -l pyfiles.lst
This does not add any complexity while it helps to abstract away the .py/.pyc. IOW it still removes the "pyc" lines, while it opens the door for the future.
I'm afraid your proposal doesn't open it wide enough: to support py-less installs, it will have to be changed (or removed) anyway. I also disagree disagree about the extra complexity, if only in the packager's mental model: "%pycached" expands to the given file and its cache; "%pyfile" writes these to a file and then lets you use "%files -l" to read that file back in.
Also, "%pyfile" would introduce extra issues. For example, how would one split several Python modules across several subpackages?
Yeah, I think using file lists works OK-ish for some things (translations with %find_lang, Java packaging, etc.), but it doesn't really map to the way python packages work in fedora. On the other hand, Miro's proposal is an incremental improvement over the status quo without introducing more complexity (it just expands one %files entry into two), and as such, I think it's a good idea.
Fabio
packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject....
Le 2019-12-16 12:52, Fabio Valentini a écrit :
On Mon, Dec 16, 2019 at 12:46 PM Petr Viktorin pviktori@redhat.com wrote:
Hi Fabio
Yeah, I think using file lists works OK-ish for some things (translations with %find_lang, Java packaging, etc.), but it doesn't really map to the way python packages work in fedora.
Practically, automating the packaging of any upstream distribution format, that does not resumes itself to a single kind of file in a single filesystem directory, will end up as file lists.
File lists are the only mechanism provided by rpm to communicate between %prep, %build, %install and %deploy (aka %files) sections. So as soon as you automate anything in any previous section, that has consequences on the deploy phase, you have to resort to file lists (shell variables do not pass the section barrier, rpm variables are evaluated before the spec is executed)
And even within an rpm section, the counter-productive decision to use bash in sh mode, means you do not have access to all the array manipulating facilities built-in bash (mapfile…). Since manipulating lists in legacy shell mode is horrible, you may as well resort to file lists even within a section.
Regards,
Dne 16. 12. 19 v 12:46 Petr Viktorin napsal(a):
On 2019-12-16 12:22, Vít Ondruch wrote:
Dne 16. 12. 19 v 11:22 Petr Viktorin napsal(a):
On 2019-12-16 10:19, Vít Ondruch wrote:
I though about this a bit and to be more constructive, I think that you should probably move the file list creation out of %files section. Do it probably in %install section instead. That way, you would have more flexibility. You could have something like "%pyfile %{python3_sitelib}/foo.py" and it would do on background something like:
echo %{python3_sitelib}/foo.py > pyfiles.lst echo %{python3_sitelib}/foo.pyc > pycache.lst
Later you would just submit these file lists to the appropriate packages. This would be IMO more flexible.
Vít
I believe this is a good general direction to go in, but the devil is in the details. You're asking to change the way file lists are generated, which is a lot more than a macro to simplify current practice.
I think I understand and agree with your concerns and you are right that I don't understand the details, because I was not aware about the placement of *.pyc files. But I disagree with that it "is a lot more than a macro to simplify current practice"
The Miro's proposal is to replace:
%files %{python3_sitelib}/foo.py %{python3_sitelib}/__pycache__/foo.cpython-38.*.pyc
by
%pycached %{python3_sitelib}/foo.py
which is IMO making any future progress harder. Instead, my proposal is to replace:
%install ... snip ... %files %{python3_sitelib}/foo.py %{python3_sitelib}/__pycache__/foo.cpython-38.*.pyc
by
%install ... snip ... %pyfile %{python3_sitelib}/foo.py %files -l pyfiles.lst
This does not add any complexity while it helps to abstract away the .py/.pyc. IOW it still removes the "pyc" lines, while it opens the door for the future.
I'm afraid your proposal doesn't open it wide enough: to support py-less installs, it will have to be changed (or removed) anyway.
Not sure what does "py-less installs" mean ¯_(ツ)_/¯
I also disagree disagree about the extra complexity, if only in the packager's mental model: "%pycached" expands to the given file and its cache; "%pyfile" writes these to a file and then lets you use "%files -l" to read that file back in.
I am not saying this is the best name. This is the first name I was able come with which makes at least a bit of sense.
Also, "%pyfile" would introduce extra issues. For example, how would one split several Python modules across several subpackages?
My proposal is high level idea. It is easy to shot it down based on "how would one split several Python modules across several subpackages?". Of course next step would be to output to wile lists, one for py files while the other for pyc files. If you are talking about other possilbe subpackages, it could be possible to use some parameter, or block to create appropriately named file list.
Vít
On 16. 12. 19 13:03, Vít Ondruch wrote:
My proposal is high level idea. It is easy to shot it down based on "how would one split several Python modules across several subpackages?". Of course next step would be to output to wile lists, one for py files while the other for pyc files. If you are talking about other possilbe subpackages, it could be possible to use some parameter, or block to create appropriately named file list.
My proposal is a low level idea. It is easy to shot it down based on "my high level idea I don't fully comprehend* is better than yours".
(* That's what you have admitted in this very thread.)
So let me summarize this here:
- I hear your high level idea. I heard it for the first time. - I plan to summarize Python Minimization ideas early 2020, including yours. - %pycached macro does not go against your idea. - You idea can be realized in rawhide only (if at all). - %pycached makes existing specs easier. - %pycached can be backported to older releases. - There is a volunteer to implement %pycached (done). - AFAIK There is no volunteer to implement your idea.
I appreciate your ideas, but providing them as negative feedback to %pycached is not fair. And call me bullheaded if you want, but I will not consider such feedback, as it is IMHO not helpful.
Now, can we please focus on actual feedback to %pycached? Possibly in a different subthread?
On Mon, Dec 16, 2019 at 7:22 AM Miro Hrončok mhroncok@redhat.com wrote:
On 16. 12. 19 13:03, Vít Ondruch wrote:
My proposal is high level idea. It is easy to shot it down based on "how would one split several Python modules across several subpackages?". Of course next step would be to output to wile lists, one for py files while the other for pyc files. If you are talking about other possilbe subpackages, it could be possible to use some parameter, or block to create appropriately named file list.
My proposal is a low level idea. It is easy to shot it down based on "my high level idea I don't fully comprehend* is better than yours".
(* That's what you have admitted in this very thread.)
So let me summarize this here:
- I hear your high level idea. I heard it for the first time.
- I plan to summarize Python Minimization ideas early 2020, including yours.
- %pycached macro does not go against your idea.
- You idea can be realized in rawhide only (if at all).
- %pycached makes existing specs easier.
- %pycached can be backported to older releases.
- There is a volunteer to implement %pycached (done).
- AFAIK There is no volunteer to implement your idea.
I appreciate your ideas, but providing them as negative feedback to %pycached is not fair. And call me bullheaded if you want, but I will not consider such feedback, as it is IMHO not helpful.
Now, can we please focus on actual feedback to %pycached? Possibly in a different subthread?
For what it's worth. I think %pycached is a nice improvement. I'm not sure if I like the name of it specifically, but the behavior is quite desirable.
In the next year, we could look at leveraging the INSTALLED_FILES stuff that setuptools, et al. produce and enhance it with %pycached (or whatever) so that we grab not just the files it copied, but all the byte-compiled files too. The mechanism in which to do so will be somewhat interesting, since making that file get produced during python builds is no longer quite so simple with the advent of PEP 517/518 based projects... But it's something to look forward to.
On 16. 12. 19 16:56, Neal Gompa wrote:
For what it's worth. I think %pycached is a nice improvement. I'm not sure if I like the name of it specifically, but the behavior is quite desirable.
Naming things ¯_(ツ)_/¯
I was thinking:
%with_pycache - collides with bconds %include_pycache - quite long %pyc - quite indecipherable %pycache - might be mistaken to only include the pyc files
%pycached doesn't sound that bad to me - include that file, but also its cached Python bytecode. However I am not a native speaker and I realize I often desecrate the English language. Got any better suggestions?
In the next year, we could look at leveraging the INSTALLED_FILES stuff that setuptools, et al. produce and enhance it with %pycached (or whatever) so that we grab not just the files it copied, but all the byte-compiled files too. The mechanism in which to do so will be somewhat interesting, since making that file get produced during python builds is no longer quite so simple with the advent of PEP 517/518 based projects... But it's something to look forward to.
See also this e-mail (the second half):
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
On Mon, Dec 16, 2019 at 1:00 PM Miro Hrončok mhroncok@redhat.com wrote:
On 16. 12. 19 16:56, Neal Gompa wrote:
For what it's worth. I think %pycached is a nice improvement. I'm not sure if I like the name of it specifically, but the behavior is quite desirable.
Naming things ¯_(ツ)_/¯
I was thinking:
%with_pycache - collides with bconds %include_pycache - quite long %pyc - quite indecipherable %pycache - might be mistaken to only include the pyc files
%pycached doesn't sound that bad to me - include that file, but also its cached Python bytecode. However I am not a native speaker and I realize I often desecrate the English language. Got any better suggestions?
%pyfile is probably quite enough for that. It looks and feels like a marker instead of something strange, and the underlying behavior is somewhat immaterial.
In the next year, we could look at leveraging the INSTALLED_FILES stuff that setuptools, et al. produce and enhance it with %pycached (or whatever) so that we grab not just the files it copied, but all the byte-compiled files too. The mechanism in which to do so will be somewhat interesting, since making that file get produced during python builds is no longer quite so simple with the advent of PEP 517/518 based projects... But it's something to look forward to.
See also this e-mail (the second half):
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
Yeah, it's going to get very ugly to do this reasonably well.
On Mon, Dec 16, 2019 at 1:18 PM Neal Gompa ngompa13@gmail.com wrote:
On Mon, Dec 16, 2019 at 1:00 PM Miro Hrončok mhroncok@redhat.com wrote:
On 16. 12. 19 16:56, Neal Gompa wrote:
For what it's worth. I think %pycached is a nice improvement. I'm not sure if I like the name of it specifically, but the behavior is quite desirable.
Naming things ¯_(ツ)_/¯
I was thinking:
%with_pycache - collides with bconds %include_pycache - quite long %pyc - quite indecipherable %pycache - might be mistaken to only include the pyc files
%pycached doesn't sound that bad to me - include that file, but also its cached Python bytecode. However I am not a native speaker and I realize I often desecrate the English language. Got any better suggestions?
%pyfile is probably quite enough for that. It looks and feels like a marker instead of something strange, and the underlying behavior is somewhat immaterial.
I support this incremental improvement.
I agree with Neal here. %pyfile is probably the best/simplest option. It will read best too, as it basically says "Deploy the functionality of this python file in the appropriate manner". The implementation can then be adjusted as needed.
On 07. 01. 20 19:23, Stephen Gallagher wrote:
I agree with Neal here. %pyfile is probably the best/simplest option. It will read best too, as it basically says "Deploy the functionality of this python file in the appropriate manner". The implementation can then be adjusted as needed.
Sorry to hear that now. We have decided (with Petr Viktorin) that we like %pycached more and that %pyfile is too mysterious - and that in either way it is bikeshedding.
%pycached has been merged and deployed, the ship has sailed.
See https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/37#comment...
On 2020-01-07 19:23, Stephen Gallagher wrote:
On Mon, Dec 16, 2019 at 1:18 PM Neal Gompa ngompa13@gmail.com wrote:
On Mon, Dec 16, 2019 at 1:00 PM Miro Hrončok mhroncok@redhat.com wrote:
On 16. 12. 19 16:56, Neal Gompa wrote:
For what it's worth. I think %pycached is a nice improvement. I'm not sure if I like the name of it specifically, but the behavior is quite desirable.
Naming things ¯_(ツ)_/¯
I was thinking:
%with_pycache - collides with bconds %include_pycache - quite long %pyc - quite indecipherable %pycache - might be mistaken to only include the pyc files
%pycached doesn't sound that bad to me - include that file, but also its cached Python bytecode. However I am not a native speaker and I realize I often desecrate the English language. Got any better suggestions?
%pyfile is probably quite enough for that. It looks and feels like a marker instead of something strange, and the underlying behavior is somewhat immaterial.
I support this incremental improvement.
I agree with Neal here. %pyfile is probably the best/simplest option. It will read best too, as it basically says "Deploy the functionality of this python file in the appropriate manner". The implementation can then be adjusted as needed.
While "Deploy the functionality of this python file in the appropriate manner" would be great, it would need to guess too much. The filename doesn't have enough info. %pycached's very limited scope is a feature. It should be a reliable tool that you can use manually, or later as part of a more complex system.
As for the name, I hope y'all don't mind that we skipped the bikeshedding and went with %pycached after considering the alternatives privately.
On Wed, Jan 8, 2020 at 5:12 AM Petr Viktorin pviktori@redhat.com wrote:
While "Deploy the functionality of this python file in the appropriate manner" would be great, it would need to guess too much. The filename doesn't have enough info. %pycached's very limited scope is a feature. It should be a reliable tool that you can use manually, or later as part of a more complex system.
As for the name, I hope y'all don't mind that we skipped the bikeshedding and went with %pycached after considering the alternatives privately.
Not a problem. I didn't note the date to the email I was replying to. (I am still processing through a ridiculous amount of email from my vacation). I'm good with how things turned out.
On 2019-12-16 13:03, Vít Ondruch wrote: [...]
My proposal is high level idea. It is easy to shot it down based on "how would one split several Python modules across several subpackages?". Of course next step would be to output to wile lists, one for py files while the other for pyc files. If you are talking about other possilbe subpackages, it could be possible to use some parameter, or block to create appropriately named file list.
Yes. Ultimately, your proposal is quite complex and will require lots of thought to implement properly. IMO it is a good general idea, but quite out of scope here.
Le 2019-12-16 12:46, Petr Viktorin a écrit :
Also, "%pyfile" would introduce extra issues. For example, how would one split several Python modules across several subpackages?
Just look at how %gofiles and %fontfiles do it https://pagure.io/fonts-rpm-macros/blob/master/f/templates/rpm/spectemplate-... https://pagure.io/go-rpm-macros/blob/master/f/templates/rpm/spectemplate-go-...
I would be the last to call the supporting ma
Le 2019-12-16 12:46, Petr Viktorin a écrit :
Also, "%pyfile" would introduce extra issues. For example, how would one split several Python modules across several subpackages?
Just look at how %gofiles and %fontfiles do it
https://pagure.io/fonts-rpm-macros/blob/master/f/templates/rpm/spectemplate-... https://pagure.io/go-rpm-macros/blob/master/f/templates/rpm/spectemplate-go-...
I would be the last to call the supporting macro code in redhat-rpm-config pretty but it works and is available for reuse by other fedora packages.
It enables reusing the same macro set for as many upstream modules and subpackages as one may need to (and it also allows the mixing of those subpackages when an upstream project provides several kinds of modules, as is common for language binding projects)
Reagrds,
On Monday, 16 December 2019 at 10:19, Vít Ondruch wrote:
I though about this a bit and to be more constructive, I think that you should probably move the file list creation out of %files section. Do it probably in %install section instead. That way, you would have more flexibility. You could have something like "%pyfile %{python3_sitelib}/foo.py" and it would do on background something like:
echo %{python3_sitelib}/foo.py > pyfiles.lst echo %{python3_sitelib}/foo.pyc > pycache.lst
Later you would just submit these file lists to the appropriate packages. This would be IMO more flexible.
+1. %find_lang macro does this and is very flexible.
Regards, Dominik
packaging@lists.fedoraproject.org