Greetings,
I hope everyone is having a good day. Here in the Fedora ARM project, we're having a great time overall, but we've run up against a familiar foe in our preparation for Fedora 17 and rawhide: superfluous deps in SPEC files. Today's example is a shell-escaped call to Ruby just to determine some gcc flags for an unrelated package build, but that's just today's example. There are many others, so some standard would help.
Why is this a problem? The most obvious issue is the superfluous dependencies, pulling in ruby (and all of its deps), for example. But the bigger problem for the distribution as a whole is that we are becoming less bootstrappable[0] over time, and we are suffering from dependency creep that makes life difficult for secondary architectures (or even for the primary architecture if a bunch of rebuilds were to fail and there were no deps to satisfy a circular path).
Eventually, the issue of dependency creep is going to bite the broader distribution. In the meantime, can we please impose some kind of rule surrounding which superfluous build deps are allowed? I'm not advocating for the removal of Ruby, Lua, and so on and a return to old fashioned shell, but I am suggesting that we cap where we are and try to avoid introducing any more bizzare build dependencies in the future. For example, say a package grew a SPEC file need for a call to an INTERCAL script just to determine gcc flags? It's not really all that different from calling Ruby to expand gcc flags and maybe I happen to like using INTERCAL over Ruby? Or why not have a hardware compiled state machine implement some custom logic to determine gcc flags? Maybe I find that easier than writing some good old fashioned shell, etc.
In all seriousness, can we please have some direction? If this is already implied or codified, could someone remind packagers to refrain from adding build dependencies like this? I'd recommend that, in general, if a one line shell escape can be used, that ought to be the preference over adding a one-line build dep on $random scripting lang.
Thanks!
Jon.
[0] Rebuilding the entire distribution from first principles, using nothing more than a toolchain to begin the process. We did this for Fedora 15 on ARMv7. I eventually would like it that Fedora be able to bootstrap itself on any architecture trivially, and I know Debian (and other distributions) are similarly keen to return to that world.
Jon Masters (jcm@redhat.com) said:
I hope everyone is having a good day. Here in the Fedora ARM project, we're having a great time overall, but we've run up against a familiar foe in our preparation for Fedora 17 and rawhide: superfluous deps in SPEC files. Today's example is a shell-escaped call to Ruby just to determine some gcc flags for an unrelated package build, but that's just today's example. There are many others, so some standard would help.
OK, so how about:
... Scripting inside of spec files
Inside of a spec file, sometimes it is necessary to call a programming or scripting language during %setup, %build, or %install. In Fedora, spec files may in general only use the following languages for this purpose:
1. Python 2. Perl 3. awk/sed
Also, if your package already BuildRequires a specific scripting language (such as Ruby, or Tcl) as part of its normal compile process, it may also be called from the spec file.
...
Bill
Bill Nottingham notting@redhat.com writes:
OK, so how about:
Scripting inside of spec files
Inside of a spec file, sometimes it is necessary to call a programming or scripting language during %setup, %build, or %install. In Fedora, spec files may in general only use the following languages for this purpose:
- Python
- Perl
- awk/sed
Also, if your package already BuildRequires a specific scripting language (such as Ruby, or Tcl) as part of its normal compile process, it may also be called from the spec file.
Couple of questions:
1. Which of the above alternatives demand a BuildRequires for the language? (Is BuildRequires the right thing for calls that aren't in %build?)
2. Can we put even stricter limits on scripting language calls from macro definitions? As an example, the specfile for python-psycopg2 contains
%{expand: %%define py3ver %(python3 -c 'import sys;print(sys.version[0:3])')}
which I find to be a perpetual annoyance because I don't have python3 installed on my system, and so *any* tool that parses the specfile burps out an error message, for example pretty nearly any fedpkg command complains about that. In the particular case, %py3ver isn't used anywhere that's critical except in an actual build, so things work, but it's unsightly as can be. I have seen specfiles in other people's packages where specfile parsing fails entirely unless you have $randompackage installed, and that seems bad to me.
Now, just instituting a policy edict against this isn't going to fix it, and if I knew how to get rid of the python-psycopg2 usage I would have done so already. But if we're going to have anything about scripting languages in the policy then I'd like to have some guidance about cases like this.
regards, tom lane
On Fri, Feb 17, 2012 at 10:10:48AM -0500, Tom Lane wrote:
Bill Nottingham notting@redhat.com writes:
OK, so how about:
Scripting inside of spec files
Inside of a spec file, sometimes it is necessary to call a programming or scripting language during %setup, %build, or %install. In Fedora, spec files may in general only use the following languages for this purpose:
- Python
- Perl
- awk/sed
Also, if your package already BuildRequires a specific scripting language (such as Ruby, or Tcl) as part of its normal compile process, it may also be called from the spec file.
Couple of questions:
- Which of the above alternatives demand a BuildRequires for the
language? (Is BuildRequires the right thing for calls that aren't in %build?)
Technically, only python. For good practice I would say both perl and python. I would never say awk/sed.
Which reminds me... maybe we need to specify gawk rather than awk... if someone wrote something that worked only with mawk we wouldn't like that either.
BuildRequires is correct for %setup, %build, %install, and %check... the sections that are run when building a package. %pre/%post/etc would use a form of Requires([pre,post,etc]): instead as they are run on the end user's system at package install time.
- Can we put even stricter limits on scripting language calls from
macro definitions? As an example, the specfile for python-psycopg2 contains
%{expand: %%define py3ver %(python3 -c 'import sys;print(sys.version[0:3])')}
which I find to be a perpetual annoyance because I don't have python3 installed on my system, and so *any* tool that parses the specfile burps out an error message, for example pretty nearly any fedpkg command
The very general approach of using a scripting language that the package (or a subpackage) is building for seems valid. So I don't think that this should be restricted here.
That macro could be better, though. For instance, if you use this: %{!?py3ver: %global py3ver %(%{__python3} -c 'import sys; print(sys.version[0:3])')}
You won't get the errors that are coming from that %{expand} line.
However, that won't prevent you from having other errors/warnings like this: sh: /usr/bin/python3: No such file or directory
Without python3 installed, macros in the spec file can't be expanded correctly (because their definitions depend on python3). The spec file is BuildRequireing python3 so it shouldn't be expected that you can operate on the spec file without python3 installed.
-Toshio
On Fri, 17 Feb 2012 08:43:19 -0800 Toshio Kuratomi a.badger@gmail.com wrote:
On Fri, Feb 17, 2012 at 10:10:48AM -0500, Tom Lane wrote:
Bill Nottingham notting@redhat.com writes:
OK, so how about:
Scripting inside of spec files
Inside of a spec file, sometimes it is necessary to call a programming or scripting language during %setup, %build, or %install. In Fedora, spec files may in general only use the following languages for this purpose:
- Python
- Perl
- awk/sed
Also, if your package already BuildRequires a specific scripting language (such as Ruby, or Tcl) as part of its normal compile process, it may also be called from the spec file.
Couple of questions:
- Which of the above alternatives demand a BuildRequires for the
language? (Is BuildRequires the right thing for calls that aren't in %build?)
Technically, only python. For good practice I would say both perl and python. I would never say awk/sed.
Which reminds me... maybe we need to specify gawk rather than awk... if someone wrote something that worked only with mawk we wouldn't like that either.
BuildRequires is correct for %setup, %build, %install, and %check... the sections that are run when building a package. %pre/%post/etc would use a form of Requires([pre,post,etc]): instead as they are run on the end user's system at package install time.
- Can we put even stricter limits on scripting language calls from
macro definitions? As an example, the specfile for python-psycopg2 contains
%{expand: %%define py3ver %(python3 -c 'import sys;print(sys.version[0:3])')}
which I find to be a perpetual annoyance because I don't have python3 installed on my system, and so *any* tool that parses the specfile burps out an error message, for example pretty nearly any fedpkg command
The very general approach of using a scripting language that the package (or a subpackage) is building for seems valid. So I don't think that this should be restricted here.
That macro could be better, though. For instance, if you use this: %{!?py3ver: %global py3ver %(%{__python3} -c 'import sys; print(sys.version[0:3])')}
You won't get the errors that are coming from that %{expand} line.
However, that won't prevent you from having other errors/warnings like this: sh: /usr/bin/python3: No such file or directory
You could get rid of those by using:
%{!?py3ver: %global py3ver %(%{?__python3} -c 'import sys; print(sys.version[0:3])' 2>/dev/null)}
And sometimes you might even prefer:
%{!?py3ver: %global py3ver %(%{?__python3} -c 'import sys; print(sys.version[0:3])' 2>/dev/null || echo 3.0)}
That would give a result that "looked right" in the absence of python3, giving other bits of the spec that depend on that definition a better chance of working as expected.
Without python3 installed, macros in the spec file can't be expanded correctly (because their definitions depend on python3). The spec file is BuildRequireing python3 so it shouldn't be expected that you can operate on the spec file without python3 installed.
I'd prefer to see specs a bit more robust so that for instance you could run "spectool" on them to download upstream sources and then do a mockbuild, which wouldn't require python3 or whatever to be installed on the build host.
Paul.
On Fri, Feb 17, 2012 at 06:23:25PM +0000, Paul Howarth wrote:
Toshio Kuratomi a.badger@gmail.com wrote:
However, that won't prevent you from having other errors/warnings like this: sh: /usr/bin/python3: No such file or directory
You could get rid of those by using:
%{!?py3ver: %global py3ver %(%{?__python3} -c 'import sys; print(sys.version[0:3])' 2>/dev/null)}
And sometimes you might even prefer:
%{!?py3ver: %global py3ver %(%{?__python3} -c 'import sys; print(sys.version[0:3])' 2>/dev/null || echo 3.0)}
That would give a result that "looked right" in the absence of python3, giving other bits of the spec that depend on that definition a better chance of working as expected.
Without python3 installed, macros in the spec file can't be expanded correctly (because their definitions depend on python3). The spec file is BuildRequireing python3 so it shouldn't be expected that you can operate on the spec file without python3 installed.
I'd prefer to see specs a bit more robust so that for instance you could run "spectool" on them to download upstream sources and then do a mockbuild, which wouldn't require python3 or whatever to be installed on the build host.
I tested this with spectool -g and python-psycopg2 and all variants of that py3ver line work.(without even a warning). So spectool isn't really an issue.
-Toshio
Toshio Kuratomi a.badger@gmail.com writes:
I tested this with spectool -g and python-psycopg2 and all variants of that py3ver line work.(without even a warning). So spectool isn't really an issue.
I don't know what spectool is, but I can demonstrate what I see every time I work with this package:
[tgl@rh3 ~]$ cd f-pkg/python-psycopg2/master/ [tgl@rh3 master]$ ls psycopg2-2.4.4.tar.gz python-psycopg2.spec sources [tgl@rh3 master]$ fedpkg srpm sh: python3: command not found error: Macro %py3ver has empty body sh: python3: command not found error: Macro %py3ver has empty body
sh: python3: command not found error: Macro %py3ver has empty body Wrote: /home/tgl/f-pkg/python-psycopg2/master/python-psycopg2-2.4.4-1.fc18.src.rpm
The generated SRPM is fine, and I can (and do) test it in mock, but those messages would certainly scare anybody not familiar with the package. I've seen similar behavior with other packages. I'd like this guideline to show how this sort of thing can be avoided.
regards, tom lane
On Fri, Feb 17, 2012 at 03:23:36PM -0500, Tom Lane wrote:
Toshio Kuratomi a.badger@gmail.com writes:
I tested this with spectool -g and python-psycopg2 and all variants of that py3ver line work.(without even a warning). So spectool isn't really an issue.
I don't know what spectool is, but I can demonstrate what I see every time I work with this package:
Paul's concern was about spectool -g python-psycopg2.spec doing the right thing.
I checked, and it does.
[tgl@rh3 ~]$ cd f-pkg/python-psycopg2/master/ [tgl@rh3 master]$ ls psycopg2-2.4.4.tar.gz python-psycopg2.spec sources [tgl@rh3 master]$ fedpkg srpm sh: python3: command not found error: Macro %py3ver has empty body sh: python3: command not found error: Macro %py3ver has empty body
sh: python3: command not found error: Macro %py3ver has empty body Wrote: /home/tgl/f-pkg/python-psycopg2/master/python-psycopg2-2.4.4-1.fc18.src.rpm
The generated SRPM is fine, and I can (and do) test it in mock, but those messages would certainly scare anybody not familiar with the package. I've seen similar behavior with other packages. I'd like this guideline to show how this sort of thing can be avoided.
I would keep the concerns separate. The original post was about not having extraneous deps simply because someone is more comfortable writing a routine check for information in $RANDOM_UNRELATED_LANGUAGE rather than shell.
This concern is about making things that you actually must BuildRequire in order to build the package not display warnings when the BuildRequires are not satisfied.
They're very different. And I'm not sure that I would vote the same for each of those.
-Toshio
Paul Howarth paul@city-fan.org writes:
On Fri, 17 Feb 2012 08:43:19 -0800 Toshio Kuratomi a.badger@gmail.com wrote:
On Fri, Feb 17, 2012 at 10:10:48AM -0500, Tom Lane wrote:
[ whinging about python-psycopg2 ]
Without python3 installed, macros in the spec file can't be expanded correctly (because their definitions depend on python3). The spec file is BuildRequireing python3 so it shouldn't be expected that you can operate on the spec file without python3 installed.
I'd prefer to see specs a bit more robust so that for instance you could run "spectool" on them to download upstream sources and then do a mockbuild, which wouldn't require python3 or whatever to be installed on the build host.
Precisely. There are *lots* of situations where we expect to be able to parse specfiles without necessarily having all their buildreqs installed (for the most obvious case: to find out what BRs are needed). I think one of the goals of this guideline should be to prevent creep in the set of packages that have to be present before tools like fedpkg will operate on a specfile without complaint.
regards, tom lane
On Fri, Feb 17, 2012 at 09:16:40AM -0500, Bill Nottingham wrote:
Jon Masters (jcm@redhat.com) said:
I hope everyone is having a good day. Here in the Fedora ARM project, we're having a great time overall, but we've run up against a familiar foe in our preparation for Fedora 17 and rawhide: superfluous deps in SPEC files. Today's example is a shell-escaped call to Ruby just to determine some gcc flags for an unrelated package build, but that's just today's example. There are many others, so some standard would help.
This isn't codified in current Guidelines that I know of. I do recall one thread on it from the very early days of Fedora (possibly before the FPC).
OK, so how about:
... Scripting inside of spec files
Inside of a spec file, sometimes it is necessary to call a programming or scripting language during %setup, %build, or %install. In Fedora, spec files may in general only use the following languages for this purpose:
- Python
- Perl
- awk/sed
Also, if your package already BuildRequires a specific scripting language (such as Ruby, or Tcl) as part of its normal compile process, it may also be called from the spec file.
That seems about right. Maybe be a little looser and allow using languages that are either BuildRequires or Requires already.
For instance, take a language that doesn't byte compile itself. If you had a package where upstream consists of a single file that you then have to cp to the right filesystem location to become a module for that language it wouldn't need the interpreter to compile. However, the best way to determine the filesystem location might be something like:: "foolanguage -e 'print libpath'"
It seems appropriate to use that in a spec file.
Also, #3 might be better if it was a little more vague::
"Standard programs used in shell programming, for instance awk and sed"
Another clarificaation might be to make the initial paragraph a little more specific:
"Sometimes it is necessary to write a short script (perhaps a one-liner) that's executed in %setup, %build, or %install to get some information about the build environment. In Fedora, spec files may in general only use the following languages for this purpose: [...]"
-Toshio
On 02/17/2012 09:16 AM, Bill Nottingham wrote:
Jon Masters (jcm@redhat.com) said:
I hope everyone is having a good day. Here in the Fedora ARM project, we're having a great time overall, but we've run up against a familiar foe in our preparation for Fedora 17 and rawhide: superfluous deps in SPEC files. Today's example is a shell-escaped call to Ruby just to determine some gcc flags for an unrelated package build, but that's just today's example. There are many others, so some standard would help.
OK, so how about:
... Scripting inside of spec files
Inside of a spec file, sometimes it is necessary to call a programming or scripting language during %setup, %build, or %install. In Fedora, spec files may in general only use the following languages for this purpose:
- Python
- Perl
- awk/sed
Also, if your package already BuildRequires a specific scripting language (such as Ruby, or Tcl) as part of its normal compile process, it may also be called from the spec file.
Following up, I like the suggestion you made and I wonder what the next step would be?
Jon.
Jon Masters (jonathan@jonmasters.org) said:
On 02/17/2012 09:16 AM, Bill Nottingham wrote:
Jon Masters (jcm@redhat.com) said:
I hope everyone is having a good day. Here in the Fedora ARM project, we're having a great time overall, but we've run up against a familiar foe in our preparation for Fedora 17 and rawhide: superfluous deps in SPEC files. Today's example is a shell-escaped call to Ruby just to determine some gcc flags for an unrelated package build, but that's just today's example. There are many others, so some standard would help.
OK, so how about:
... Scripting inside of spec files
Inside of a spec file, sometimes it is necessary to call a programming or scripting language during %setup, %build, or %install. In Fedora, spec files may in general only use the following languages for this purpose:
- Python
- Perl
- awk/sed
Also, if your package already BuildRequires a specific scripting language (such as Ruby, or Tcl) as part of its normal compile process, it may also be called from the spec file.
Following up, I like the suggestion you made and I wonder what the next step would be?
Probably needs further polishing with Toshio's comments and pushed into an actual FPC ticket. Hopefully will get to that this week.
Bill
Bill Nottingham (notting@redhat.com) said:
Following up, I like the suggestion you made and I wonder what the next step would be?
Probably needs further polishing with Toshio's comments and pushed into an actual FPC ticket. Hopefully will get to that this week.
Done: https://fedorahosted.org/fpc/ticket/150
Bill
packaging@lists.fedoraproject.org
-
Bill Nottingham -
Jon Masters -
Jon Masters -
Paul Howarth -
Tom Lane -
Toshio Kuratomi