Hey all,
Here's my first draft of a SourceURL guideline. This tries to encapsulate current practices but a few new things had to be added related to SRPMs where no upstream source exists. This draft will probably need some touching up as I whipped it up pretty quickly but hopefully it captures the spirit of what we're trying to achieve.
The latest version is available at: http://www.fedoraproject.org/wiki/PackagingDrafts/SourceUrl
''' = Referencing Source =
One of the design goals of rpm is to cleanly separate upstream source from vendor modifications. For the Fedora packager, this means that sources used to build a package should be the vanilla sources available from upstream. To help reviewers and QA scripts verify this, the packager needs to indicate where a reviewer can find the source that was used to make the rpm.
The most common case is where upstream distributes source as a tar.gz, tar.bz2 or zip archive that we can download from an upstream website. In these cases you must use a full URL to the package in the SourceX: line. For example::
{{{ Source0: http://download.sourceforge.net/%%7Bname%7D/%%7Bname%7D-%%7Bversion%7D.tar.g...
Source0: http://ftp.gnome.org/pub/GNOME/sources/gnome-common/2.12/gnome-common-2.12.0... }}}
There are several cases where upstream is not providing the source to you in an upstream tarball. In these cases you must document how to generate the tarball used in the rpm either through a spec file comment or a script included as a separate SourceX:.
Here are some specific examples:
== Using Revision Control ==
In some cases you may want to pull sources from upstream's revision control system because there have been many changes since the last release and you think that a tarball that you generate from there will more accurately show how the package relates to upstream's development. Here's how you can use a comment to show where the source came from::
{{{ # The source for this package was pulled from upstream's cvs. Use the # following commands to generate the tarball: # svn export -r 250 http://www.foo.org/svn/foo/trunk foo-20070221 # tar -czvf foo-20070221.tar.gz foo-20070221 Source0: foo-20070221.tar.gz }}}
When pulling from revision control, please remember to use a Name-version-release compatible with the release guidelines. In particular, check the section on Naming Snapshots.
== When Upstream uses Prohibited Code ==
Some upstream packages include patents or trademarks that we are not allowed to ship even as source code. In these cases you have to modify the source tarball to remove this code before you even upload it to the build system. Here's an example of using a script to document how you went from the upstream tarball to the one included in the package:
From the spec: {{{ Source0: libfoo-1.0-nopatents.tar.gz # libfoo contains patented code that we cannot ship. Therefore we use # this script to remove the patented code before shipping it. # Download the upstream tarball and invoke this script while in the # tarball's directory: # ./generate-tarball.sh 1.0 Source1: generate-tarball.sh }}}
generate-tarball.sh: {{{ #!/bin/sh
VERSION=$1
tar -xzvf libfoo-$VERSION.tar.gz rm libfoo-$VERSION/src/patentedcodec.c sed -i -e 's/patentedcodec.c//' libfoo-$VERSION/src/Makefile
tar -czvf libfoo-$VERSION-nopatents.tar.gz }}}
== We are Upstream ==
For some packages where we are the upstream authors, for instance, the system-config-* tools, the source rpm that we distribute is the canonical source of the files. There is no public revision control system or publically released tarball for these programs so there is no tarball to list. Add a comment like the following to the spec:
{{{ # This is a Red Hat maintained package which is specific to # our distribution. Thus the source is only available from # within this srpm. Source0: system-config-foo-1.0.tar.gz }}} '''
-Toshio
On Wed, Feb 14, 2007 at 01:45:25PM -0800, Toshio Kuratomi wrote:
Hey all,
Here's my first draft of a SourceURL guideline. This tries to encapsulate current practices but a few new things had to be added related to SRPMs where no upstream source exists. This draft will probably need some touching up as I whipped it up pretty quickly but hopefully it captures the spirit of what we're trying to achieve.
Looks OK. But since we're commenting on source origin could somewhere a kind request ("SHOULD") to (srpm-)package upstream sources/patches with original timestamps where possible be embedded?
On Wed, 2007-02-14 at 22:57 +0100, Axel Thimm wrote:
On Wed, Feb 14, 2007 at 01:45:25PM -0800, Toshio Kuratomi wrote:
Hey all,
Here's my first draft of a SourceURL guideline. This tries to encapsulate current practices but a few new things had to be added related to SRPMs where no upstream source exists. This draft will probably need some touching up as I whipped it up pretty quickly but hopefully it captures the spirit of what we're trying to achieve.
Looks OK. But since we're commenting on source origin could somewhere a kind request ("SHOULD") to (srpm-)package upstream sources/patches with original timestamps where possible be embedded?
That sounds like a good best practice. It sounds like a separate item the way things are currently phrased. Do you have some wording to fit it in, or do you just want to throw it in a separate recommendation.
-Toshio
Toshio Kuratomi wrote:
On Wed, 2007-02-14 at 22:57 +0100, Axel Thimm wrote:
Looks OK. But since we're commenting on source origin could somewhere a kind request ("SHOULD") to (srpm-)package upstream sources/patches with original timestamps where possible be embedded?
That sounds like a good best practice. It sounds like a separate item the way things are currently phrased. Do you have some wording to fit it in, or do you just want to throw it in a separate recommendation.
Is this section from the packaging guidelines what you're after?
http://fedoraproject.org/wiki/Packaging/Guidelines#head-0239576e441f9ef53d17...
"When downloading sources, patches etc, consider using a client that preserves the upstream timestamps. For example wget -N or curl -R. To make the change global for wget, add this to your ~/.wgetrc: timestamping = on, and for curl, add to your ~/.curlrc: -R."
-- Todd OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp ====================================================================== To have a successful relationship, I must learn to make it look like I'm giving as much as I'm getting.
On Wed, Feb 14, 2007 at 06:30:43PM -0500, Todd Zullinger wrote:
Toshio Kuratomi wrote:
On Wed, 2007-02-14 at 22:57 +0100, Axel Thimm wrote:
Looks OK. But since we're commenting on source origin could somewhere a kind request ("SHOULD") to (srpm-)package upstream sources/patches with original timestamps where possible be embedded?
That sounds like a good best practice. It sounds like a separate item the way things are currently phrased. Do you have some wording to fit it in, or do you just want to throw it in a separate recommendation.
Is this section from the packaging guidelines what you're after?
http://fedoraproject.org/wiki/Packaging/Guidelines#head-0239576e441f9ef53d17...
"When downloading sources, patches etc, consider using a client that preserves the upstream timestamps. For example wget -N or curl -R. To make the change global for wget, add this to your ~/.wgetrc: timestamping = on, and for curl, add to your ~/.curlrc: -R."
Indeed, thanks for pointing this out, Todd!
Toshio, forget my request then, unless you want to consolidate or rearange parts of the guidelines talking about handling upstream sources/patches.
On Wed, Feb 14, 2007 at 02:40:06PM -0800, Toshio Kuratomi wrote:
On Wed, 2007-02-14 at 22:57 +0100, Axel Thimm wrote:
On Wed, Feb 14, 2007 at 01:45:25PM -0800, Toshio Kuratomi wrote:
Hey all,
Here's my first draft of a SourceURL guideline. This tries to encapsulate current practices but a few new things had to be added related to SRPMs where no upstream source exists. This draft will probably need some touching up as I whipped it up pretty quickly but hopefully it captures the spirit of what we're trying to achieve.
Looks OK. But since we're commenting on source origin could somewhere a kind request ("SHOULD") to (srpm-)package upstream sources/patches with original timestamps where possible be embedded?
That sounds like a good best practice. It sounds like a separate item the way things are currently phrased. Do you have some wording to fit it in, or do you just want to throw it in a separate recommendation.
since it's a weak suggestion and only a subsentence it would be nice to interweave in the part that discusses unmodified upstream sources. I agree the topic "URL" is not exactly describing timestamps :)
Maybe the general topic could be abstracted to "Dealing with sources and patches" or similar, so it wouldn't be completely out of the water.
As a phrasing I would suggest "Whenever possible try to maintain timestamps of sources or patches".
On Wed, Feb 14, 2007 at 01:45:25PM -0800, Toshio Kuratomi wrote:
Hey all,
Here's my first draft of a SourceURL guideline. This tries to
Overall it looks good to me. I see 2 other situations that may be worth mentionning:
1. the tarball was found on a regular site, kept in a srpm, or a disk drive, a mirror, the lookaside cache or whatever place, and afterwards the initial download source disappeared. It is similar than the "we are upstream" case but not exactly the same. The fedora maintainer should maintain the package in that case, that is fix bugs and so on but without being upstream, only in the fedora cvs (with patches). These packages may be on the road to retirement, but not necessarily, especially when these are old, stable, and little packages.
2. sometimes upstream don't have a properly versioned tarball. In that case it should be asked to upstream to version properly, but it is not always possible. I think that the rule for those cases should be to provide the full url in a comment, and add a version to the tarball based, for example on the timestamp of the file (but not necessarily). For example:
# Source1 is only used for documentation # renamed to tex4ht-all-YYYYMMDD.zip - based on last timestamp in # directory Source1: tex4ht-all-20050228.zip # unversioned upstream source, downloaded with wget -N #Source1 http://www.cse.ohio-state.edu/~gurari/TeX4ht/tex4ht-all.zip
-- Pat
On Wed, 2007-02-14 at 13:45 -0800, Toshio Kuratomi wrote:
''' = Referencing Source =
One of the design goals of rpm is to cleanly separate upstream source from vendor modifications. For the Fedora packager, this means that sources used to build a package should be the vanilla sources available from upstream. To help reviewers and QA scripts verify this, the packager needs to indicate where a reviewer can find the source that was used to make the rpm.
caillon had this to say in the bug which spawned this: ''' Looks good from the brief glance I took, but I strongly feel this whole thing should be a "good practises" recommendation and not a requirement. If you're trying to prevent against "bad" RPMs, well you're not going to do that if there are exceptions... Even for a good SRPM, someone could simply fork an open source project, not have a repo other than the SRPM, and distribute whatever code they want that way in extras, theoretically. This has no bearing on the actual packaging or quality of RPMs. It's only redeeming quality is that it might potentially help with automated verification of upstream sources, but that does not exist right now and that potential benefit should be enough to convince most packagers to do this. There's simply no reason to make it a hard requirement IMO other than because it's always been that way (which is no real reason). '''
Toshio Kuratomi wrote:
On Wed, 2007-02-14 at 13:45 -0800, Toshio Kuratomi wrote:
''' = Referencing Source =
One of the design goals of rpm is to cleanly separate upstream source from vendor modifications. For the Fedora packager, this means that sources used to build a package should be the vanilla sources available from upstream. To help reviewers and QA scripts verify this, the packager needs to indicate where a reviewer can find the source that was used to make the rpm.
caillon had this to say in the bug which spawned this: ''' Looks good from the brief glance I took, but I strongly feel this whole thing should be a "good practises" recommendation and not a requirement.
Or, I don't know, something like "Packaging Guidelines"? (:
I like your draft, good work.
-- Rex
On Wednesday 14 February 2007, Toshio Kuratomi wrote:
Source0: http://download.sourceforge.net/%%7Bname%7D/%%7Bname%7D-%%7Bversion%7D.tar.g...
Tiny detail: s/download./downloads./ - the former has been shaky for quite a while now, but I've never seen the latter fail.
On Mittwoch 14 Februar 2007, Ville Skyttä wrote:
On Wednesday 14 February 2007, Toshio Kuratomi wrote:
Source0: http://download.sourceforge.net/%%7Bname%7D/%%7Bname%7D-%%7Bversion%7D.tar.g...
Tiny detail: s/download./downloads./ - the former has been shaky for quite a while now, but I've never seen the latter fail.
I just remembered, that in another thread I read prdownloads.sourceforge.net as the recommended sourceforge download source. But I do not know, which works better or if there are any differences.
Regards, Till
On Thursday 15 February 2007, Till Maas wrote:
On Mittwoch 14 Februar 2007, Ville Skyttä wrote:
On Wednesday 14 February 2007, Toshio Kuratomi wrote:
Source0: http://download.sourceforge.net/%%7Bname%7D/%%7Bname%7D-%%7Bversion%7D.tar.g...
Tiny detail: s/download./downloads./ - the former has been shaky for quite a while now, but I've never seen the latter fail.
I just remembered, that in another thread I read prdownloads.sourceforge.net as the recommended sourceforge download source. But I do not know, which works better or if there are any differences.
prdownloads tarball URLs aren't suitable, they're not directly downloadable.
On Donnerstag 15 Februar 2007, Ville Skyttä wrote:
prdownloads tarball URLs aren't suitable, they're not directly downloadable.
When I try to download http://prdownloads.sourceforge.net/optipng/optipng-0.5.4.tar.gz, it works, but it is a redirection to http://downloads.sourceforge.net/optipng/optipng-0.5.4.tar.gz, so I guess downloads is the better choice.
Regards, Till
On Mittwoch 14 Februar 2007, Toshio Kuratomi wrote:
The latest version is available at: http://www.fedoraproject.org/wiki/PackagingDrafts/SourceUrl
I made some minor corrections, e.g. using example.com instead of foo.org and links to the mentioned NamingGuidelines sections. I also added a section about the correct sourceforge.net URL, afaik it is not documented anywhere else, and a section why using %{version} in SourceX: is good. I hope this is ok. :-)
I noticed, that in the section titled "When Upstream uses Prohibited Code" there is not mentioned, that the URL of the original tarball must be included in the generate-tarball.sh script, or, that it also must download it, e.g.
curl http://www.example.com/files/libfoo-$VERSION.tar.gz | xzvf libfoo-$VERSION.tar.gz
Also, one could argue, that the script must always be named "generate-tarball.sh" (or whatever name is appropriate) because then QA-scripts could automatically use them.
Regards, Till
Toshio Kuratomi (a.badger@gmail.com) said:
{{{ # The source for this package was pulled from upstream's cvs. Use the # following commands to generate the tarball: # svn export -r 250 http://www.foo.org/svn/foo/trunk foo-20070221 # tar -czvf foo-20070221.tar.gz foo-20070221 Source0: foo-20070221.tar.gz }}}
Might want to match your SCMs in the example. :)
As for the whole thing, I'd argue that they should be suggests-guidelines, not required-guidelines.
Bill
On Thu, 2007-02-15 at 11:48 -0500, Bill Nottingham wrote:
Toshio Kuratomi (a.badger@gmail.com) said:
{{{ # The source for this package was pulled from upstream's cvs. Use the # following commands to generate the tarball: # svn export -r 250 http://www.foo.org/svn/foo/trunk foo-20070221 # tar -czvf foo-20070221.tar.gz foo-20070221 Source0: foo-20070221.tar.gz }}}
Might want to match your SCMs in the example. :)
As for the whole thing, I'd argue that they should be suggests-guidelines, not required-guidelines.
Why? Is there any reason why we don't want this information in the package?
~spot
Tom 'spot' Callaway (tcallawa@redhat.com) said:
As for the whole thing, I'd argue that they should be suggests-guidelines, not required-guidelines.
Why? Is there any reason why we don't want this information in the package?
I dunno - it just feels like something that should be 'please do this', not 'package rejected because you didn't do this.'
Bill
Bill Nottingham wrote:
Tom 'spot' Callaway (tcallawa@redhat.com) said:
As for the whole thing, I'd argue that they should be suggests-guidelines, not required-guidelines.
Why? Is there any reason why we don't want this information in the package?
I dunno - it just feels like something that should be 'please do this', not 'package rejected because you didn't do this.'
If folks want to be able to reliably verify upstream sources, this kind of stuff is pretty much required. I can't think of very many legit cases otherwise.
-- Rex
On Thu, 2007-02-15 at 12:48 -0500, Bill Nottingham wrote:
Tom 'spot' Callaway (tcallawa@redhat.com) said:
As for the whole thing, I'd argue that they should be suggests-guidelines, not required-guidelines.
Why? Is there any reason why we don't want this information in the package?
I dunno - it just feels like something that should be 'please do this', not 'package rejected because you didn't do this.'
Sorry. If you don't eat your meat, you cannot have any pudding.
HOW CAN YOU HAVE ANY PUDDING IF YOU DON'T EAT YOUR MEAT?!?
~spot
Tom 'spot' Callaway (tcallawa@redhat.com) said:
Sorry. If you don't eat your meat, you cannot have any pudding.
HOW CAN YOU HAVE ANY PUDDING IF YOU DON'T EAT YOUR MEAT?!?
OK, just a little pinprick...
More seriously, if we're trying to encode this sort of metadata, it would be nice to have a real format/mechanism, rather than 'a defined comment syntax.'
Bill
On Thu, 2007-02-15 at 12:56 -0500, Bill Nottingham wrote:
Tom 'spot' Callaway (tcallawa@redhat.com) said:
Sorry. If you don't eat your meat, you cannot have any pudding.
HOW CAN YOU HAVE ANY PUDDING IF YOU DON'T EAT YOUR MEAT?!?
OK, just a little pinprick...
More seriously, if we're trying to encode this sort of metadata, it would be nice to have a real format/mechanism, rather than 'a defined comment syntax.'
OK. Suggestions on format welcome. :)
~spot
On Thu, Feb 15, 2007 at 07:11:45PM +0100, Till Maas wrote:
On Donnerstag 15 Februar 2007, Tom 'spot' Callaway wrote:
OK. Suggestions on format welcome. :)
I suggest a shell script with the name "generate-tarball.sh" that generates the desired tarball and is included as SourceX:.
There should be the package name prepended. Something like foo-generate-tarball.sh
-- Pat
packaging@lists.fedoraproject.org
-
Axel Thimm -
Bill Nottingham -
Patrice Dumas -
Rex Dieter -
Till Maas -
Todd Zullinger -
Tom Callaway -
Toshio Kuratomi -
Ville Skyttä