On Dom, 2015-04-19 at 09:49 -0700, Dave Johansen wrote:

On Sat, Apr 18, 2015 at 2:38 PM, Matthew Miller mattdm@fedoraproject.org wrote: On Sat, Apr 18, 2015 at 01:28:33PM -0700, Dave Johansen wrote: > What's the right way to handle github URLs? Currently, I'm working on the > spec file for breathe and the URL is

    There's actually a section in the guidelines precisely on
    this:

    https://fedoraproject.org/wiki/Packaging:SourceURL?rd=Packaging/SourceURL#Github

I'm not all that familiar with the details of github, so do the downloads available in the "releases" section count as "If the upstream does create tarballs" or not? If it does, then I still have the same question about the URL. If not, then I'll just use the method described in the referenced section of the wiki.

I got this example of pngquant, tags [1] and releases [2]. When upstream make a tag it will create a tarball and upstream can mark it as a release or not, now seems github shows all tags as releases . We can see that upstream doesn't mark last tag as a release !.

BTW [3] Upstream Release Monitoring detects a new tag and try to build it , that was a new and cool feature !

Hope can help you in some way .

[1] https://github.com/pornel/pngquant/tags [2] https://github.com/pornel/pngquant/releases [3] https://bugzilla.redhat.com/show_bug.cgi?id=1213193

Thanks,

Dave

-- packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging

Hi,

On Sex, 2015-05-15 at 21:50 -0700, Dave Johansen wrote:

On Sun, Apr 19, 2015 at 9:49 AM, Dave Johansen davejohansen@gmail.com wrote: On Sat, Apr 18, 2015 at 2:38 PM, Matthew Miller mattdm@fedoraproject.org wrote: On Sat, Apr 18, 2015 at 01:28:33PM -0700, Dave Johansen wrote: > What's the right way to handle github URLs? Currently, I'm working on the > spec file for breathe and the URL is

            There's actually a section in the guidelines precisely
            on this:

            https://fedoraproject.org/wiki/Packaging:SourceURL?rd=Packaging/SourceURL#Github


    I'm not all that familiar with the details of github, so do
    the downloads available in the "releases" section count as "If
    the upstream does create tarballs" or not? If it does, then I
    still have the same question about the URL. If not, then I'll
    just use the method described in the referenced section of the
    wiki.

For the sake of documentation, I ended up using the URL from srcurl.net like Sergio pointed out.

I found the solution in git-extras package .

Name: git-extras Version: 2.2.0 Source0: https://github.com/visionmedia/%%7Bname%7D/archive/%%7Bversion%7D.tar.gz#/%%...

Also documented in https://fedoraproject.org/wiki/Packaging:SourceURL#Troublesome_URLs

And also succeeds in fedora-review !

Source checksums ---------------- https://github.com/visionmedia/git-extras/archive/2.2.0.tar.gz#/git-extras-2... : CHECKSUM(SHA256) this package : 6a933114d276761a078e653a961566d9517a6a4eaadacb62e655d39481548f63 CHECKSUM(SHA256) upstream package : 6a933114d276761a078e653a961566d9517a6a4eaadacb62e655d39481548f63

Best regards,

On Tue, Jun 16, 2015 at 3:16 PM, Sérgio Basto sergio@serjux.com wrote:

Also documented in https://fedoraproject.org/wiki/Packaging:SourceURL#Troublesome_URLs

...

Relating to Github: https://fedoraproject.org/wiki/Packaging:SourceURL#Github

One thing to remember is the use of the snapshot guidelines; you'll want to do that unless you are using a release tag commit.

Several ways to handle: one is to add:

%global checkout YYYYMMDDgit%shortcommit (where YYYY = year, MM = month, DD day you pulled the commit)

There is a bit of discretion left to the packager to the ultimate format of %{checkout} - but I did a quick search and found most folks were using the format outlined above, so I figured go with the crowd.

then Release n.%{checkout}%{?dist}

finally in %changelog, don't forget to add the %{checkout} information to your release version there or you'll get a mismatch.

Also, you need to manual state the information, i.e.: n.n.n.YYYYMMDDgitnnnnnnnn - you'll get an error if you try to actually use %{checkout} in %changelog.

I just went through all this, so it's fresh in my mind... ;-)

If anyone else has any recommendations on this, would appreciate reading them.

Thanks!

On Wed, Jun 17, 2015 at 10:22 AM, Sérgio Basto sergio@serjux.com wrote:

So document Github snapshot guidelines with Git Submodules, it would be awesome .

Yup... I just used that exact same approach for submodules. I started another thread earlier that is specific to submodules and I plan to document and submit all the various methods to FPC for their consideration.

Thanks for taking the time to post Sérgio!

On Sáb, 2015-04-18 at 23:11 +0100, Sérgio Basto wrote:

On Sáb, 2015-04-18 at 13:28 -0700, Dave Johansen wrote:

...

What's the right way to handle github URLs? Currently, I'm working on the spec file for breathe and the URL is https://github.com/michaeljones/breathe/archive/v4.0.0.tar.gz and the actual file that's downloaded is breathe-4.0.0.tar.gz. How do I make that play nice in the spec file and with rpmlint?

Thanks,

Dave

from https://lists.fedoraproject.org/pipermail/packaging/2014-September/010287.ht...

you got a webservice that redirects url correctly, for example on pngquant:

-Source0: https://github.com/pornel/%%7Bname%7D/archive/%%7Bname%7D-%%7Bversion%7D.tar...

+Source0: http://github.srcurl.net/pornel/%%7Bname%7D/%%7Bversion%7D/%%7Bname%7D-%%7Bv...

but just work for releases of github and not for tags of guthub , at least don't found a way to work on tags ... . So I use srcurl.net for review package and remove it after (because doesn't worked on tags )

I think, I got confused with tags and releases . Now it seems they work exactly in same way ... . It is hard to say, I don't know if github change since then, sorry about introducing confusion about tags .

Best regards ,

Nice example (thanks bochecha):

https://github.com/Cangjians/libcangjie/releases/tag/v1.3

"libcangjie..." URL are valid, as created by uptream

https://github.com/Cangjians/libcangjie/releases/download/v1.3/libcangjie-1....

"Source code" URL are NOT immutable, so not valid.

https://github.com/Cangjians/libcangjie/archive/v1.3.tar.gz

Remi.

On Tue, Jun 23, 2015 at 10:21:50AM -0500, Jason L Tibbitts III wrote:

...

...

...

...

...

"RC" == Remi Collet Fedora@FamilleCollet.com writes:

RC> AFAIK, this is NOT immutable (upstream can delete/update tag) so not RC> valid.

It's true that with raw git at least, upstream can delete or move the tag, but this is one of those operations you really should not do with git (because it screws everyone who might have pulled your repo) and for all I know github may disallow it.

It does not, you can easily remove a tag in github. All it takes is git tag -d followed by a git push -f

The incidence of github upstreams moving tags is probably less than regular upstreams modifying tarballs and keeping the same name. Our system already handles this kind of thing, and I don't see it is a problem.

In which case we're ending up with no way of knowing which version of 0.0.1 we built since upstream may tag it multiple times (and people do!).

P.Yves

On Tue, Jun 23, 2015 at 09:15:58AM -0700, Gerald B. Cox wrote:

On Tue, Jun 23, 2015 at 8:38 AM, Pierre-Yves Chibon pingou@pingoured.fr wrote:

 In which case we're ending up with no way of knowing which version of
 0.0.1 we
 built since upstream may tag it multiple times (and people do!).

People can do all kinds of things not only in Git, but other VCS platforms toA cause mischief.A Once you tag something, you shouldn't change it.A It justA isn't a good practice, especially if you're creating a package for distribution. A Nothing is preventing someone from changing an archive and using the same name. A If I was working on a package with upstream and found that they were abusing tags, I'd point out to them why they shouldn't be doing that.A I've found that when you explain something, most people get it. Regarding the packaging guidelines, I've put together a draft revision of the SourceURL document that I plan to submit to FPC. A You can review it here: https://fedoraproject.org/wiki/User:Gbcox/PackagingDrafts/SourceURL

I was going to reply that it's not because shouldn't do something that they don't do it, but reading your draft it seems we are agreeing on the fact that the commit hash should be the one used.

However, your draft seems to include two changes, a rephrase of the regarding the point discussed here (URLs) as well as a proposal for handling sub-modules. Might be wise to split these into two proposals.

Pierre

On Tue, Jun 23, 2015 at 9:23 AM, Pierre-Yves Chibon pingou@pingoured.fr wrote:

but reading your draft it seems we are agreeing on the fact that the commit hash should be the one used.

Yes... but... ;-)

My take is this. If upstream uses tags, we should use them. It's easier to follow. Commit is for when there is no tag, and when you use commit, you need to use the full 40 character hash.

I've updated the draft to clarify that. I've also included a link within the draft regarding Git Tags. There is a discussion within that link regarding re-tagging; which I think explains the situation. Re-using the same tag in Git just isn't normal procedure - and to say it is frowned upon is a gross understatement. They call it "insane". I can't imagine any responsible upstream doing it.

On Tue, Jun 23, 2015 at 10:52 PM, Pierre-Yves Chibon pingou@pingoured.fr wrote:

The guidelines are not meant for projects that already follow the standards but quite the opposite, to be able to handle projects that do not behave correctly (and sometime projects that won't follow the standards despite asking them).

Did you read: https://git-scm.com/docs/git-tag

Specifically the portion regarding re-use of Git Tags?

What about the project that decides to change the contents of their archive and re-uses the exact same URL when re-posting? How is that different? If it's the same thing, what are we doing about it? What is the impact to Fedora if we did happen to discover an application which re-used the same tag?

I think disqualifying the use of widely used feature because a handful of ignorant people may be abusing it, isn't appropriate.

If I encountered a project that was doing that (and I never have), I would point them to the Git link above and tell them stop it. It isn't acceptable and reflects rather poorly on them.

On Wed, Jun 24, 2015 at 12:32 AM, Pierre-Yves Chibon pingou@pingoured.fr wrote:

That is a valid point, but we can reverse the problem: We can't do anything about upstreams that re-generate multiple times the archive for the same release, but with the current guidelines we can do something about the upstreams that are moving tags around.

That sounds to me like a good reason to do so.

...

The only counter-argument I can think of right now is the frequency at which the abuse occurs on github. In my experience, it is much more frequent.

Thanks again for taking the time to reply!

That is a good point, and I agree. We should always strive to fix things if we can. My concern is the practical effect of implementing a guideline that says you must always use commit hash when packaging source from Git repositories; but if the Project takes that same archive and uploads it to some URL, that it is now somehow magically golden. That just strikes me as a bit capricious.

I was a bit surprised at first with the frequency you believe this is happening with Git; but I can believe there are many people playing around, experimenting and learning about Git; and these people can make mistakes mainly because they haven't RTFM; but I don't think we should be concerned about those particular repositories.

The ones we are concerned about are the ones that we would want to package; and I really find it hard to imagine that someone would advance so far in developing something we would want to package, yet has a remained completely ignorant on how to properly use their VCS.

Sorry about the epistle ;-) but to sum up:

Yes, I agree with your point we should fix things if we can; but I don't believe mandating commit hash in all circumstances is the way to do it.

Perhaps the better approach (and I plan modify my draft to include this) would be to add a short explanation cautioning the packager to be aware that there is a chance of abuse when it comes to Git Tags. If the packager believes there is abuse, they MUST request upstream to resolve the situation. If upstream refuses, they can either drop their effort to package the application OR use the commit hash method.

If they use the commit hash method, they MUST put a comment in the Spec file documenting the fact that upstream is in violation of Git practices and commit hash was used to circumvent the situation. This identifies the bad apples.

Perhaps it's just easier to put a blanket ban on using Git Tags, but I think that approach is a bit ham-handed. This is a bit more nuanced, addresses the concern and goes after the root cause, which is an uneducated upstream.

Dne 25.6.2015 v 07:05 Remi Collet napsal(a):

Le 24/06/2015 20:02, Gerald B. Cox a écrit :

...

but I don't believe mandating commit hash in all circumstances is the way to do it.

I think current Guideline is "clear" and doesn't need to be changed.

Please explain how you can check the sources used to build a package is the correct one ?

When upstream provides a tarball (usually because they run "make dist" to provide a usable archive), if they regenerate this tarball and reupload it, the checksum will change.

So now you have new checksum, but in dist-git, there is probably already uploaded tarball of the same name with different checksum and now you don't know what happened.

Also, not git expert, but I believe that if I force the Git repository, the hash might be completely missing next time. Not sure what the hash recorded in .spec file will help you.

So as for me, I am using and supporting the approach Gerald is proposing, because I believe it works in 99,9% of cases and it is intuitive and simple, which I cannot say about the current guidelines.

Vít

With TAG auto-generated archives, the checksum is not reliable.

As explained in the Guidelines :

"Keep in mind that github tarballs are generated on-demand, so their modification dates will vary and cause checksum tests to fail."

So again

"For a number of reasons (immutability, availability, uniqueness), you must use the full commit revision hash when referring to the sources."

Yes, there is a number of packages which doesn't respect this Guidelines and use tag/release archive (probably old packages). But there is also a number of packages which respect it.

And it is the role of the reviewer to check and explain this. Nothing complex. Enough examples in the wiki/repo to look at.

Remi.

-- packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging

On Wed, Jun 24, 2015 at 10:05 PM, Remi Collet Fedora@famillecollet.com wrote:

When upstream provides a tarball (usually because they run "make dist" to provide a usable archive), if they regenerate this tarball and reupload it, the checksum will change.

The existing guideline covers that and considers it compliant, so it doesn't matter. However, I've found that statement doesn't apply anyway in the situation when you use either commit hash or Git tag. I thought I had deleted that checksum error reference, but I forgot. Thanks for pointing it out to remind me to delete it. The checksum matches, I've tested it out myself using fedora-review.

You can read about it here: http://git-scm.com/docs/git-archive

On Thu, Jun 25, 2015 at 1:05 AM, Pierre-Yves Chibon pingou@pingoured.fr wrote:

I think this is the core of our disagreement, many of the project we package are doing this. Ask Remi, he maintains several hundreds of packages and he can testify that this behavior is occurring also for projects that are packaged in Fedora and in other distributions. The fact that you find it hard to imagine is unfortunately not correlated with the fact that it occurs, also for projects we (as in Fedora) are interested in.

I don't believe it is appropriate to throw the baby out with the bath water. I've clarified that IF you believe a project is engaging in re-tagging you use the commit hash method. Just because some projects are abusing a function doesn't mean you issue a blanket ban.

What my draft clarifies is that it is packagers decision on which method he wants to use; and it describes a clear procedure that must be followed if upstream is engaging in re-tagging.

If you wish to use the commit hash method all the time when dealing with Git, you can. There is nothing in my text which says otherwise.

On Tue, Jun 23, 2015 at 8:11 AM, Remi Collet Fedora@famillecollet.com wrote:

Le 18/04/2015 22:28, Dave Johansen a écrit :

...

What's the right way to handle github URLs? Currently,

I think Guidelines are clear.

https://fedoraproject.org/wiki/Packaging:SourceURL?rd=Packaging/SourceURL#Gi...

"For a number of reasons (immutability, availability, uniqueness), you must use the full commit revision hash when referring to the sources."

...

I'm working on the spec file for breathe and the URL is https://github.com/michaeljones/breathe/archive/v4.0.0.tar.gz and the

AFAIK, this is NOT immutable (upstream can delete/update tag) so not valid.

I believe it's true that the tag can be deleted or changed, which could cause problems with reproducability. I changed the above URL to use the hash of the commit and that gives the same results but with the guarantee of being reproducible. For the sake of documentation, here's the updated lines from the .spec:

%global owner michaeljones %global commit0 aa95a879dccd63cfac5156bd9a303d41ea1d2490

Name: breathe Source0: https://github.com/%%7Bowner%7D/%%7Bname%7D/archive/%%7Bname%7D-%%7Bcommit0%...

%setup -qn %{name}-%{commit0}