Hi,
I am proposing for inclusion a macro set aimed at automating the packaging of forge-hosted projects.
— Packaging draft: https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation — FPC ticket: https://pagure.io/packaging-committee/issue/719 (without the “hasdraft” tag because I don't know how to add it in pagure) — fedora-rpm-macros RFE with the macro file: https://bugzilla.redhat.com/show_bug.cgi?id=1523779
What it does: conversion of a forge url, version, tag, commit set to the values expected in rpm specfiles, in optional rpm variables. Computation of the corresponding %{dist}.
Objective: centralize forge structure know-how in a single technical place, deprecate all the complex manual forge URL handling spread over many Fedora spec files, simplify packaging and spend time on more interesting stuff.
What's currently implemented: definitions for github.com and code.googlesource.com (I didn't want to propose stuff I didn't use myself. Adding more definitions is trivial. The macros are in Lua which is change-friendly, no arcane rpm syntax knowledge is needed).
Please consult packaging draft for full information.
This is a spin-off of the work I'm currently doing on Go packaging, as Go is heavily forge-oriented. I took the time to extract the generic non-Go-specific forge knowledge in a separate macro file. The macros have been heavily tested on real-life Go projects with quite a lot of variance, on EL7 and rawhide. That's why they come with built-in error handling.
Regards,
On Fri, Dec 08, 2017 at 07:03:48PM +0100, nicolas.mailhot@laposte.net wrote:
I am proposing for inclusion a macro set aimed at automating the packaging of forge-hosted projects.
Could we be more specific about what "forges" are supported? I see github and googlesource.com in the examples, but I think this is more discoverable for people packaging if we call them out more clearly.
Also, is "forge" synonymous with "git hosting service" as used here? https://fedoraproject.org/wiki/Packaging:SourceURL#Git_Hosting_Services
Ah, here....
What's currently implemented: definitions for github.com and code.googlesource.com
Could open-source solutions pagure.io and gitlab.com be added, please?
De: "Matthew Miller" On Fri, Dec 08, 2017 at 07:03:48PM +0100, nicolas.mailhot@laposte.net wrote:
I am proposing for inclusion a macro set aimed at automating the packaging of forge-hosted projects.
Also, is "forge" synonymous with "git hosting service" as used here?
Here a "forge" is pretty much anything that allows access to version/commit/tag archives on normalized paths, that can be deduced from project root URL on the "forge" + version/commit/tag/scm info.
What's currently implemented: definitions for github.com and code.googlesource.com
Could open-source solutions pagure.io and gitlab.com be added, please?
Well I currently do not use those, so I wouldn't know what definitions to add :)
It takes maximum 5 min, for someone who knows the structure of a "forge" to add it to the macro, it's just a copy of an existing "if (forge == something) then" block, adaptation of the lua regexps to extract the correct variables from the URL path, and then use of the result in the inner if tag/version/commit block to set rpm variables
The hard part was to define a generic macro structure, with a packager-friendly API, sane rpm variable names, reasonable rule ordering and error handling.
Regards,
De: "Matthew Miller"
Hi
Could open-source solutions pagure.io and gitlab.com be added, please?
Since I'm a nice person I added GitLab support this morning (both gitlab.com and hosted gitlab). Releases are clearly an afterthought in GitLab, they depend on free-form tags and can't be used in rpm without knowing the corresponding hash (so worst-case, packaging a GitLab release requires declaration of version + tag + commit and not just version as on other forges).
As far as I've seen pagure.io does not allow direct download of release, tag or commit project archives, so it can not be supported right now. RFE: https://pagure.io/pagure/issue/2845
Regards,
On Sat, Dec 9, 2017 at 3:33 AM, nicolas.mailhot@laposte.net wrote:
Since I'm a nice person I added GitLab support this morning (both gitlab.com and hosted gitlab). Releases are clearly an afterthought in GitLab, they depend on free-form tags and can't be used in rpm without knowing the corresponding hash (so worst-case, packaging a GitLab release requires declaration of version + tag + commit and not just version as on other forges).
I made an RFE for this a while back, people can thumbs it up so that it gets more priority: https://gitlab.com/gitlab-org/gitlab-ce/issues/38830
On Sat, Dec 09, 2017 at 09:33:17AM +0100, nicolas.mailhot@laposte.net wrote:
Since I'm a nice person I added GitLab support this morning (both gitlab.com and hosted gitlab). Releases are clearly an afterthought in GitLab, they depend on free-form tags and can't be used in rpm without knowing the corresponding hash (so worst-case, packaging a GitLab release requires declaration of version + tag + commit and not just version as on other forges).
As far as I've seen pagure.io does not allow direct download of release, tag or commit project archives, so it can not be supported right now. RFE: https://pagure.io/pagure/issue/2845
Cool -- thanks for both of these!
On Fri, Dec 08, 2017 at 07:03:48PM +0100, nicolas.mailhot@laposte.net wrote:
Hi,
I am proposing for inclusion a macro set aimed at automating the packaging of forge-hosted projects.
— Packaging draft: https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation — FPC ticket: https://pagure.io/packaging-committee/issue/719 (without the “hasdraft” tag because I don't know how to add it in pagure) — fedora-rpm-macros RFE with the macro file: https://bugzilla.redhat.com/show_bug.cgi?id=1523779
What it does: conversion of a forge url, version, tag, commit set to the values expected in rpm specfiles, in optional rpm variables. Computation of the corresponding %{dist}.
Impressive! I just tested this on some random package using github and everything works great.
Would it be possible to drop the requirement to have "/" at the end of a github URL? I think it's natural to paste the URL without the trailing slash...
Your instructions say "just copy the file into /usr/lib/rpm/macros.d", suggesting that the name can by anything, but I think it has to start with "macros.". At least "forgeforge.macros" did not work here.
Wouldn't it be better to recommend %autosetup instead of %setup? It's one less thing to change if patches are added.
In the docs, the instructions under "Packaging a pre-release commit" actually apply to post-release commits. Maybe change the title to "Packaging a pre-release/post-release commit" and add a text like "See https://fedoraproject.org/wiki/Packaging:Versioning how to adjust Release tag for pre-/post- release commits.".
Zbyszek
Objective: centralize forge structure know-how in a single technical place, deprecate all the complex manual forge URL handling spread over many Fedora spec files, simplify packaging and spend time on more interesting stuff.
What's currently implemented: definitions for github.com and code.googlesource.com (I didn't want to propose stuff I didn't use myself. Adding more definitions is trivial. The macros are in Lua which is change-friendly, no arcane rpm syntax knowledge is needed).
Please consult packaging draft for full information.
This is a spin-off of the work I'm currently doing on Go packaging, as Go is heavily forge-oriented. I took the time to extract the generic non-Go-specific forge knowledge in a separate macro file. The macros have been heavily tested on real-life Go projects with quite a lot of variance, on EL7 and rawhide. That's why they come with built-in error handling.
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
De: "Zbigniew Jędrzejewski-Szmek"
Impressive! I just tested this on some random package using github and everything works great.
Thanks for the nice feedback
Would it be possible to drop the requirement to have "/" at the end of a github URL? I think it's natural to paste the URL without the trailing slash...
Not having to handle special cases was easier for me, but ok done, the macro is now less pedantic.
Your instructions say "just copy the file into /usr/lib/rpm/macros.d", suggesting that the name can by anything, but I think it has to start with "macros.".
You're right, it's documented now.
Wouldn't it be better to recommend %autosetup instead of %setup? It's one less thing to change if patches are added.
I don't have a good history with %autosetup :) It tends to hate the patches I produce.
OTOH it would be nice if the macro could adjust %setup to mean %setup -n %{archivename} when necessary, but I couldn't figure how to do it cleanly.
add a text like "See https://fedoraproject.org/wiki/Packaging:Versioning how to adjust Release tag for pre-/post- release commits.".
Done, thank you for the review.
Regards,
On Sat, 2017-12-09 at 13:34 +0100, nicolas.mailhot@laposte.net wrote:
I don't have a good history with %autosetup :) It tends to hate the patches I produce.
Did you know you still can/have to pass it a prefix level? That tripped me up for a while. For most cases, you want:
%autosetup -p1
If you have patches that apply at different levels, you can't use it, unless there's a trick I don't know about.
De: "Adam Williamson"
On Sat, 2017-12-09 at 13:34 +0100, nicolas.mailhot@laposte.net wrote:
I don't have a good history with %autosetup :) It tends to hate the patches I produce.
Did you know you still can/have to pass it a prefix level? That tripped me up for a while. For most cases, you want:
%autosetup -p1
That's probably what I needed last time, but it was faster to add a %patch line than to investigate :). I find out nowadays there are lots of things I could do better if only I had the time to look them up.
If you have patches that apply at different levels, you can't use it, unless there's a trick I don't know about.
My patches are all -p1 as taught by ancient rpm lore, but sometimes I mix patches from other origins and those can be anything.
Thanks for the tip, I will try to remember it
Regards,
On Tue, 2017-12-12 at 21:11 +0100, nicolas.mailhot@laposte.net wrote:
If you have patches that apply at different levels, you can't use it, unless there's a trick I don't know about.
My patches are all -p1 as taught by ancient rpm lore, but sometimes I mix patches from other origins and those can be anything.
Thanks for the tip, I will try to remember it
No problem. I find %autosetup so handy that usually if I need to use a patch from somewhere else that's at a different prefix level, I just hand-edit that patch to be at level 1 so I can keep using it...
On Tue, Dec 12, 2017 at 1:16 PM, Adam Williamson adamwill@fedoraproject.org wrote:
On Tue, 2017-12-12 at 21:11 +0100, nicolas.mailhot@laposte.net wrote:
If you have patches that apply at different levels, you can't use it, unless there's a trick I don't know about.
My patches are all -p1 as taught by ancient rpm lore, but sometimes I mix patches from other origins and those can be anything.
Thanks for the tip, I will try to remember it
No problem. I find %autosetup so handy that usually if I need to use a patch from somewhere else that's at a different prefix level, I just hand-edit that patch to be at level 1 so I can keep using it...
Definitely agree with rewriting the patch to fit %autosetup. It also reminds me to resubmit the patch upstream whenever I find patches that old :)
- Ken
Hi
Anyway, to answer some of the questions posted during review and in:
https://meetbot.fedoraproject.org/fedora-meeting-2/2017-12-13/fpc.2017-12-13...
1. I just posted the second part of the proposal (the Go-specific bits). Read it there https://fedoraproject.org/wiki/More_Go_packaging to understand some of the choices of this draft.
2. I added a %forgeautosetup helper for %autosetup users (though I don't use it myself, some testing would be appreciated)
3. I added some optional parameters for parameter lovers. Though the macro processes many rpm variables, and sets many others, which are used in other parts of the spec including in the proposed Go autoprovides, so full parametrization is neither suitable nor technically possible.
This is not a single-purpose macro that influences a single part of the spec. Forge and SCM data has effects on all parts of the spec, like %{name} or %{version}. A %{commit} or %{tag} is just a form of %{version} anyway.
People will have to set the variables to specific names anyway for the rest of the automation to work, our guidelines already mandate lots of variables to handle forge hosting, this proposal actually shrinks the number of variables needed to handle this kind of project (and don't make me start on the way different parts of our guidelines name the same variable in slightly different ways, causing a mess when you try to put everything together).
4. There is a bug in EL7 that causes spectool not to process the resulting files. rpmbuild and mock work fine though. I added a -i switch to the macro that prints the resolved source url, you can then dump it in curl, wget or whatever in EL7. Alternatively, get someone to fix the EL7 toolchain.
5. The macro does not handle nor intends to handle multiple downloaded tarballs. This is quite a rare case and it becomes infinitesimal when you add modern scm-based software publishing services to the mix. Plus, the Go automation is centered on a single root import path.
6. Most specs lag the current GitHub guidelines, the current GitHub guidelines are broken as written (because GiHub changed), and I'm pretty sure the resulting packages would fail a source download test (either because the url no longer exists or because it names the downloaded file some other way). That pretty conclusively shows the current way of handling such services does not work. If you want more proof, go look at some Go spec files that rely on GitHub (for example golang-googlecode-google-api-client). I haven't tested the other services our guidelines cover, they are probably also broken in some way.
7. the way of handling corner cases is already documented in the proposal (that's why it's so long, it treats all cases). If you have some other corner case in mind, please post it.
8. the macro could certainly grow as more software hosting services are added. That's not a problem because the complexity is not in the handling of a specific service, it's in the other common parts. Adding a service it mostly writing the Lua patterns (~regexes) to extract the needed parts from the project url, and then combining them the same way our guidelines currently do. The difference is that individual packagers do not need to read or apply those guidelines, the result is made available to the whole distro as soon as the macro package is updated. So the process complexity actually shrinks. Here is the full length of the GitHub-specific block for example
if (string.match(forge, "^github[%.-]") or string.match(forge, "[%.-]github[%.]")) then forgeurl = string.match(forgeurl, "https://%5B%5E/%5D+/%5B%5E/%5D+/%5B%5E/#?%5D+") if (forgeurl == "") then if not silent then rpm.expand("%{error:GitHub URLs must match https://(%E2%80%A6%5B-.%5D)github%5B-.%5D%E2%80%A6/owner/repo !\n}") end else explicitset("forgeurl", forgeurl) safeset("archiveext", "tar.gz") safeset("forgesetupargs", "-n %{archivename}") if (commit ~= "") or (tag ~= "") then safeset("scm", "git") end local owner = string.match(forgeurl, "^[^:]+://[^/]+/([^/]+)") local repo = string.match(forgeurl, "^[^:]+://[^/]+/[^/]+/([^/]+)") if (tag ~= "") then safeset("archivename", repo .. "-%{tag}") safeset("archiveurl", "%{forgeurl}/archive/%{tag}.%{archiveext}") else if (commit ~= "") then safeset("archivename", repo .. "-%{commit}") safeset("archiveurl", "%{forgeurl}/archive/%{commit}/" .. repo .. "-%{commit}.%{archiveext}") else safeset("archivename", repo .. "-%{version}") safeset("archiveurl", "%{forgeurl}/archive/v%{version}.%{archiveext}") end end end end
Don't tell me that's overly difficult to understand or adapt. I'm sure the mediawiki markup dedicated to GitHub in our guidelines is actually longer (plus, it does not work and it is not applied consistently)
It could be condensed by removing error handling, but what would be the point. A macro that actually does error handling is nice, for a change.
9. added with the new Go-specific proposal this proposal achieves a shrinking of some Go specs by 90%, so it is way over the "half spec" simplification bar
And I think I covered all the raised objections.
Regards,
Hi,
- There is a bug in EL7 that causes spectool not to process the resulting files. rpmbuild and mock work fine though. I
added a -i switch to the macro that prints the resolved source url, you can then dump it in curl, wget or whatever in EL7. Alternatively, get someone to fix the EL7 toolchain.
Anyway, with the latest changes, we don't hit the EL7 bug anymore so spectool also works in EL7
Regards,
On 12/08/2017 08:03 PM, nicolas.mailhot@laposte.net wrote:
Hi,
I am proposing for inclusion a macro set aimed at automating the packaging of forge-hosted projects.
— Packaging draft: https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation — FPC ticket: https://pagure.io/packaging-committee/issue/719 (without the “hasdraft” tag because I don't know how to add it in pagure) — fedora-rpm-macros RFE with the macro file: https://bugzilla.redhat.com/show_bug.cgi?id=1523779
What it does: conversion of a forge url, version, tag, commit set to the values expected in rpm specfiles, in optional rpm variables. Computation of the corresponding %{dist}.
Objective: centralize forge structure know-how in a single technical place, deprecate all the complex manual forge URL handling spread over many Fedora spec files, simplify packaging and spend time on more interesting stuff.
Kudos for work on reducing repetitive complex error prone cruft in specs!
But don't override %setup. There's no need for such abuse (yes overriding rpm build-ins is abuse even if it still doesn't prevent it) and no good will come out of it.
Use something like %forgesetup instead which signifies it's something a bit special, and allows you to %autosetup underneath on versions where macro arguments are expanded (rpm >= 4.14)
- Panu -
De: "Panu Matilainen"
Hi Panu,
Kudos for work on reducing repetitive complex error prone cruft in specs!
Thanks!
But don't override %setup. There's no need for such abuse
It is really pretty safe, the macro controls the downloaded file, the file structure is known, the only time it won't "just work" is when a spec needs to call %setup several times (in that case the arguments will be wrong for the second call).
And, how many specs need to do that? Especially when the target project uses modern forge hosting? That's already deep rpm black magic land for most packagers.
I can add a guard so the override is only set when %{setupargs} is non empty if you want.
Use something like %forgesetup instead which signifies it's something a bit special
I don't want to signify it's a bit special, that's one more step the packager can do wrong, I'd rather have the common case easy and the complex cases possible.
Granted, the macro makes the complex cases a tad more complex, but if you're already in multiple-setup-call hell, how difficult is it to set %{setupargs} to "" ?
But, I'll bow to whatever FPC decides. *I* won't make the wrong macro call in my specs :).
and allows you to %autosetup underneath on versions where macro arguments are expanded (rpm >= 4.14)
Interesting, are the changes described somewhere? Not that I want to break compat with el7 from the startup if I can avoid it
Regards,
De: "Panu Matilainen"
Hi Panu,
But don't override %setup. There's no need for such abuse
It is really pretty safe, the macro controls the downloaded file, the file structure is known, the only time it won't "just work" is when a spec needs to call %setup several times (in that case the arguments will be wrong for the second call).
BTW the macro know even more than that, it knows the archive filename for which setup needs specific arguments so it could be made 100% transparent and safe if it was possible to specify
"if archive == %{archivename}%{archiveext} change %setup to %setup %{?setupargs}"
Regards,
On Mon, Dec 11, 2017 at 5:57 AM, nicolas.mailhot@laposte.net wrote:
De: "Panu Matilainen"
Hi Panu,
But don't override %setup. There's no need for such abuse
It is really pretty safe, the macro controls the downloaded file, the file structure is known, the only time it won't "just work" is when a spec needs to call %setup several times (in that case the arguments will be wrong for the second call).
BTW the macro know even more than that, it knows the archive filename for which setup needs specific arguments so it could be made 100% transparent and safe if it was possible to specify
"if archive == %{archivename}%{archiveext} change %setup to %setup %{?setupargs}"
%setup is a very special macro. It's not actually written in macro language, (and neither is %patch, for that matter). It's really not a good idea to override it, especially if people need to do multi-source things.
And the issue you're having that requires %setupargs is not a problem in RPM 4.14, which ships in Fedora 27 and newer, and will be part of RHEL 8.
If you'd like this to be fixed in EL7 or something, then file a bug report against RHEL 7 and see if anything can be done about it.
Hi Neal,
And the issue you're having that requires %setupargs is not a problem in RPM 4.14
I don't have an issue with %setupargs, I have an issue with requiring packagers to change stuff in the spec header *and* at %prep level, which is not in the same place of the spec. That is something which has wasted huge quantities of man-hours in the past, even for experienced packagers.
The automation knows how the downloaded source archive will be named, what is the structure of the source archive (the arguments that need passing to %setup for this archive). The question is just how to pass that knowledge from the automation macro call to %setup or %autosetup.
Overriding %setup makes this work transparently with little risk. If there is a strong opposition to that what is the best way to achieve the same result?
Using a specific setup-ish macro name like suggested by Panu is trivial technically but has the huge drawback that it requires a specific call by the packager (and many will forget it, at least as first). So it de-optimizes the packager workflow. I'd frankly prefer to optimize the packager workflow over helping tooling – that's what costs actual money and potential contributors.
If there is now way to do it cleanly or safely in rpm, I'll de-optimize the packager side. I don't want to cause problems to anyone. But that would be pretty sad.
Regards,
On Mon, Dec 11, 2017 at 01:23:19PM +0100, nicolas.mailhot@laposte.net wrote:
Hi Neal,
And the issue you're having that requires %setupargs is not a problem in RPM 4.14
I don't have an issue with %setupargs, I have an issue with requiring packagers to change stuff in the spec header *and* at %prep level, which is not in the same place of the spec. That is something which has wasted huge quantities of man-hours in the past, even for experienced packagers.
The automation knows how the downloaded source archive will be named, what is the structure of the source archive (the arguments that need passing to %setup for this archive). The question is just how to pass that knowledge from the automation macro call to %setup or %autosetup.
Overriding %setup makes this work transparently with little risk. If there is a strong opposition to that what is the best way to achieve the same result?
Using a specific setup-ish macro name like suggested by Panu is trivial technically but has the huge drawback that it requires a specific call by the packager (and many will forget it, at least as first). So it de-optimizes the packager workflow. I'd frankly prefer to optimize the packager workflow over helping tooling – that's what costs actual money and potential contributors.
I think it's OK. After all you already require a specific '%setup -n <arg>' invocation (<arg> in this case is different then the github default), and setting a few different fields in a specific order. So in practice people are going to go by the template provided in the docs, and whether it's %setup -n something or %forgesetup doesn't make much difference.
Zbyszek
Hi all,
Since most participants seems to be in favor of explicit %setup handling, I've updated the wiki and the macro file
Regards,
On 12/11/2017 06:07 PM, nicolas.mailhot@laposte.net wrote:
Hi all,
Since most participants seems to be in favor of explicit %setup handling, I've updated the wiki and the macro file
Thank you.
- Panu -
On 12/11/2017 12:51 PM, nicolas.mailhot@laposte.net wrote:
De: "Panu Matilainen"
Hi Panu,
Kudos for work on reducing repetitive complex error prone cruft in specs!
Thanks!
But don't override %setup. There's no need for such abuse
It is really pretty safe, the macro controls the downloaded file, the file structure is known, the only time it won't "just work" is when a spec needs to call %setup several times (in that case the arguments will be wrong for the second call).
No. It's not about whether it's "pretty safe" or not, it's an rpm built-in and it's simply not yours to override (and what about the next guy who wants to override it for their own purposes, say, go-specific thing?). The fact that you can do so is basically a loophole in rpm that just hasn't gotten closed, yet. Don't bet the future on loopholes.
- Panu -
On 12/11/2017 12:51 PM, nicolas.mailhot@laposte.net wrote:
De: "Panu Matilainen"
and allows you to %autosetup underneath on versions where macro arguments are expanded (rpm >= 4.14)
Interesting, are the changes described somewhere? Not that I want to break compat with el7 from the startup if I can avoid it
Sorry, forgot to reply to this part.
http://rpm.org/wiki/Releases/4.14.0#Macros has the bullet points, other than that it's pretty scarce.
https://bugzilla.redhat.com/show_bug.cgi?id=1397209 is the original case with pointers to upstream commit etc.
- Panu -
Hi,
I am proposing for inclusion a set of rpm technical files aimed at automating the packaging of forge-hosted projects.
— Packaging draft: https://fedoraproject.org/wiki/More_Go_packaging — https://pagure.io/packaging-committee/issue/734 — go-srpm-macros RFE with the technical files: https://bugzilla.redhat.com/show_bug.cgi?id=1526721
This proposal is integrated with and depends on the https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft It builds on the hard work of the Go SIG and reuses the rpm automation of https://fedoraproject.org/wiki/PackagingDrafts/Go when it exists, and produces compatible packages.
What it does:
— drastically shorter spec files, up to 90% in some cases, often removing hundreds of lines per spec. — simple, packager-friendly spec syntax — automated package naming derived from the native identifier (import path). No more packages names without any relation with current upstream naming. — working Go autoprovides. No forgotten provides anymore. — working Go autorequires. No forgotten requires anymore. — strict automated directory ownership (used by autorequires and autoprovides). — centralized computation of source URLs (via Forge-hosted projects packaging automation). No more packages lacking guidelines. No more broken guidelines no one notices. — easy switch between commits, tags and releases (via Forge-hosted projects packaging automation). No more packages stuck on commits when upstream starts releasing. — guidelines-compliant automated snapshot naming, including snapshot timestamps (via Forge-hosted projects packaging automation). No more packages stuck in 2014. — guidelines-compliant bootstraping. — systematic use of the Go switches defined by the Go maintainer. Easy to do changes followed by a mass rebuild. — flexibility, do the right thing transparently by default, leave room for special cases and overrides. — no bundling (a.k.a. vendoring) due to the pain of packaging one more Go dependency. — centralized Go macros that can be audited and enhanced over time. — aggressive leverage of upstream unit tests to detect quickly broken code. — no reliance on external utilities to compute code requirements. No more dependencies that do not match the shipped Go code. — no maze of variable indirections. No more packages everyone is afraid of touching. — no maze of cargo-culted and bitrotting shell code. No more packages everyone is afraid of touching. — compatibility with existing packages (though many are so obsolete they need complete replacement)
Please consult packaging draft for full information.
The proposal has been tested in Rawhide and EL7 over a set of ~ 140 Go packages. This set is a mix of current Fedora packages, bumped to a more recent version, rewrites of Fedora packages, and completely new packages.
I hope posting the second part of the automation will answer some questions people had on the https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft
Regards,
Hi,
I am proposing for inclusion a set of rpm technical files aimed at automating the packaging of forge-hosted projects.
— Packaging draft: https://fedoraproject.org/wiki/More_Go_packaging — https://pagure.io/packaging-committee/issue/734 — go-srpm-macros RFE with the technical files: https://bugzilla.redhat.com/show_bug.cgi?id=1526721
This proposal is integrated with and depends on the https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft It builds on the hard work of the Go SIG and reuses the rpm automation of https://fedoraproject.org/wiki/PackagingDrafts/Go when it exists, and produces compatible packages.
What it does:
— drastically shorter spec files, up to 90% in some cases, often removing hundreds of lines per spec. — simple, packager-friendly spec syntax — automated package naming derived from the native identifier (import path). No more packages names without any relation with current upstream naming. — working Go autoprovides. No forgotten provides anymore. — working Go autorequires. No forgotten requires anymore. — strict automated directory ownership (used by autorequires and autoprovides). — centralized computation of source URLs (via Forge-hosted projects packaging automation). No more packages lacking guidelines. No more broken guidelines no one notices. — easy switch between commits, tags and releases (via Forge-hosted projects packaging automation). No more packages stuck on commits when upstream starts releasing. — guidelines-compliant automated snapshot naming, including snapshot timestamps (via Forge-hosted projects packaging automation). No more packages stuck in 2014. — guidelines-compliant bootstraping. — systematic use of the Go switches defined by the Go maintainer. Easy to do changes followed by a mass rebuild. — flexibility, do the right thing transparently by default, leave room for special cases and overrides. — no bundling (a.k.a. vendoring) due to the pain of packaging one more Go dependency. — centralized Go macros that can be audited and enhanced over time. — aggressive leverage of upstream unit tests to detect quickly broken code.
Please consult packaging draft for full information.
The proposal has been tested in Rawhide and EL7 over a set of ~ 140 Go packages. This set is a mix of current Fedora packages, bumped to a more recent version, rewrites of Fedora packages, and completely new packages.
I hope posting the second part of the automation will answer some questions people had on the https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft
Regards,
Hi,
I am proposing for inclusion a set of rpm technical files aimed at automating the packaging of forge-hosted projects.
- Packaging draft: https://fedoraproject.org/wiki/More_Go_packaging - https://pagure.io/packaging-committee/issue/734 - go-srpm-macros RFE with the technical files: https://bugzilla.redhat.com/show_bug.cgi?id=1526721
This proposal is integrated with and depends on the https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft It builds on the hard work of the Go SIG and reuses the rpm automation of https://fedoraproject.org/wiki/PackagingDrafts/Go when it exists, and produces compatible packages.
What it does:
- drastically shorter spec files, up to 90% in some cases, often removing hundreds of lines per spec. - simple, packager-friendly spec syntax - automated package naming derived from the native identifier (import path). No more packages names without any relation with current upstream naming. - working Go autoprovides. No forgotten provides anymore. - working Go autorequires. No forgotten requires anymore. - strict automated directory ownership (used by autorequires and autoprovides). - centralized computation of source URLs (via Forge-hosted projects packaging automation). No more packages lacking guidelines. No more broken guidelines no one notices. - easy switch between commits, tags and releases (via Forge-hosted projects packaging automation). No more packages stuck on commits when upstream starts releasing. - guidelines-compliant automated snapshot naming, including snapshot timestamps (via Forge-hosted projects packaging automation). No more packages stuck in 2014. - guidelines-compliant bootstraping. - systematic use of the Go switches defined by the Go maintainer. Easy to do changes followed by a mass rebuild. - flexibility, do the right thing transparently by default, leave room for special cases and overrides. - no bundling (a.k.a. vendoring) due to the pain of packaging one more Go dependency. - centralized Go macros that can be audited and enhanced over time. - aggressive leverage of upstream unit tests to detect quickly broken code.
Please consult packaging draft for full information.
The proposal has been tested in Rawhide and EL7 over a set of ~ 140 Go packages. This set is a mix of current Fedora packages, bumped to a more recent version, rewrites of Fedora packages, and completely new packages.
I hope posting the second part of the automation will answer some questions people had on the https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft
Regards,
On Sun, Dec 17, 2017 at 2:11 AM, nicolas.mailhot@laposte.net wrote:
Hi,
I am proposing for inclusion a set of rpm technical files aimed at automating the packaging of forge-hosted projects.
- Packaging draft: https://fedoraproject.org/wiki/More_Go_packaging
- https://pagure.io/packaging-committee/issue/734
- go-srpm-macros RFE with the technical files: https://bugzilla.redhat.com/show_bug.cgi?id=1526721
This proposal is integrated with and depends on the https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft It builds on the hard work of the Go SIG and reuses the rpm automation of https://fedoraproject.org/wiki/PackagingDrafts/Go when it exists, and produces compatible packages.
What it does:
- drastically shorter spec files, up to 90% in some cases, often removing hundreds of lines per spec.
- simple, packager-friendly spec syntax
- automated package naming derived from the native identifier (import path). No more packages names without any relation with current upstream naming.
- working Go autoprovides. No forgotten provides anymore.
- working Go autorequires. No forgotten requires anymore.
- strict automated directory ownership (used by autorequires and autoprovides).
- centralized computation of source URLs (via Forge-hosted projects packaging automation). No more packages lacking guidelines. No more broken guidelines no one notices.
- easy switch between commits, tags and releases (via Forge-hosted projects packaging automation). No more packages stuck on commits when upstream starts releasing.
- guidelines-compliant automated snapshot naming, including snapshot timestamps (via Forge-hosted projects packaging automation). No more packages stuck in 2014.
- guidelines-compliant bootstraping.
- systematic use of the Go switches defined by the Go maintainer. Easy to do changes followed by a mass rebuild.
- flexibility, do the right thing transparently by default, leave room for special cases and overrides.
- no bundling (a.k.a. vendoring) due to the pain of packaging one more Go dependency.
- centralized Go macros that can be audited and enhanced over time.
- aggressive leverage of upstream unit tests to detect quickly broken code.
Please consult packaging draft for full information.
The proposal has been tested in Rawhide and EL7 over a set of ~ 140 Go packages. This set is a mix of current Fedora packages, bumped to a more recent version, rewrites of Fedora packages, and completely new packages.
I hope posting the second part of the automation will answer some questions people had on the https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft
I really do like this. There are only two issues I have with it:
1. This seems to mandate that all packages must be named by their import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
2. Mandating a forge is going to be tricky for self-hosted stuff, or people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
I really do like this. There are only two issues I have with it:
- This seems to mandate that all packages must be named by their
import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
- Mandating a forge is going to be tricky for self-hosted stuff, or
people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
By not using the macros for packages not fitting the model?
I think this is very helpful especially when it's the common practice, and I certainly won't blame anyone doing proper releases and not just a git tag with github releases notes ;)
Regarding naming, I think python packages must be prefixed with python[23]- and can Provides: the upstream project name. On the other hand we have packages like docker that are clearly named after upstream's name, so I don't think that would be a problem for snapd. (and maybe an exception needs to be granted?)
Dridi
On Mon, Jan 22, 2018 at 8:33 AM, Dridi Boukelmoune dridi.boukelmoune@gmail.com wrote:
I really do like this. There are only two issues I have with it:
- This seems to mandate that all packages must be named by their
import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
- Mandating a forge is going to be tricky for self-hosted stuff, or
people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
By not using the macros for packages not fitting the model?
The issue is that the new Go macros are tightly wound into the forge macros. I just want to be sure that we can leverage things like the dependency generators without all the other stuff.
I think this is very helpful especially when it's the common practice, and I certainly won't blame anyone doing proper releases and not just a git tag with github releases notes ;)
Regarding naming, I think python packages must be prefixed with python[23]- and can Provides: the upstream project name. On the other hand we have packages like docker that are clearly named after upstream's name, so I don't think that would be a problem for snapd. (and maybe an exception needs to be granted?)
This rule only applies to Python packages that have modules that are designed to be imported by other Python code. Otherwise, this is not necessary.
On Mon, Jan 22, 2018 at 2:45 PM, Neal Gompa ngompa13@gmail.com wrote:
On Mon, Jan 22, 2018 at 8:33 AM, Dridi Boukelmoune dridi.boukelmoune@gmail.com wrote:
I really do like this. There are only two issues I have with it:
- This seems to mandate that all packages must be named by their
import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
- Mandating a forge is going to be tricky for self-hosted stuff, or
people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
By not using the macros for packages not fitting the model?
The issue is that the new Go macros are tightly wound into the forge macros. I just want to be sure that we can leverage things like the dependency generators without all the other stuff.
I think this is very helpful especially when it's the common practice, and I certainly won't blame anyone doing proper releases and not just a git tag with github releases notes ;)
Regarding naming, I think python packages must be prefixed with python[23]- and can Provides: the upstream project name.
I'm not sure if this matters in this discussion but an example Python3 part of a spec file https://fedoraproject.org/wiki/Packaging:Python to accommodate also EPEL (which on CentOS7 prefixes Python3 packages with python34 and not python3) would look like:
%package -n python%{python3_pkgversion}-%{pname} %{?python_provide:%python_provide python%{python3_pkgversion}-%{pname}}
Macin
On the
other hand we have packages like docker that are clearly named after upstream's name, so I don't think that would be a problem for snapd. (and maybe an exception needs to be granted?)
This rule only applies to Python packages that have modules that are designed to be imported by other Python code. Otherwise, this is not necessary.
-- 真実はいつも一つ!/ Always, there's only one truth! _______________________________________________ packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org
----- Original Message -----
From: "Marcin Dulak" marcin.dulak@gmail.com To: "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Cc: golang@lists.fedoraproject.org, "Development discussions related to Fedora" devel@lists.fedoraproject.org Sent: Monday, January 22, 2018 4:04:19 PM Subject: Re: [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
On Mon, Jan 22, 2018 at 2:45 PM, Neal Gompa < ngompa13@gmail.com > wrote:
On Mon, Jan 22, 2018 at 8:33 AM, Dridi Boukelmoune < dridi.boukelmoune@gmail.com > wrote:
I really do like this. There are only two issues I have with it:
- This seems to mandate that all packages must be named by their
import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
- Mandating a forge is going to be tricky for self-hosted stuff, or
people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
By not using the macros for packages not fitting the model?
The issue is that the new Go macros are tightly wound into the forge macros. I just want to be sure that we can leverage things like the dependency generators without all the other stuff.
I think this is very helpful especially when it's the common practice, and I certainly won't blame anyone doing proper releases and not just a git tag with github releases notes ;)
Regarding naming, I think python packages must be prefixed with python[23]- and can Provides: the upstream project name.
I'm not sure if this matters in this discussion but an example Python3 part of a spec file https://fedoraproject.org/wiki/Packaging:Python to accommodate also EPEL (which on CentOS7 prefixes Python3 packages with python34 and not python3) would look like:
%package -n python%{python3_pkgversion}-%{pname} %{?python_provide:%python_provide python%{python3_pkgversion}-%{pname}}
Macin
Hopefully something like this will never happen as generally I'm strongly against shipping multiple versions(of one implementation) of Go concurrently.
JC
On the
other hand we have packages like docker that are clearly named after upstream's name, so I don't think that would be a problem for snapd. (and maybe an exception needs to be granted?)
This rule only applies to Python packages that have modules that are designed to be imported by other Python code. Otherwise, this is not necessary.
-- 真実はいつも一つ!/ Always, there's only one truth! _______________________________________________ packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
----- Mail original ----- De: "Neal Gompa"
On Mon, Jan 22, 2018 at 8:33 AM, Dridi Boukelmoune
I really do like this. There are only two issues I have with it:
- This seems to mandate that all packages must be named by their
import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
- Mandating a forge is going to be tricky for self-hosted stuff, or
people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
By not using the macros for packages not fitting the model?
The issue is that the new Go macros are tightly wound into the forge macros. I just want to be sure that we can leverage things like the dependency generators without all the other stuff.
Hi Neal,
I should probably not let this pass without clarifying:
1. if your concern is that the *forge* macros will block packaging of projects not hosted on known forges, they won't: — they include code to disable transparently and without exploding the associated bits if the hosting site is unknown — the packager can pre-define the variables that would have been computed if the site was known and all the rest of the automation will just work — baring that the packager can ignore the site-related computations and use whatever he wants instead in Source: and *setup, which are the only parts that depend on hosting structure knowledge,
2. if your concern is that the *forge* macros are defective somewhere I'd be curious where as you'd be the first to report an actual technical problem. I've used them intensively in rawhide and el7 with many different rpm tools and they are rock solid for me
3. if your concern is that they won't ever be merged due to some people non-constructive obstruction, well I share it to some point but I won't spend more time arguing unicorns and what-if-maybe I-don't-like-commits-let's-make-them-awful-to-use. The people blocking there are free to take up the maintenance of my ~400 specs and redo them as they want, we'll see how they love to waste tens of minutes per spec to position setup args manually when they have to do it for hundreds of them, every time there is a package update or a forge relayout.
The main difficulty with the proposal is not the *forge part, it's that automated autodeps are well, automated. As every automation they are strict and unyielding and do not take approximations well. They will fail spectacularly in the following cases:
1. the packager does not package all the deps of its code → the resulting package is uninstallable because autodeps add requires on missing bits
2. the packager packages code with garbage imports (very common in example or test code) → the resulting package is uninstallable because autodeps add require
3. the packager does not own properly the directories where its Go code is installed → given that Go deps depend on directories, Go autodeps also depend on proper directory ownership. Positioning autodeps on Go files themselves would make package build times increase many many times as rpm would try to compute the same autodeps for every single .go file in a directory. (and Go autodeps are already sloooow as snails because invoking the go command is slow)
All those things are mitigated by the use of %goinstall that filters usual example/test/vendor dirs so they don't trigger autodeps, and tries very hard to own all relevant directories.
For those reasons I don't propose to activate autodeps in old-style golang packages. They need conversion (and review by a human to check no problem code is deployed) first.
Regards,
De: "Neal Gompa"
The issue is that the new Go macros are tightly wound into the forge macros. I just want to be sure that we can leverage things like the dependency generators without all the other stuff.
Hi Neal,
I should probably not let this pass without clarifying: 3. if your concern is that they won't ever be merged
And the forge macros are now available since redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream renaming the file). Heartfelt thanks to Jason Tibbitts !
I'll focus on pushing the Go parts to go-srpm-macros now so everyone can play with them
Regards,
I wish this message wasn't crossposted everywhere, but I don't want to lose any discussion by trimming the CC list. Sorry if replies generate bounces for some.
"nm" == nicolas mailhot nicolas.mailhot@laposte.net writes:
nm> And the forge macros are now available since nm> redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream nm> renaming the file). Heartfelt thanks to Jason Tibbitts !
Please don't forget to let me know when it's time to start thinking about pushing this down to F27. And maybe F26. And as far as I can tell it should work with only minor modification in EPEL7 (via epel-rpm-macros). I don't know about EPEL6, but we really should look at it given some of the other discussions about specfile compatibility. Some packagers wouldn't ever use it if it doesn't work everywhere.
Finally, we should also talk about whether there is any integration or automation possible between fedpkg and specfiles configured with these macros.
- J<
----- Mail original ----- De: "Jason L Tibbitts III"
"nm" == nicolas mailhot nicolas.mailhot@laposte.net writes:
nm> And the forge macros are now available since nm> redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream nm> renaming the file). Heartfelt thanks to Jason Tibbitts !
Please don't forget to let me know when it's time to start thinking about pushing this down to F27. And maybe F26. And as far as I can tell it should work with only minor modification in EPEL7 (via epel-rpm-macros).
I don't know about EPEL6, but we use it as-is in EL7 and it works just as well (except maybe for the %autosetup bits but IIRC that's autosetup which is broken in EL7). In fact it has probably been used more heavily in EL7 than in fedora-devel so far. Maybe it also works in EPEL 6 but I've never tried it. I guess it depends mostly on the level of lua support in EL6 rpm and rpm-related tools now the forge macro code is lua only. I'm pretty sure many of the problems in the early versions of the macro were due to non-lua code and its interactions with lua code once the lua-ification started.
It's a good idea to let people play with it in fedora-devel maybe a month, in case I missed something, but from a technical POW I'm prety sure it could be merged up to EL7 now. I'll submit fedora-devel specific tweaks later (just like I submitted bitkeeper.org support today), right now the code is distro-agnostic. For my part I doubt I'll ever use it in EL6 since I did it for Go and the EL6 Go stack is really too old for a merge to be interesting. Anyway I'll certainly let you know when I feel the time is right (but do not block on me!)
Finally, we should also talk about whether there is any integration or automation possible between fedpkg and specfiles configured with these macros.
I'm afraid my knowledge of recent fedpkg enhancements is too sparse to be of any use there. Though I'm not opposed to the idea at all.
Thanks again!
"nm" == nicolas mailhot nicolas.mailhot@laposte.net writes:
nm> I don't know about EPEL6, but we use it as-is in EL7 and it works nm> just as well (except maybe for the %autosetup bits but IIRC that's nm> autosetup which is broken in EL7).
I had ported autosetup to EPEL6 and then at the next release the macros showed up (without any discussion) in base RHEL6. So they should be there. I'm not entirely sure what's actually broken about %autosetup in EL7, though; I hadn't heard about breakage before this. epel-rpm-macros could conceivably carry %epel_autosetup or something which contains fixes. I mostly understand the internals of %autosetup so maybe there's something I can do.
nm> Maybe it also works in EPEL 6 but I've never tried it. I guess it nm> depends mostly on the level of lua support in EL6 rpm and nm> rpm-related tools now the forge macro code is lua only.
I think it would be worth a try. In my experience not all that much has changed in the Lua interfaces since RHEL6 and even RHEL5. (EL5 just didn't have sources and patches in the Lua namespace.)
nm> For my part I doubt I'll ever use it in EL6 since I did it for Go nm> and the EL6 Go stack is really too old for a merge to be nm> interesting.
Well, sure, Go on EL6 is probably out but these macros were at least presented as being far more general, and I'm sure there are plenty of EPEL6 packages which could benefit. Potential anything that packages a git snapshot of something.
nm> I'm afraid my knowledge of recent fedpkg enhancements is too sparse nm> to be of any use there. Though I'm not opposed to the idea at all.
It's just worth a brainstorm, I think. I can imagine that loads of people would love it if fedpkg could just auto-bump the bits necessary to update a package to today's git head or some specific commit or something like that. Before we didn't really have a good standard way to format a spec when you're using a checkout. Now....
Doesn't necessarily have to start in fedpkg, either. A standalone utility for doing a couple of things like that could be useful as a prototype.
- J<
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: "Jason L Tibbitts III" tibbs@math.uh.edu Cc: golang@lists.fedoraproject.org, "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Tuesday, January 23, 2018 9:28:15 PM Subject: Re: Proposed Fedora packaging guideline: More Go packaging
----- Mail original ----- De: "Jason L Tibbitts III"
"nm" == nicolas mailhot nicolas.mailhot@laposte.net writes:
nm> And the forge macros are now available since nm> redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream nm> renaming the file). Heartfelt thanks to Jason Tibbitts !
Please don't forget to let me know when it's time to start thinking about pushing this down to F27. And maybe F26. And as far as I can tell it should work with only minor modification in EPEL7 (via epel-rpm-macros).
I don't know about EPEL6, but we use it as-is in EL7 and it works just as well (except maybe for the %autosetup bits but IIRC that's autosetup which is broken in EL7). In fact it has probably been used more heavily in EL7 than in fedora-devel so far. Maybe it also works in EPEL 6 but I've never tried it. I guess it depends mostly on the level of lua support in EL6 rpm and rpm-related tools now the forge macro code is lua only. I'm pretty sure many of the problems in the early versions of the macro were due to non-lua code and its interactions with lua code once the lua-ification started.
It's a good idea to let people play with it in fedora-devel maybe a month, in case I missed something, but from a technical POW I'm prety sure it could be merged up to EL7 now. I'll submit fedora-devel specific tweaks later (just like I submitted bitkeeper.org support today), right now the code is distro-agnostic. For my part I doubt I'll ever use it in EL6 since I did it for Go and the EL6 Go stack is really too old for a merge to be interesting. Anyway I'll certainly let you know when I feel the time is right (but do not block on me!)
If we are talking about EPEL6 stack, it is fairly fresh(1.9.2) and stable(it will be on 1.9 for whole of its upstream support), although Go packaging macros are missing.
JC
Finally, we should also talk about whether there is any integration or automation possible between fedpkg and specfiles configured with these macros.
I'm afraid my knowledge of recent fedpkg enhancements is too sparse to be of any use there. Though I'm not opposed to the idea at all.
Thanks again!
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
----- Mail original ----- De: "Jakub Cajka"
Hi Jakub,
For my part I doubt I'll ever use it in EL6 since I did it for Go and the EL6 Go stack is really too old for a merge to be interesting. Anyway I'll certainly let you know when I feel the time is right (but do not block on me!)
If we are talking about EPEL6 stack, it is fairly fresh(1.9.2) and stable(it will be on 1.9 for whole of its upstream support), although Go packaging macros are missing.
Yes the core golang package is there but first the other Go macros are not available in EL6 and second the software level of many common Go packages is very old in EL6.
So, assuming the forge macros just work in EL6, because the little bit of rpm lua they need works the same as in EL7 and devel, updating EL6 to the level needed for a Go spec to be shared mostly unchanged between EL6/EL7 and devel would require:
— making the other Go macros available in EL6 (and I'm far from sure they would work unchanged in EL6, the forge part is easy since it's 100% lua but the other parts exercise the shell and rpm and other stuff that may behave in a slightly different way in such an ancient codebase. Not necessarily a show-stopper but definitely something that requires testing and adjusting time by someone)
— updating all the common Go software packages to the same level as in devel. I don't say it can't be done (after all I'm doing it for EL7 on hundreds of packages) but that's a *lot* more work than just updating a few macro files. Especially if it hits bootstraping issues not existing in EL7 and devel.
Therefore, it requires someone with time and motivation, and should really not be attempted before EL7 and devel are done and things have settled a bit, and by that point will there still be enough interest in upgrading the Go state in EL6 for it to be worth the pain?
Of course one could limit oneself to making the macros available (which still requires some testing and eventually some code porting), that would enable using the same Go spec coding style in EL6 EL7 and devel, if not sharing the spec themselves, but the interest of autodeps is severely limited, if you do not have a large baseline of packages providing the deps (and the dep version) other packages need to build.
That's why I wrote the forge part could be made available in EL6, it depends on little except the lua built in rpm, and can be useful as-is, while the Go part is something else entirely, as its utility is directly linked to the quantity and freshness of Go software packages in the distro.
Regards,
Hi Jason,
"nm" == nicolas mailhot wrote:
Jason L Tibbitts wrote:
nm> And the forge macros are now available since nm> redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream nm> renaming the file). Heartfelt thanks to Jason Tibbitts ! Please don't forget to let me know when it's time to start thinking about pushing this down to F27. And maybe F26. And as far as I can tell it should work with only minor modification in EPEL7 (via epel-rpm-macros).
I don't know about EPEL6, but we use it as-is in EL7 and it works just as well (except maybe for the %autosetup bits but IIRC that's autosetup which is broken in EL7).
It's a good idea to let people play with it in fedora-devel maybe a month, in case I missed something, but from a technical POW I'm prety sure it could be merged up to EL7 now.
Now that the appropriate observation time has passed, and people have not find new problems (in fact they found scenarii that worked I had not thought about) can you please merge
https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/18
and push it to stable?
The Go macros need it and Fedora Go maintainers now want to fork forgemeta if redhat-rpm-config does not make forgemeta available in the Fedora stable versions they target, which would all make a huge mess. I guess it's been too successful ;).
Regards,
----- Original Message -----
From: "Jason L Tibbitts III" tibbs@math.uh.edu To: "nicolas mailhot" nicolas.mailhot@laposte.net Cc: golang@lists.fedoraproject.org, "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Tuesday, January 23, 2018 6:00:24 PM Subject: Re: Proposed Fedora packaging guideline: More Go packaging
I wish this message wasn't crossposted everywhere, but I don't want to lose any discussion by trimming the CC list. Sorry if replies generate bounces for some.
"nm" == nicolas mailhot nicolas.mailhot@laposte.net writes:
nm> And the forge macros are now available since nm> redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream nm> renaming the file). Heartfelt thanks to Jason Tibbitts !
Please don't forget to let me know when it's time to start thinking about pushing this down to F27. And maybe F26. And as far as I can tell it should work with only minor modification in EPEL7 (via epel-rpm-macros). I don't know about EPEL6, but we really should look at it given some of the other discussions about specfile compatibility. Some packagers wouldn't ever use it if it doesn't work everywhere.
I think that it would be great to land it also in the EPEL6/7.
JC
Finally, we should also talk about whether there is any integration or automation possible between fedpkg and specfiles configured with these macros.
- J<
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On Tue, Jan 23, 2018 at 5:45 AM, nicolas.mailhot@laposte.net wrote:
----- Mail original ----- De: "Neal Gompa"
On Mon, Jan 22, 2018 at 8:33 AM, Dridi Boukelmoune
I really do like this. There are only two issues I have with it:
- This seems to mandate that all packages must be named by their
import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
- Mandating a forge is going to be tricky for self-hosted stuff, or
people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
By not using the macros for packages not fitting the model?
The issue is that the new Go macros are tightly wound into the forge macros. I just want to be sure that we can leverage things like the dependency generators without all the other stuff.
Hi Neal,
I should probably not let this pass without clarifying:
- if your concern is that the *forge* macros will block packaging of projects not hosted on known forges, they won't:
— they include code to disable transparently and without exploding the associated bits if the hosting site is unknown — the packager can pre-define the variables that would have been computed if the site was known and all the rest of the automation will just work — baring that the packager can ignore the site-related computations and use whatever he wants instead in Source: and *setup, which are the only parts that depend on hosting structure knowledge,
This was my main concern. There are plenty of people who don't like any of the known forges. And Go ecosystem incompetence for mandating URI-style deps and build paths doesn't really stop people from wanting to do that.
- if your concern is that the *forge* macros are defective somewhere I'd be curious where as you'd be the first to report an actual technical problem. I've used them intensively in rawhide and el7 with many different rpm tools and they are rock solid for me
I don't consider them defective. They make me a bit squeamish because I don't see the spec preamble and such because they're autogenerated, but you've mentioned that it's possible to selectively use these things with Go and forge macros.
SUSE does something similar with Python for their "singlespec" part, and it's fantastic[1][2]. But I'm slightly uncomfortable with the idea that there's no preamble *at all* in the spec using the Go macros and it's generated fully by macros.
[1]: https://github.com/openSUSE/python-rpm-macros [2]: https://build.opensuse.org/package/view_file/openSUSE:Factory/python-inipars...
- if your concern is that they won't ever be merged due to some people non-constructive obstruction, well I share it to some point but I won't spend more time arguing unicorns and what-if-maybe I-don't-like-commits-let's-make-them-awful-to-use. The people blocking there are free to take up the maintenance of my ~400 specs and redo them as they want, we'll see how they love to waste tens of minutes per spec to position setup args manually when they have to do it for hundreds of them, every time there is a package update or a forge relayout.
Technically, no one has to "accept" anything for the macros to be available. It's whether the macros would be broadly used. I could, in fact, tomorrow, make various extra useful macros that I use available in a package for people to pull in. It doesn't mean anyone has to use them.
The main difficulty with the proposal is not the *forge part, it's that automated autodeps are well, automated. As every automation they are strict and unyielding and do not take approximations well. They will fail spectacularly in the following cases:
the packager does not package all the deps of its code → the resulting package is uninstallable because autodeps add requires on missing bits
the packager packages code with garbage imports (very common in example or test code) → the resulting package is uninstallable because autodeps add require
the packager does not own properly the directories where its Go code is installed → given that Go deps depend on directories, Go autodeps also depend on proper directory ownership. Positioning autodeps on Go files themselves would make package build times increase many many times as rpm would try to compute the same autodeps for every single .go file in a directory. (and Go autodeps are already sloooow as snails because invoking the go command is slow)
All those things are mitigated by the use of %goinstall that filters usual example/test/vendor dirs so they don't trigger autodeps, and tries very hard to own all relevant directories.
For those reasons I don't propose to activate autodeps in old-style golang packages. They need conversion (and review by a human to check no problem code is deployed) first.
As long as I can do Obsoletes/Provides for the old name for the devel, unit-test, etc. stuff so the old names work, it won't bother me too much. I want people to be able to do "snapd-devel" or whatnot and get the right thing. Though I'm not sure why package name enforcement is required for autodeps, since that's based on file lists passed to a dep generator...
----- Mail original ----- De: "Neal Gompa"
- if your concern is that the *forge* macros are defective somewhere I'd be curious where as you'd be the
first to report an actual technical problem. I've used them intensively in rawhide and el7 with many different rpm tools and they are rock solid for me
I don't consider them defective. They make me a bit squeamish because I don't see the spec preamble and such because they're autogenerated, but you've mentioned that it's possible to selectively use these things with Go and forge macros.
I'm curious, what are you missing in the preamble ? As far as I can see it's all there (even though some values set to variables %gometa precomputes). I had it's right autogenerated some parts of it in the past but it's all converted to variable use now to avoid packager surprise and permit customization.
For example, what are you missing there?
%<--- %global goipath github.com/hashicorp/consul Version: 1.0.2
%gometa
%global common_description %{expand: Consul is a tool for service discovery […]}
Name: consul Release: 5%{?dist} Summary: A tool for easy service discovery, monitoring and configuration License: MPLv2.0 URL: https://www.consul.io/ Source: %{gosource} %<---
Or here ?
%<--- %global goipath github.com/hashicorp/go-sockaddr %global commit 9b4c5fa5b10a683339a270d664474b9f4aee62fc
%gometa
%global common_description %{expand: Socket address convenience functions for Go. go-sockaddr is a convenience library…}
Name: %{goname} Version: 0 Release: 0.14%{?dist} Summary: A Go convenience library to manipulate IP Addresses and UNIX sockets License: MPLv2.0 URL: %{gourl} Source: %{gosource} %<---
Regards,
On Tue, Jan 23, 2018 at 8:54 AM, nicolas.mailhot@laposte.net wrote:
----- Mail original ----- De: "Neal Gompa"
- if your concern is that the *forge* macros are defective somewhere I'd be curious where as you'd be the
first to report an actual technical problem. I've used them intensively in rawhide and el7 with many different rpm tools and they are rock solid for me
I don't consider them defective. They make me a bit squeamish because I don't see the spec preamble and such because they're autogenerated, but you've mentioned that it's possible to selectively use these things with Go and forge macros.
I'm curious, what are you missing in the preamble ? As far as I can see it's all there (even though some values set to variables %gometa precomputes). I had it's right autogenerated some parts of it in the past but it's all converted to variable use now to avoid packager surprise and permit customization.
For example, what are you missing there?
%<--- %global goipath github.com/hashicorp/consul Version: 1.0.2
%gometa
%global common_description %{expand: Consul is a tool for service discovery […]}
Name: consul Release: 5%{?dist} Summary: A tool for easy service discovery, monitoring and configuration License: MPLv2.0 URL: https://www.consul.io/ Source: %{gosource} %<---
Or here ?
%<--- %global goipath github.com/hashicorp/go-sockaddr %global commit 9b4c5fa5b10a683339a270d664474b9f4aee62fc
%gometa
%global common_description %{expand: Socket address convenience functions for Go. go-sockaddr is a convenience library…}
Name: %{goname} Version: 0 Release: 0.14%{?dist} Summary: A Go convenience library to manipulate IP Addresses and UNIX sockets License: MPLv2.0 URL: %{gourl} Source: %{gosource} %<---
Actually, this is fine with me. This is what disturbed me a bit: https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation#Pa...
----- Mail original ----- De: "Neal Gompa"
I'm curious, what are you missing in the preamble ? As far as I can see it's all there (even though some values set to variables %gometa precomputes). I had it's right autogenerated some parts of it in the past but it's all converted to variable use now to avoid packager surprise and permit customization.
Actually, this is fine with me. This is what disturbed me a bit: https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation#Pa...
Ah, yes the examples snip the lines that do not need specific changes for readability Is the … not clear enough? I fear that putting a full preamble would be even more confusing for some readers. Though if people disagree, I'l try to find some time to add the skipped lines back in
On Tue, Jan 23, 2018 at 9:40 AM, nicolas.mailhot@laposte.net wrote:
----- Mail original ----- De: "Neal Gompa"
I'm curious, what are you missing in the preamble ? As far as I can see it's all there (even though some values set to variables %gometa precomputes). I had it's right autogenerated some parts of it in the past but it's all converted to variable use now to avoid packager surprise and permit customization.
Actually, this is fine with me. This is what disturbed me a bit: https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation#Pa...
Ah, yes the examples snip the lines that do not need specific changes for readability Is the … not clear enough? I fear that putting a full preamble would be even more confusing for some readers. Though if people disagree, I'l try to find some time to add the skipped lines back in
For snipping, use "[...]" notation to indicate skipped stuff. It's hard to tell otherwise.
----- Mail original ----- De: "Neal Gompa"
For snipping, use "[...]" notation to indicate skipped stuff. It's hard to tell otherwise.
Ok, that was easy to fix :)
----- Mail original ----- De: "Neal Gompa"
As long as I can do Obsoletes/Provides for the old name for the devel, unit-test,
BTW is anyone using the unit-test packages? Right now I do not generate them, I don't need them, and making them work with autodeps would be hairy (deploying without autodeps should be trivial however)
To be honest, given all the parts current packages fail to install, I'd expect many of the current unit test packages to fail in mysterious ways, so I'm curious: what use has been found for them?
Regards,
On Tue, Jan 23, 2018 at 9:00 AM, nicolas.mailhot@laposte.net wrote:
----- Mail original ----- De: "Neal Gompa"
As long as I can do Obsoletes/Provides for the old name for the devel, unit-test,
BTW is anyone using the unit-test packages? Right now I do not generate them, I don't need them, and making them work with autodeps would be hairy (deploying without autodeps should be trivial however)
To be honest, given all the parts current packages fail to install, I'd expect many of the current unit test packages to fail in mysterious ways, so I'm curious: what use has been found for them?
No idea, but I just checked, and I don't even have the unit-test subpackage enabled on snapd, so I don't particularly care too much there...
----- Mail original ----- De: "Neal Gompa"
Hi,
Thanks for the review !
I really do like this. There are only two issues I have with it:
- This seems to mandate that all packages must be named by their
import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
Well that was not the intent, I probably need to clarify things a bit more. The *devel* package must absolutely be named after the import path (for sanity sake). The core package can be named something else (usually because it's the packaging of an app)
For example in my etcd spec
Name: etcd ← well-known app name %package -n %{goname}-devel ← go naming for go devel stuff
- Mandating a forge is going to be tricky for self-hosted stuff, or
people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
The proposal automates integration with forges, it does not mandate it
You can declare manually archivename archiveurl and forgesetup before calling the %gometa macro and you should benefit from the rest of the automation even on an hosting site it does not know anything about. The whole code tries very hard to let the packager pre-define everything it may guess wrong or not know about to avoid giving up on all the automation just because a little part is wrong.
And even after that you can still declare Source manually and pass arguments to setup the old way it that's what you want
Not calling %gometa at all will kill stuff like goname which is kind of mandatory for consistency.
Regards,
Hi,
Now the technical PR is submitted https://src.fedoraproject.org/rpms/go-srpm-macros/pull-request/1
and waiting for action from the go-srpm-macros maintainers, I took (quite a long) time to refresh and flesh out the corresponding packaging guidelines proposal. It should be fairly complete now:
https://fedoraproject.org/wiki/More_Go_packaging
I'd appreciate review to check if I have forgotten an important case, if people understand the text, if they have enhancements or corrections to propose, and so on.
Then I will push it FPC side again.
Actual practice should be fairly simple and self-explanatory, the proposal length can be scary but that's because it documents all kinds of corner cases that required digging through specs and mailing lists to find resolution examples before. The basic Go packaging skeleton will be sufficient is most cases without requiring to read any documentation.
Regards,
On Tue, Jan 30, 2018 at 10:11 AM, nicolas.mailhot@laposte.net wrote:
Hi,
Now the technical PR is submitted https://src.fedoraproject.org/rpms/go-srpm-macros/pull-request/1
and waiting for action from the go-srpm-macros maintainers, I took (quite a long) time to refresh and flesh out the corresponding packaging guidelines proposal. It should be fairly complete now:
https://fedoraproject.org/wiki/More_Go_packaging
I'd appreciate review to check if I have forgotten an important case, if people understand the text, if they have enhancements or corrections to propose, and so on.
Then I will push it FPC side again.
Actual practice should be fairly simple and self-explanatory, the proposal length can be scary but that's because it documents all kinds of corner cases that required digging through specs and mailing lists to find resolution examples before. The basic Go packaging skeleton will be sufficient is most cases without requiring to read any documentation.
The only thing I see that might be missing is autogenerating bundled(golang()) Provides when a vendor tree exists (with the appropriate automatic filters on Requires).
De: "Neal Gompa"
The only thing I see that might be missing is autogenerating bundled(golang()) Provides when a vendor tree exists (with the appropriate automatic filters on Requires).
I had though a little about doing it but first, as many Go elements, vendoring relies on conventions not standards. The nasty thing about conventions is that they are not applied 100% the same way by everyone, making automation a PITA. And second interactions with autodeps can be nasty: you can filter out provides, but do you filter out requires? What about all the junk code many projects ship as testing and examples and which is vendored with the rest?
I don't say it can't be done, or that it would be difficult to do once the rest is merged, but I'll live it to someone that absolutely want to ship a Go package with vendored parts.
Right now we sidestep the issue in our packages by rm -fr ing anything that looks like vendored code in %prep. Unbundling takes time but it has a positive cumulative effect: the more you unbundle the less you need to worry about in other packages with the same requirements. And unbundling reveals code/legal problems that it would take about as much work to solve by auditing the vendored code manually.
Regards,
Hi Nicolas,
Is there a guide for Fedora packagers about how to handle unbundling for golang packages? The draft guidelines don't seem to go into any details. I've looked at packaging a few golang packages unbundled, and have immediately run into:
A) lots of unpackaged dependencies B) dependencies that are packaged in Fedora with different, often much older versions
A) is a pretty known quantity for any type of package, but I found B) more intimidating. It seems like to package the new package, I have to get the maintainer of the library to upgrade the version, and someone needs to rebuild everything that depends on the rebuilt library and test that the rebuilt packages work.
Some tutorial showing a practical example of packaging a golang package for the first time I think would be very helpful.
Owen
On Wed, Jan 31, 2018 at 5:30 AM, nicolas.mailhot@laposte.net wrote:
De: "Neal Gompa"
The only thing I see that might be missing is autogenerating bundled(golang()) Provides when a vendor tree exists (with the appropriate automatic filters on Requires).
I had though a little about doing it but first, as many Go elements, vendoring relies on conventions not standards. The nasty thing about conventions is that they are not applied 100% the same way by everyone, making automation a PITA. And second interactions with autodeps can be nasty: you can filter out provides, but do you filter out requires? What about all the junk code many projects ship as testing and examples and which is vendored with the rest?
I don't say it can't be done, or that it would be difficult to do once the rest is merged, but I'll live it to someone that absolutely want to ship a Go package with vendored parts.
Right now we sidestep the issue in our packages by rm -fr ing anything that looks like vendored code in %prep. Unbundling takes time but it has a positive cumulative effect: the more you unbundle the less you need to worry about in other packages with the same requirements. And unbundling reveals code/legal problems that it would take about as much work to solve by auditing the vendored code manually.
Regards,
-- Nicolas mAilhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
----- Original Message -----
From: "Owen Taylor" otaylor@redhat.com To: "Development discussions related to Fedora" devel@lists.fedoraproject.org Cc: golang@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Wednesday, January 31, 2018 6:50:21 PM Subject: Re: <DKIM> [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
Hi Nicolas,
Is there a guide for Fedora packagers about how to handle unbundling for golang packages? The draft guidelines don't seem to go into any details. I've looked at packaging a few golang packages unbundled, and have immediately run into:
A) lots of unpackaged dependencies B) dependencies that are packaged in Fedora with different, often much older versions
A) is a pretty known quantity for any type of package, but I found B) more intimidating. It seems like to package the new package, I have to get the maintainer of the library to upgrade the version, and someone needs to rebuild everything that depends on the rebuilt library and test that the rebuilt packages work.
Some tutorial showing a practical example of packaging a golang package for the first time I think would be very helpful.
Owen
Unfortunately this is how the situation stand thanks to the wide spread use of vendoring and generally no release and API stability in Go projects. You have to package all not packaged deps and work with other maintainers(which mostly means with Jan Chaloupka, cc'ed) to get other already packaged deps to commonly acceptable version/commit level, although it might not be even possible thanks to API breaks between commits. So you have to stick to bundling in some cases, if you really want to package that project.
JC
On Wed, Jan 31, 2018 at 5:30 AM, < nicolas.mailhot@laposte.net > wrote:
De: "Neal Gompa"
The only thing I see that might be missing is autogenerating bundled(golang()) Provides when a vendor tree exists (with the appropriate automatic filters on Requires).
I had though a little about doing it but first, as many Go elements, vendoring relies on conventions not standards. The nasty thing about conventions is that they are not applied 100% the same way by everyone, making automation a PITA. And second interactions with autodeps can be nasty: you can filter out provides, but do you filter out requires? What about all the junk code many projects ship as testing and examples and which is vendored with the rest?
I don't say it can't be done, or that it would be difficult to do once the rest is merged, but I'll live it to someone that absolutely want to ship a Go package with vendored parts.
Right now we sidestep the issue in our packages by rm -fr ing anything that looks like vendored code in %prep. Unbundling takes time but it has a positive cumulative effect: the more you unbundle the less you need to worry about in other packages with the same requirements. And unbundling reveals code/legal problems that it would take about as much work to solve by auditing the vendored code manually.
Regards,
-- Nicolas mAilhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
----- Mail original ----- De: "Owen Taylor"
Hi Owen,
Is there a guide for Fedora packagers about how to handle unbundling for golang packages? The draft guidelines don't seem to go into any details.
I don't think there is, nor that it is necessarily needed. The posted guidelines should be sufficient technically, there are no magic I know of I didn't document, the rest is just a lot of work (but see ↓)
I've looked at packaging a few golang packages unbundled, and have immediately run into:
A) lots of unpackaged dependencies B) dependencies that are packaged in Fedora with different, often much older versions
Yes the state of Go packages in Fedora is pretty sad right now. I wouldn't expect anyone to be able to package anything but the most trivial app in unbundled mode. Too many common parts are missing, when they are not missing they are too old, trying to update an existing Go package is an exercise in frustration (too much package-specific shell code, that is difficult to understand and does not really work with new code versions), and trying to update or add missing parts just reveals more breakage and work to be done.
However, accepting to package complex Go apps in bundled mode is how Fedora got to this state in the first place. In plain speak, bundling (vendoring in Go linguo) just does not scale. You need an awful lot of manpower to audit the hundreds of other projects each app bundles, bundling removes all the package tooling that may help you to do so, and the result is not shared with any other package, or with other versions of the same package, so you get zero positive network effects. Worse, big bundled apps that do not actually try to work in unbundled mode, do not actually test the code they export (they are bundling, remember) so the result is toxic to small Go packages that try to work with them.
So bundling parts is a direct road to bundling everything, bundling everything is a direct road to bundling everything blindly, bundling everything blindly is error-prone and dangerous (because upstreams are only human and do make lots of mistakes), and pretty much removes any value Fedora can add to end users, to other parts of Fedora that would like to integrate with Go software, or to the upstream projects themselves (no Fedora QA, no stream of Fedora-originated fixes, no Fedora pressure to stabilize parts when upstream is lost in tunnel effect mode and does not realize that it is wasting everyone's time starting with its own).
Therefore, trying to get all this it a better state, requires the following steps IMHO:
1. completely refactor our Go packaging style it's less painful to update Go packages and they do not need a packager with deep knowledge of package-specific shell glue. That takes documentation, and factoring out common Go packaging functions in shared rpm code (macros and autodeps) → that's what I posted
2. using the new documentation and tooling to clean up years of Fedora technical debt, and create a new set of up-to-date Go packages that can serve as new baseline → I have hundreds of specs that I'm waiting for step 1 to complete to submit. They won't constitute a full baseline by themselves, they're not all 100% done, it's too much work to do alone but working with others requires the common macros and documentation to be merged and adopted. This step is going to be painful I'm afraid, Fedora dug itself a deep hole, leaving it is going to be hard.
3. hopefully, the result will be streamlined enough it won't be too painful to keep up to date, having an up to date baseline will help attract new Go packagers like you, everyone will benefit and be happy. We had to package some ostree bits for example to create this baseline, and I'm pretty sure ostree people within Fedora would prefer to maintain those themselves, if the rest of the Fedora go universe didn't make it too hard
B) more intimidating. It seems like to package the new package, I have to get the maintainer of the library to upgrade the version,
The guidelines and automation aim at making upgrading easy, and avoid one package or packager blocking others
and someone needs to rebuild everything that depends on the rebuilt library and test that the rebuilt packages work.
I hope that normalizing Fedora Go packages means it will be possible to automate QA tests such as "try to rebuild all the packages that depend on golang(foo) every time you see the package providing golang(foo) changing, and report what broke"
That would be expensive computing-side but a lot less expensive and long than expecting each Go packager to do it himself with his own means. And that is certainly not overkill, given how lax Go projects are about maintaining API stability.
And then in case of breakage, revert or create a compat package. That's why there is a long chapter dedicated to compat package creation in the proposed guidelines.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Thursday, February 1, 2018 11:21:59 AM Subject: Re: <DKIM> [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
----- Mail original ----- De: "Owen Taylor"
Hi Owen,
Is there a guide for Fedora packagers about how to handle unbundling for golang packages? The draft guidelines don't seem to go into any details.
I don't think there is, nor that it is necessarily needed. The posted guidelines should be sufficient technically, there are no magic I know of I didn't document, the rest is just a lot of work (but see ↓)
I've looked at packaging a few golang packages unbundled, and have immediately run into:
A) lots of unpackaged dependencies B) dependencies that are packaged in Fedora with different, often much older versions
Yes the state of Go packages in Fedora is pretty sad right now. I wouldn't expect anyone to be able to package anything but the most trivial app in unbundled mode. Too many common parts are missing, when they are not missing they are too old, trying to update an existing Go package is an exercise in frustration (too much package-specific shell code, that is difficult to understand and does not really work with new code versions), and trying to update or add missing parts just reveals more breakage and work to be done.
It depends (as everything) on available manpower, if you are willing to own your dependencies you can package anything and everything debundled.
However, accepting to package complex Go apps in bundled mode is how Fedora got to this state in the first place. In plain speak, bundling (vendoring in Go linguo) just does not scale. You need an awful lot of manpower to audit the hundreds of other projects each app bundles, bundling removes all the package tooling that may help you to do so, and the result is not shared with any other package, or with other versions of the same package, so you get zero positive network effects. Worse, big bundled apps that do not actually try to work in unbundled mode, do not actually test the code they export (they are bundling, remember) so the result is toxic to small Go packages that try to work with them.
On contrary, to this state you dislike(or seems to) Fedora got because of effort of many to de-bundle Go based projects.
So bundling parts is a direct road to bundling everything, bundling everything is a direct road to bundling everything blindly, bundling everything blindly is error-prone and dangerous (because upstreams are only human and do make lots of mistakes), and pretty much removes any value Fedora can add to end users, to other parts of Fedora that would like to integrate with Go software, or to the upstream projects themselves (no Fedora QA, no stream of Fedora-originated fixes, no Fedora pressure to stabilize parts when upstream is lost in tunnel effect mode and does not realize that it is wasting everyone's time starting with its own).
Therefore, trying to get all this it a better state, requires the following steps IMHO:
- completely refactor our Go packaging style it's less painful to update Go
packages and they do not need a packager with deep knowledge of package-specific shell glue. That takes documentation, and factoring out common Go packaging functions in shared rpm code (macros and autodeps) → that's what I posted
I don't see way how it makes it less painful(you have still the whole distro issue). It makes specs more abstract and streamlined when it works, but more opaque and hard to read when its not(and you have to debug some arcane srpm/lua macros).
- using the new documentation and tooling to clean up years of Fedora
technical debt, and create a new set of up-to-date Go packages that can serve as new baseline → I have hundreds of specs that I'm waiting for step 1 to complete to submit. They won't constitute a full baseline by themselves, they're not all 100% done, it's too much work to do alone but working with others requires the common macros and documentation to be merged and adopted. This step is going to be painful I'm afraid, Fedora dug itself a deep hole, leaving it is going to be hard.
On contrary Fedora is trying to fill the hole that upstream Go projects dug them selves in to. IMNHO Go have traded any subjectively perceived "RPM/deb hell" for even deeper levels of "Go (vendor) hell".
- hopefully, the result will be streamlined enough it won't be too painful
to keep up to date, having an up to date baseline will help attract new Go packagers like you, everyone will benefit and be happy. We had to package some ostree bits for example to create this baseline, and I'm pretty sure ostree people within Fedora would prefer to maintain those themselves, if the rest of the Fedora go universe didn't make it too hard
Pain will not be eased in any significant way as you will still have to carefully evaluate every change so you don't break any dependent package.
B) more intimidating. It seems like to package the new package, I have to get the maintainer of the library to upgrade the version,
The guidelines and automation aim at making upgrading easy, and avoid one package or packager blocking others
This is not really and automation per se. It is minor re-factoring and spec streamlining. It might or might not enable easier implementation of packaging automation.
and someone needs to rebuild everything that depends on the rebuilt library and test that the rebuilt packages work.
I hope that normalizing Fedora Go packages means it will be possible to automate QA tests such as "try to rebuild all the packages that depend on golang(foo) every time you see the package providing golang(foo) changing, and report what broke"
That would be expensive computing-side but a lot less expensive and long than expecting each Go packager to do it himself with his own means. And that is certainly not overkill, given how lax Go projects are about maintaining API stability.
Until upstreams will be working in true CI/CD manners(when they do not make any releases). Fedora can't really fix this. Currently AFAIK we have GoFed and accompanying projects that provide you with possibility to gather such data.
And then in case of breakage, revert or create a compat package. That's why there is a long chapter dedicated to compat package creation in the proposed guidelines.
Compat packages do not scale. Imagine that you will have in worst case maintain 10s of different versions of same package for every release, doing security backports, etc....
JC
Regards,
-- Nicolas Mailhot _______________________________________________ golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
De: "Jakub Cajka"
Hi Jakub,
It depends (as everything) on available manpower, if you are willing to own your dependencies you can package anything and everything debundled.
Sure, but available manpower depends on how high the bar you put for people wanting to join, and right now this bar is pretty high.
On contrary, to this state you dislike(or seems to) Fedora got because of effort of many to de bundle Go based projects.
I'm *not* disparaging the past efforts of many. They did a lot of good. Unfortunately, the result still falls short of the mark. The baseline is not complete enough, the baseline is old and aging fast, would-be Go packagers like Owen are ready to give up when they start getting an idea of the Go state in Fedora.
It's time to invest in tooling and simplification so the few Go Fedora packagers there is, can do more with less, so casual Go packagers can contribute their little bit instead of being repulsed by complexity that can be avoided and factored out, so people already familiar with Fedora packaging can package the Go bits they need without needing a lengthy formation on Go specifics.
There *are* lots of people that would like to do Go packaging. Many Go software projects currently make headlines. That there are so few Go packagers in Fedora, at the time Go is in the limelight, is a pretty good indicator we are making it too hard.
I don't see way how it makes it less painful(you have still the whole distro issue). It makes specs more abstract and streamlined when it works, but more opaque and hard to read when its not(and you have to debug some arcane srpm/lua macros).
It makes it less painful because there is no package-specific magic in each and every Go spec file.
You do not need to debug code in each spec because code that already just works for hundreds of packages is unlikely to not work on the next spec file.
You do not need to comb each and every line of a Go spec file wondering if the line is cut and pasting of a common template (multiply by all past revisions of common templates) or if it is package-specific. The common parts are in common macros not spread right and left.
You do not need to worry wether common code is complete or not because factoring out common code removes the temptation to document 90% of what's needed, leaving others to fill in the missing 10%. 10% amounts to an astonishing level of work when you multiply it by the number of Go packagers and the number of Go packages we would ideally need.
Plus, when you do find a bug in common code, the fix is shared with the whole distribution, not hidden in a single spec file.
That's what successful factoring of common code earns you.
And given I have more than 470 Go spec files that just work with the proposed macros with no special package-specific adjustment needed, and most time I compare one of those to the Fedora equivalent I'm executing more unit tests and building more binaries, with drastically shorter spec code, I'm pretty sure the factoring out *is* successful.
This step is going to be painful I'm afraid, Fedora dug itself a deep hole, leaving it is going to be hard.
On contrary Fedora is trying to fill the hole that upstream Go projects dug them selves in to.
Filling upstream holes is pretty much the definition of an integrator/distributor role. In Go they are particularly numerous and deep, but Fedora users do want their docker and kubernetes (and Kubernetes, BTW, is astonishingly free of the problems that plague many Go projects, proving it *is* possible to do good release engineering in Go).
Fedora did dug itself a hole by trying the bundling shortcut around EL7 time. Where are the EL7 Go packages? Why are Fedora Atomic people not spearheading Go packaging in Fedora? Waiting for bundling to be solved upstream-side made Fedora worse not better.
IMNHO Go have traded any subjectively perceived "RPM/deb hell" for even deeper levels of "Go vendor) hell".
I fully agree. What I meant to convey is that replicating upstream vendoring Fedora-side only makes the situation worse for users, for upstreams, and for Fedora.
Pain will not be eased in any significant way as you will still have to carefully evaluate every change so you don't break any dependent package.
Pain *will* be eased in a significant way because we will have the tooling to detect breakage automatically, and because fixes once they're done will be packaged and shared.
The guidelines and automation aim at making upgrading easy, and avoid one package or packager blocking others
This is not really and automation per se. It is minor re-factoring and spec streamlining. It might or might not enable easier implementation of packaging automation.
If it's minor let's do it. I'm quite sure you will be pleasantly surprised by the level of pain those minor changes remove.
That would be expensive computing-side but a lot less expensive and long than expecting each Go packager to do it himself with his own means. And that is certainly not overkill, given how lax Go projects are about maintaining API stability.
Until upstreams will be working in true CI/CD manners(when they do not make any releases). Fedora can't really fix this.
Fedora can detect the breakage which is the first step to fix it. It does not take an army of Go experts, just unbundled packages that can be rebuilt automatically to identify incompatibilities. Go experts are better employed working on fixes than doing rebuild checks that can be automated.
Currently AFAIK we have GoFed and accompanying projects that provide you with possibility to gather such data.
And nothing stops you for continuing to use those. Except, the golden test will always be to actually try to build the code, which is something that can be automated reliably once the dependencies exported to Fedora tools are themselves complete and reliable, and Go packages are exported as rpm packages, not hidden in private vendor trees.
And then in case of breakage, revert or create a compat package. That's why there is a long chapter dedicated to compat package creation in the proposed guidelines.
Compat packages do not scale.
They scale a lot better than bundling, when an API break results in dozens of projects choosing to vendor a pre-break commit. With each of them choosing a slightly different state, resulting in ten times the amount of code to audit and maintain.
Imagine that you will have in worst case maintain 10s of different versions of same package for every release, doing security backports, etc....
When you do not unbundle you are still shipping the vendored code, you are still responsible for its bugs (security or not). Compat packages do not augment the code surface you have to care about, on the contrary they offer you the possibility to *reduce* the number of code states in the distribution.
Plus, they are the occasion, to notify upstreams they should move to a more recent code state. And reduce the number of code states you have to care about yet more. Not all upstreams are dead set against updating their vendoring. A lot of them are nice and responsible and just didn't see they had an API break to manage yet because their use of vendoring hid the problem.
Of course that supposes that creating a compat package does not add a significant package-creation burden. Which I *did* try to avoid. Organization-wise Go will still need a form of exemption to create many many packages easily, because Go code is spread over many many projects, and not consolidated in a few big libraries like in the C world.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Thursday, February 1, 2018 2:51:13 PM Subject: Re: <DKIM> [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
De: "Jakub Cajka"
Hi Jakub,
It depends (as everything) on available manpower, if you are willing to own your dependencies you can package anything and everything debundled.
Sure, but available manpower depends on how high the bar you put for people wanting to join, and right now this bar is pretty high.
On contrary, to this state you dislike(or seems to) Fedora got because of effort of many to de bundle Go based projects.
I'm *not* disparaging the past efforts of many. They did a lot of good. Unfortunately, the result still falls short of the mark. The baseline is not complete enough, the baseline is old and aging fast, would-be Go packagers like Owen are ready to give up when they start getting an idea of the Go state in Fedora.
It's time to invest in tooling and simplification so the few Go Fedora packagers there is, can do more with less, so casual Go packagers can contribute their little bit instead of being repulsed by complexity that can be avoided and factored out, so people already familiar with Fedora packaging can package the Go bits they need without needing a lengthy formation on Go specifics.
There *are* lots of people that would like to do Go packaging. Many Go software projects currently make headlines. That there are so few Go packagers in Fedora, at the time Go is in the limelight, is a pretty good indicator we are making it too hard.
I don't see way how it makes it less painful(you have still the whole distro issue). It makes specs more abstract and streamlined when it works, but more opaque and hard to read when its not(and you have to debug some arcane srpm/lua macros).
It makes it less painful because there is no package-specific magic in each and every Go spec file.
You do not need to debug code in each spec because code that already just works for hundreds of packages is unlikely to not work on the next spec file.
You do not need to comb each and every line of a Go spec file wondering if the line is cut and pasting of a common template (multiply by all past revisions of common templates) or if it is package-specific. The common parts are in common macros not spread right and left.
You do not need to worry wether common code is complete or not because factoring out common code removes the temptation to document 90% of what's needed, leaving others to fill in the missing 10%. 10% amounts to an astonishing level of work when you multiply it by the number of Go packagers and the number of Go packages we would ideally need.
Plus, when you do find a bug in common code, the fix is shared with the whole distribution, not hidden in a single spec file.
That's what successful factoring of common code earns you.
And given I have more than 470 Go spec files that just work with the proposed macros with no special package-specific adjustment needed, and most time I compare one of those to the Fedora equivalent I'm executing more unit tests and building more binaries, with drastically shorter spec code, I'm pretty sure the factoring out *is* successful.
This step is going to be painful I'm afraid, Fedora dug itself a deep hole, leaving it is going to be hard.
On contrary Fedora is trying to fill the hole that upstream Go projects dug them selves in to.
Filling upstream holes is pretty much the definition of an integrator/distributor role. In Go they are particularly numerous and deep, but Fedora users do want their docker and kubernetes (and Kubernetes, BTW, is astonishingly free of the problems that plague many Go projects, proving it *is* possible to do good release engineering in Go).
??
Fedora did dug itself a hole by trying the bundling shortcut around EL7 time. Where are the EL7 Go packages? Why are Fedora Atomic people not spearheading Go packaging in Fedora? Waiting for bundling to be solved upstream-side made Fedora worse not better.
I'm not sure that Fedora. Go ask them don't ask me or other regular Fedora maintainers. Fedora haven't been bundling and has been doing as much as possible to un-bundle. Fedora can't solve others non related problems.
IMNHO Go have traded any subjectively perceived "RPM/deb hell" for even deeper levels of "Go vendor) hell".
I fully agree. What I meant to convey is that replicating upstream vendoring Fedora-side only makes the situation worse for users, for upstreams, and for Fedora.
Pain will not be eased in any significant way as you will still have to carefully evaluate every change so you don't break any dependent package.
Pain *will* be eased in a significant way because we will have the tooling to detect breakage automatically, and because fixes once they're done will be packaged and shared.
Your proposal doesn't provide this kind of tooling, if I'm not mistaken.
The guidelines and automation aim at making upgrading easy, and avoid one package or packager blocking others
This is not really and automation per se. It is minor re-factoring and spec streamlining. It might or might not enable easier implementation of packaging automation.
If it's minor let's do it. I'm quite sure you will be pleasantly surprised by the level of pain those minor changes remove.
?? I'm in no way opposing your proposal per se.
That would be expensive computing-side but a lot less expensive and long than expecting each Go packager to do it himself with his own means. And that is certainly not overkill, given how lax Go projects are about maintaining API stability.
Until upstreams will be working in true CI/CD manners(when they do not make any releases). Fedora can't really fix this.
Fedora can detect the breakage which is the first step to fix it. It does not take an army of Go experts, just unbundled packages that can be rebuilt automatically to identify incompatibilities. Go experts are better employed working on fixes than doing rebuild checks that can be automated.
Rebuilds could be automated, analysis of them not easily.
Currently AFAIK we have GoFed and accompanying projects that provide you with possibility to gather such data.
And nothing stops you for continuing to use those. Except, the golden test will always be to actually try to build the code, which is something that can be automated reliably once the dependencies exported to Fedora tools are themselves complete and reliable, and Go packages are exported as rpm packages, not hidden in private vendor trees.
?? I bit don't understand you. If you can, please help Jan to improve them.
And then in case of breakage, revert or create a compat package. That's why there is a long chapter dedicated to compat package creation in the proposed guidelines.
Compat packages do not scale.
They scale a lot better than bundling, when an API break results in dozens of projects choosing to vendor a pre-break commit. With each of them choosing a slightly different state, resulting in ten times the amount of code to audit and maintain.
Imagine that you will have in worst case maintain 10s of different versions of same package for every release, doing security backports, etc....
When you do not unbundle you are still shipping the vendored code, you are still responsible for its bugs (security or not). Compat packages do not augment the code surface you have to care about, on the contrary they offer you the possibility to *reduce* the number of code states in the distribution.
Please don't put words in to my mouth. I haven't been talking about not un-bundling, just compat packages.
JC
Plus, they are the occasion, to notify upstreams they should move to a more recent code state. And reduce the number of code states you have to care about yet more. Not all upstreams are dead set against updating their vendoring. A lot of them are nice and responsible and just didn't see they had an API break to manage yet because their use of vendoring hid the problem.
Of course that supposes that creating a compat package does not add a significant package-creation burden. Which I *did* try to avoid. Organization-wise Go will still need a form of exemption to create many many packages easily, because Go code is spread over many many projects, and not consolidated in a few big libraries like in the C world.
Regards,
-- Nicolas Mailhot _______________________________________________ golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
De: "Jakub Cajka"
Filling upstream holes is pretty much the definition of an integrator/distributor role. In Go they are particularly numerous and deep, but Fedora users do want their docker and kubernetes (and Kubernetes, BTW, is astonishingly free of the problems that plague many Go projects, proving it *is* possible to do good release engineering in Go).
??
Just sending some kuddos Kubernetes-side. All the Kubernetes parts we had to work with so far were astonishingly free of API breakage and dependency cycles because Kubernetes people chose their deps carefully and do version their code states. It was a vacation from packaging other Go code.
Pain will not be eased in any significant way as you will still have to carefully evaluate every change so you don't break any dependent package.
Pain *will* be eased in a significant way because we will have the tooling to detect breakage automatically, and because fixes once they're done will be packaged and shared.
Your proposal doesn't provide this kind of tooling, if I'm not mistaken.
Not directly. It does provide the means to easily rev a spec to a new code state (version tag or commit), and it makes deps systematic (so Fedora tooling can accurately detect what is likely to be impacted by a change). Together, those open the way to use generic Fedora testing tools to automate build tests.
If it's minor let's do it. I'm quite sure you will be pleasantly surprised by the level of pain those minor changes remove.
?? I'm in no way opposing your proposal per se.
Thank you
Fedora can detect the breakage which is the first step to fix it. It does not take an army of Go experts, just unbundled packages that can be rebuilt automatically to identify incompatibilities. Go experts are better employed working on fixes than doing rebuild checks that can be automated.
Rebuilds could be automated, analysis of them not easily.
Sure, but at that point 1. you know if the new API is compatible or not with other packages 2. you've saved human packagers the time and effort to do the same rebuild check manually
So that's still a net win, no ?
And nothing stops you for continuing to use those. Except, the golden test will always be to actually try to build the code, which is something that can be automated reliably once the dependencies exported to Fedora tools are themselves complete and reliable, and Go packages are exported as rpm packages, not hidden in private vendor trees.
?? I bit don't understand you. If you can, please help Jan to improve them.
What I meant is that: 1. there are many ways to try to divine if a code state will build, but the final arbiter is the Go compiler 2. to use Fedora-wide QA tools, code objects need to be exported as packages. Every missing test dependency, which is not packaged as a separate Go rpm, is something Fedora QA won't be able to run.
When you do not unbundle you are still shipping the vendored code, you are still responsible for its bugs (security or not). Compat packages do not augment the code surface you have to care about, on the contrary they offer you the possibility to *reduce* the number of code states in the distribution.
Please don't put words in to my mouth. I haven't been talking about not un-bundling, just compat packages.
Sorry if I misunderstood you:(
However, I do think compat packages are a natural effect of unbundling. One does need something to fill in during the time upstream or Fedora work on the effects of an API break. Other Fedora language ecosystems do not manage to avoid them. You can strive to limit compat packages to the minimum amount, and the minimum period, but you can not fully avoid them, once an ecosystem has grown enough it is too big to be perfectly coordinated all the time.
So making compat creation easy and fast, is required to manage a huge amount of interdependent packages, which are not too careful about API stability. Even if one has little love for compat packages as such.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Thursday, February 1, 2018 4:24:52 PM Subject: Re: <DKIM> [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
De: "Jakub Cajka"
Filling upstream holes is pretty much the definition of an integrator/distributor role. In Go they are particularly numerous and deep, but Fedora users do want their docker and kubernetes (and Kubernetes, BTW, is astonishingly free of the problems that plague many Go projects, proving it *is* possible to do good release engineering in Go).
??
Just sending some kuddos Kubernetes-side. All the Kubernetes parts we had to work with so far were astonishingly free of API breakage and dependency cycles because Kubernetes people chose their deps carefully and do version their code states. It was a vacation from packaging other Go code.
Yes kubernetes is kind of better than most projects, but IMHO still uses lots of not so good practices, although getting bit better over the years.
Pain will not be eased in any significant way as you will still have to carefully evaluate every change so you don't break any dependent package.
Pain *will* be eased in a significant way because we will have the tooling to detect breakage automatically, and because fixes once they're done will be packaged and shared.
Your proposal doesn't provide this kind of tooling, if I'm not mistaken.
Not directly. It does provide the means to easily rev a spec to a new code state (version tag or commit), and it makes deps systematic (so Fedora tooling can accurately detect what is likely to be impacted by a change). Together, those open the way to use generic Fedora testing tools to automate build tests.
It makes it bit easier in some cases, but I wouldn't call it and revolution(as you are trying to sell it) rather than evolution(and omission of some currently mentioned parts), building on top of current practice.
If it's minor let's do it. I'm quite sure you will be pleasantly surprised by the level of pain those minor changes remove.
?? I'm in no way opposing your proposal per se.
Thank you
Fedora can detect the breakage which is the first step to fix it. It does not take an army of Go experts, just unbundled packages that can be rebuilt automatically to identify incompatibilities. Go experts are better employed working on fixes than doing rebuild checks that can be automated.
Rebuilds could be automated, analysis of them not easily.
Sure, but at that point
- you know if the new API is compatible or not with other packages
- you've saved human packagers the time and effort to do the same rebuild
check manually
So that's still a net win, no ?
No really, or not much, until you determine why the builds have failed(I can rebuild of whole distro each day, still I don't have enough spare time to determine and fix all FTBFS issues). You still have to have someone who would analyze those failures.
And nothing stops you for continuing to use those. Except, the golden test will always be to actually try to build the code, which is something that can be automated reliably once the dependencies exported to Fedora tools are themselves complete and reliable, and Go packages are exported as rpm packages, not hidden in private vendor trees.
?? I bit don't understand you. If you can, please help Jan to improve them.
What I meant is that:
- there are many ways to try to divine if a code state will build, but the
final arbiter is the Go compiler 2. to use Fedora-wide QA tools, code objects need to be exported as packages. Every missing test dependency, which is not packaged as a separate Go rpm, is something Fedora QA won't be able to run.
I think that I'm still missing your point. Of course you need to package all your dependencies.
When you do not unbundle you are still shipping the vendored code, you are still responsible for its bugs (security or not). Compat packages do not augment the code surface you have to care about, on the contrary they offer you the possibility to *reduce* the number of code states in the distribution.
Please don't put words in to my mouth. I haven't been talking about not un-bundling, just compat packages.
Sorry if I misunderstood you:(
However, I do think compat packages are a natural effect of unbundling. One does need something to fill in during the time upstream or Fedora work on the effects of an API break. Other Fedora language ecosystems do not manage to avoid them. You can strive to limit compat packages to the minimum amount, and the minimum period, but you can not fully avoid them, once an ecosystem has grown enough it is too big to be perfectly coordinated all the time.
So making compat creation easy and fast, is required to manage a huge amount of interdependent packages, which are not too careful about API stability. Even if one has little love for compat packages as such.
Regards,
-- Nicolas Mailhot _______________________________________________ golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
I'm strongly against general unrestricted practice of compat packages as proposed. If you need compat package you need to work with usptreams on stabilizing the API/project, fork it, or just use COPR as your projects(or its dependencies) are not yet mature enough for Fedora.
JC
De: "Jakub Cajka"
Hi Jakub
I'm strongly against general unrestricted practice of compat packages as proposed. If you need compat package you need to work with usptreams on stabilizing the API/project, fork it, or just use COPR as your projects(or its dependencies) are not yet mature enough for Fedora.
I appreciate the sentiment, and I quite hate compat packages, but the restrictions you want did not work in practice for Fedora.
Making compat package creation hard does not result in faster fixing to use new code. What it actually does is to block the packaging of all the projects that need a more recent package state, because packagers that need the old state for one of their packages, will drag their feet on updating common dependencies till the code of their packages is fixed upstream, or they have the time to fix it themselves.
And blocking packaging of new part means you do not get the new packagers that would join because they're interested in the new parts, and would contribute to the maintenance of common deps (not all of which need compat-ing), and would relieve part of the pressure on the existing packagers, that prevents them from working as much a they'd like to on updating their packages. It's a death spiral. It results in a massively obsolete Go package baselines, full of holes, because all the energy is poured in making existing stuff work, at the expense of onboarding new packages and packagers.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Saturday, February 3, 2018 4:27:36 PM Subject: Re: <DKIM> [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
De: "Jakub Cajka"
Hi Jakub
I'm strongly against general unrestricted practice of compat packages as proposed. If you need compat package you need to work with usptreams on stabilizing the API/project, fork it, or just use COPR as your projects(or its dependencies) are not yet mature enough for Fedora.
I appreciate the sentiment, and I quite hate compat packages, but the restrictions you want did not work in practice for Fedora.
Making compat package creation hard does not result in faster fixing to use new code. What it actually does is to block the packaging of all the projects that need a more recent package state, because packagers that need the old state for one of their packages, will drag their feet on updating common dependencies till the code of their packages is fixed upstream, or they have the time to fix it themselves.
I think one of the main responsibilities of Fedora packager is to work with upstreams, help them mature and generally improve their projects.
And blocking packaging of new part means you do not get the new packagers that would join because they're interested in the new parts, and would contribute to the maintenance of common deps (not all of which need compat-ing), and would relieve part of the pressure on the existing packagers, that prevents them from working as much a they'd like to on updating their packages. It's a death spiral. It results in a massively obsolete Go package baselines, full of holes, because all the energy is poured in making existing stuff work, at the expense of onboarding new packages and packagers.
Regards,
-- Nicolas Mailhot _______________________________________________ golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
Do you have any evidence supporting this claim? From my point of view Jan and other packages has been spending lot of time on on boarding packagers and working on tooling. From my perspective distribution will end up with crazy amount of bitrotting, backport, constant attention requiring package crud..., IMHO basically same as now, apart of it we will have few 1000s of Go based packages with compat names nobody cares about, instead of current 500 or so packages.
JC
----- Mail original ----- De: "Jakub Cajka"
I think one of the main responsibilities of Fedora packager is to work with upstreams, help them mature and generally improve their projects.
Sure but expecting everything to be perfect and consistent before shipping anything just DOES NOT WORK. Reality is not black and white, it's messy with shades of gray
Even the C/C++ guys do not manage to ship a compat-free package ecosystem, and their projects had a few more decades than Go projects to mature, and rpm was pretty much built around C/C++ expectations.
Compat packages are a fact of life in any large software ecosystem, where you can't achieve perfect cohesion. Go is getting *very* large. It needs compat packages. That does not mean there will be hundreds of them because, creating packages is *work*, that does not need they will need maintaining for years, because the aim of each compat package is to get obsoleted as fast as possible, but there will always be some of them at any given time. We are not building a cathedral. Bazaars are messy when full of life.
Do you have any evidence supporting this claim? From my point of view Jan and other packages has been spending lot of time on on boarding packagers and working on tooling.
No one (and certainly not me) is accusing you or Jan not to spend a crazy amount of time and energy trying to make things work. But you did not achieve a satisfactory Go state in Fedora, read again what Owen wrote, I didn't put any word in his mouth, even though I could have written pretty much the same message a few months ago.
Again, no one (and certainly not me) disputes your level of dedication to the "cause". You just made some choices in the past (trying to avoid working within rpm via godep, refusing to include different states of the same Go code in the distro when major Go apps *disagree* on the correct state to include) that didn't work out in practice, with the hindsight of several years of Fedora Go packaging and the Go package state achieved in Fedora after those years.
It's time to adjust those choices a bit.
From my perspective distribution will end up with crazy amount of bitrotting, backport, constant attention requiring package crud..., IMHO basically same as now, apart of it we will have few 1000s of Go based packages with compat names nobody cares about, instead of current 500 or so packages.
Fedora has lots of Go packages no one cares about because they do not permit building the apps people *do* care about.
Building the apps people care about requires a more aggressive update policy, and accepting people will work in parallel on apps, that demand different code states, of common dependencies. And there are not so many of those, I count 18 of them in our repo out of 476 Go packages, even though the work is not completely finished, and finalizing the build of complex tip apps is almost certain to increase the proportion a little. That's awfully *good* and *nice* given the "no Go API compatibility" scarecrows people like to invoke.
You won't *ever* attract a large pool of packagers, to work on a package baseline, that will eventually in some years permit building apps, but never this year, because upstream state is not conflict and compat-free yet.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Monday, February 5, 2018 12:16:14 PM Subject: [Fedora-packaging] Re: <DKIM> Re: Proposed Fedora packaging guideline: More Go packaging
----- Mail original ----- De: "Jakub Cajka"
I think one of the main responsibilities of Fedora packager is to work with upstreams, help them mature and generally improve their projects.
Sure but expecting everything to be perfect and consistent before shipping anything just DOES NOT WORK. Reality is not black and white, it's messy with shades of gray
Even the C/C++ guys do not manage to ship a compat-free package ecosystem, and their projects had a few more decades than Go projects to mature, and rpm was pretty much built around C/C++ expectations.
Compat packages are a fact of life in any large software ecosystem, where you can't achieve perfect cohesion. Go is getting *very* large. It needs compat packages. That does not mean there will be hundreds of them because, creating packages is *work*, that does not need they will need maintaining for years, because the aim of each compat package is to get obsoleted as fast as possible, but there will always be some of them at any given time. We are not building a cathedral. Bazaars are messy when full of life.
Do you have any evidence supporting this claim? From my point of view Jan and other packages has been spending lot of time on on boarding packagers and working on tooling.
No one (and certainly not me) is accusing you or Jan not to spend a crazy amount of time and energy trying to make things work. But you did not achieve a satisfactory Go state in Fedora, read again what Owen wrote, I didn't put any word in his mouth, even though I could have written pretty much the same message a few months ago.
Again, no one (and certainly not me) disputes your level of dedication to the "cause". You just made some choices in the past (trying to avoid working within rpm via godep, refusing to include different states of the same Go code in the distro when major Go apps *disagree* on the correct state to include) that didn't work out in practice, with the hindsight of several years of Fedora Go packaging and the Go package state achieved in Fedora after those years.
It's time to adjust those choices a bit.
From my perspective distribution will end up with crazy amount of bitrotting, backport, constant attention requiring package crud..., IMHO basically same as now, apart of it we will have few 1000s of Go based packages with compat names nobody cares about, instead of current 500 or so packages.
Fedora has lots of Go packages no one cares about because they do not permit building the apps people *do* care about.
Building the apps people care about requires a more aggressive update policy, and accepting people will work in parallel on apps, that demand different code states, of common dependencies. And there are not so many of those, I count 18 of them in our repo out of 476 Go packages, even though the work is not completely finished, and finalizing the build of complex tip apps is almost certain to increase the proportion a little. That's awfully *good* and *nice* given the "no Go API compatibility" scarecrows people like to invoke.
You won't *ever* attract a large pool of packagers, to work on a package baseline, that will eventually in some years permit building apps, but never this year, because upstream state is not conflict and compat-free yet.
Regards,
-- Nicolas Mailhot _______________________________________________ packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org
Our as Fedora or yours company/org? I believe that your contribution of those in to Fedora will be much appreciated. But IMHO even your changes will not change this, if you don't have few dedicated packager around to do all the bidding.
JC
----- Mail original ----- De: "Jakub Cajka"
Our as Fedora or yours company/org? I believe that your contribution of those in to Fedora will be much appreciated.
Our was meaning the set of specs we are preparing for inclusion. Can't really share them before the macros they depend on are merged in rawhide.
You can grab most of those from https://copr.fedorainfracloud.org/coprs/nim/More_Go_Packaging
before I nuke it to launch a new build from scratch (if seems a bit of grpc bootstraping failed this run so that's back to build log analysis, add constrains, scratch, retry, wait 3 days for the result. I soooo hate the yum version in copr, every time there is a dumb decision to make it will make it, it manages to make yum in EL7 look smart)
But IMHO even your changes will not change this, if you don't have few dedicated packager around to do all the bidding.
Sure, the changes will only remove some barriers to new Go packagers, they can't replace those packagers, or the people who take care of the baseline core.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: "Development discussions related to Fedora" devel@lists.fedoraproject.org Cc: golang@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Monday, February 5, 2018 3:27:01 PM Subject: Re: [Fedora-packaging] Re: <DKIM> Re: Proposed Fedora packaging guideline: More Go packaging
----- Mail original ----- De: "Jakub Cajka"
Our as Fedora or yours company/org? I believe that your contribution of those in to Fedora will be much appreciated.
Our was meaning the set of specs we are preparing for inclusion. Can't really share them before the macros they depend on are merged in rawhide.
You can grab most of those from https://copr.fedorainfracloud.org/coprs/nim/More_Go_Packaging
before I nuke it to launch a new build from scratch (if seems a bit of grpc bootstraping failed this run so that's back to build log analysis, add constrains, scratch, retry, wait 3 days for the result. I soooo hate the yum version in copr, every time there is a dumb decision to make it will make it, it manages to make yum in EL7 look smart)
You have to explicit about the specs, TBH I have had more fun with dnf, when it fist came out.
I believe that you can already open pull requests and review requests stating the dependency(so it doesn't get merger too soon) as it will take it some time to merge and details can get ironed out during merging/review. As IMHO your guidelines will require whole distro change/rebuild to make it compliant. I feel like you haven't been engaging with current maintainers much.
JC
But IMHO even your changes will not change this, if you don't have few dedicated packager around to do all the bidding.
Sure, the changes will only remove some barriers to new Go packagers, they can't replace those packagers, or the people who take care of the baseline core.
Regards,
-- Nicolas Mailhot _______________________________________________ golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
Hi Nicolas,
I'm embarrassed to admit that before I sent my mail I carefully read over ... the old PackageDrafts/Go :-( My only excuse is that it was in my browser history.
Having actually read the relevant parts of "More Go Packaging", the explanation of compat packages and notification procedures does make the intended operation clearer, though the social and technical barrier to a packager new to Fedora will still be high if packaging their target package requires creating a compat package and fixing multiple other packages. As you say, once Fedora packaging catches up to the modern state, things will be better, because the onus won't be on the new packager to do the work of updating old libraries.
I still worry that Fedora is not big enough to move the status-quo in the Go world - to get the point where Go programs require github.com/foo/bar >= 1.2.3 and actually have been tested with a multiple versions in that range, not exactly the one vendored version shipped with the program. Sp we might end up with lots of Fedora-specific bugs that the upstream doesn't care about. But if the Fedora Go community can get some size and momentum and start pushing fixes upstream, that will certainly be a positive force for reducing Go programs stuck on random old versions of dependencies!
I haven't had time to read through the entire proposal, but it certainly looks like a major step forward!
Owen
On Thu, Feb 1, 2018 at 5:21 AM, nicolas.mailhot@laposte.net wrote:
----- Mail original ----- De: "Owen Taylor"
Hi Owen,
Is there a guide for Fedora packagers about how to handle unbundling for golang packages? The draft guidelines don't seem to go into any details.
I don't think there is, nor that it is necessarily needed. The posted guidelines should be sufficient technically, there are no magic I know of I didn't document, the rest is just a lot of work (but see ↓)
I've looked at packaging a few golang packages unbundled, and have immediately run into:
A) lots of unpackaged dependencies B) dependencies that are packaged in Fedora with different, often much older versions
Yes the state of Go packages in Fedora is pretty sad right now. I wouldn't expect anyone to be able to package anything but the most trivial app in unbundled mode. Too many common parts are missing, when they are not missing they are too old, trying to update an existing Go package is an exercise in frustration (too much package-specific shell code, that is difficult to understand and does not really work with new code versions), and trying to update or add missing parts just reveals more breakage and work to be done.
However, accepting to package complex Go apps in bundled mode is how Fedora got to this state in the first place. In plain speak, bundling (vendoring in Go linguo) just does not scale. You need an awful lot of manpower to audit the hundreds of other projects each app bundles, bundling removes all the package tooling that may help you to do so, and the result is not shared with any other package, or with other versions of the same package, so you get zero positive network effects. Worse, big bundled apps that do not actually try to work in unbundled mode, do not actually test the code they export (they are bundling, remember) so the result is toxic to small Go packages that try to work with them.
So bundling parts is a direct road to bundling everything, bundling everything is a direct road to bundling everything blindly, bundling everything blindly is error-prone and dangerous (because upstreams are only human and do make lots of mistakes), and pretty much removes any value Fedora can add to end users, to other parts of Fedora that would like to integrate with Go software, or to the upstream projects themselves (no Fedora QA, no stream of Fedora-originated fixes, no Fedora pressure to stabilize parts when upstream is lost in tunnel effect mode and does not realize that it is wasting everyone's time starting with its own).
Therefore, trying to get all this it a better state, requires the following steps IMHO:
- completely refactor our Go packaging style it's less painful to update
Go packages and they do not need a packager with deep knowledge of package-specific shell glue. That takes documentation, and factoring out common Go packaging functions in shared rpm code (macros and autodeps) → that's what I posted
- using the new documentation and tooling to clean up years of Fedora
technical debt, and create a new set of up-to-date Go packages that can serve as new baseline → I have hundreds of specs that I'm waiting for step 1 to complete to submit. They won't constitute a full baseline by themselves, they're not all 100% done, it's too much work to do alone but working with others requires the common macros and documentation to be merged and adopted. This step is going to be painful I'm afraid, Fedora dug itself a deep hole, leaving it is going to be hard.
- hopefully, the result will be streamlined enough it won't be too
painful to keep up to date, having an up to date baseline will help attract new Go packagers like you, everyone will benefit and be happy. We had to package some ostree bits for example to create this baseline, and I'm pretty sure ostree people within Fedora would prefer to maintain those themselves, if the rest of the Fedora go universe didn't make it too hard
B) more intimidating. It seems like to package the new package, I have to get the maintainer of the library to upgrade the version,
The guidelines and automation aim at making upgrading easy, and avoid one package or packager blocking others
and someone needs to rebuild everything that depends on the rebuilt library and test that the rebuilt packages work.
I hope that normalizing Fedora Go packages means it will be possible to automate QA tests such as "try to rebuild all the packages that depend on golang(foo) every time you see the package providing golang(foo) changing, and report what broke"
That would be expensive computing-side but a lot less expensive and long than expecting each Go packager to do it himself with his own means. And that is certainly not overkill, given how lax Go projects are about maintaining API stability.
And then in case of breakage, revert or create a compat package. That's why there is a long chapter dedicated to compat package creation in the proposed guidelines.
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
De: "Owen Taylor"
I'm embarrassed to admit that before I sent my mail I carefully read over ... the old PackageDrafts/Go :-( My only excuse is that it was in my browser history.
NP, that gave you some context on where Fedora is today.
Having actually read the relevant parts of "More Go Packaging", the explanation of compat packages and notification procedures does make the intended operation clearer,
Thank you. Do not hesitate to improve the wording in the wiki, or post suggestions here, if you have ideas on how to make it clearer, simpler or more efficient
though the social and technical barrier to a packager new to Fedora will still be high if packaging their target package requires creating a compat package and fixing multiple other packages.
Unfortunately, packaging something in Go is still going to be harder than packaging software in another language in the near term :( I'd love to find more magic bullets.
I still worry that Fedora is not big enough to move the status-quo in the Go world - to get the point where Go programs require github.com/foo/bar >= .2.3 and actually have been tested with a multiple versions in that range, not exactly the one vendored version shipped with the program.
That's a legitimate worry, yes. However given container and cloud people are massively adopting Go, that critical cloud software is now mostly written in Go, I don't think Fedora can afford to pass on Go and still stay relevant server-side. That's even more true for Fedora downstreams and Fedora's main sponsor.
So I guess it all boils down to strategic choices for Fedora and Red Hat: invest in Go packaging, even if it *is* painful, or pass, and lose relevance server-side in the near future. Red Hat certainly has the assets, with Openshift and Coreos, to influence the Go world in a Fedora-positive way. I don't know if it will choose, or manage, to leverage them. It can not fail to notice the many Go projects that propose their software as Ubuntu Docker images today. It can not fail to notice that Jakub and Jan, with all their qualities, are a tad overworked and not really sufficient to pull Fedora Go forward the required amount. The Go ecosystem has just grown too big. Lastly, I understand the temptation to let Fedora and RHEL slip in favor of product lines more profitable in the near term.
On my employer's side we *do* wish to stay relevant in a Cloud world. My Go contributions are an attempt to partner with the Fedora Centos and RHEL communities on that. If the partnership does not bear fruits, and Fedora/Centos/RHEL do not move in a direction we can use, we'll have to invest elsewhere. I'd regret it but this proposal and the spec dump that will follow are just too much work to do on one's free time. And employer time requires results.
I haven't had time to read through the entire proposal, but it certainly looks like a major step forward!
Please finish reading it and propose all the fixes and comments and improvements you want! Your experience is appreciated.
Regards,
On mardi 30 janvier 2018 16:11:49 CET nicolas.mailhot@laposte.net wrote:
Hi,
Now the technical PR is submitted https://src.fedoraproject.org/rpms/go-srpm-macros/pull-request/1
and waiting for action from the go-srpm-macros maintainers, I took (quite a long) time to refresh and flesh out the corresponding packaging guidelines proposal. It should be fairly complete now:
https://fedoraproject.org/wiki/More_Go_packaging
I'd appreciate review to check if I have forgotten an important case, if people understand the text, if they have enhancements or corrections to propose, and so on.
Then I will push it FPC side again.
Actual practice should be fairly simple and self-explanatory, the proposal length can be scary but that's because it documents all kinds of corner cases that required digging through specs and mailing lists to find resolution examples before. The basic Go packaging skeleton will be sufficient is most cases without requiring to read any documentation.
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
Hello Nicolas,
Could you add two full examples, one for binary package, one for library package, like in https://fedoraproject.org/wiki/PackagingDrafts/ Go#Sample_RPM_Spec
Putting it all together (https://fedoraproject.org/wiki/ More_Go_packaging#Putting_it_all_together) is nice but don't understand if it is for a binary or a library.
Also, is there any plan to update Gofed with these new guidelines? If it's not automated like today, we will lose time instead of gaining some in the end.
Best regards,
Robert-André
Hi Robert André
That's an interesting request
I guess you can't figure if the example is for building binaries or Go libs, because there is no hard frontier between both cases in the proposed guidelines
In Go, everything is effectively a code library that can be reused elsewhere. So the approach taken in the proposed guidelines, is that everything is a code library, some of which happen to build binaries, or setup compatibility symlinks, or need bootstrapping
It's a bit of a Lego guideline, you assemble the spec blocs you need, and ignore those you don't need. The example was chosen to include as many blocks as possible, with the walkthrough explaining their respective functions. All the blocks are very short and succinct, don't search for long runs of shell code like in the old guidelines
----- Mail original ----- De: "Nicolas Mailhot"
It's a bit of a Lego guideline, you assemble the spec blocs you need, and ignore those you don't need. The example was chosen to include as many blocks as possible, with the walkthrough explaining their respective functions. All the blocks are very short and succinct, don't search for long runs of shell code like in the old guidelines
That being said if people feel it's useful I can add a barebones lib-only example with none of the optional blocks in. However, I fear that the more ways the page explains the same stuff, the more confusing and difficult to maintain it will be.
So, can people chime in to say whether they feel a supplementary example is useful of counter productive?
Regards,
----- Original Message -----
From: "Robert-André Mauchin" zebob.m@gmail.com To: devel@lists.fedoraproject.org Cc: golang@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Friday, February 2, 2018 4:54:10 PM Subject: Re: Proposed Fedora packaging guideline: More Go packaging
On mardi 30 janvier 2018 16:11:49 CET nicolas.mailhot@laposte.net wrote:
Hi,
Now the technical PR is submitted https://src.fedoraproject.org/rpms/go-srpm-macros/pull-request/1
and waiting for action from the go-srpm-macros maintainers, I took (quite a long) time to refresh and flesh out the corresponding packaging guidelines proposal. It should be fairly complete now:
https://fedoraproject.org/wiki/More_Go_packaging
I'd appreciate review to check if I have forgotten an important case, if people understand the text, if they have enhancements or corrections to propose, and so on.
Then I will push it FPC side again.
Actual practice should be fairly simple and self-explanatory, the proposal length can be scary but that's because it documents all kinds of corner cases that required digging through specs and mailing lists to find resolution examples before. The basic Go packaging skeleton will be sufficient is most cases without requiring to read any documentation.
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
Hello Nicolas,
Could you add two full examples, one for binary package, one for library package, like in https://fedoraproject.org/wiki/PackagingDrafts/ Go#Sample_RPM_Spec
Putting it all together (https://fedoraproject.org/wiki/ More_Go_packaging#Putting_it_all_together) is nice but don't understand if it is for a binary or a library.
Also, is there any plan to update Gofed with these new guidelines? If it's not automated like today, we will lose time instead of gaining some in the end.
Best regards,
Robert-André
I think that it would be best if Nicolas could fold his proposal in to the original draft as optional part for simple library/binary packages.
As his proposal doesn't cover at least two major use cases, i.e. separate packaging of tests(they have no place in devel package as they artificially inflate build root size) and shipping pre-built shared libraries.
Just my opinion,
JC
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
----- Mail original ----- De: "Jakub Cajka"
Hi Jakub
I think that it would be best if Nicolas could fold his proposal in to the original draft as optional part for simple library/binary packages.
Frankly, that's a lot of work and churn, I don't want the new parts to be refused because of something in the old parts, and I certainly do not want to spend weeks replacing bits only to put them back in and so on while people made up their mind on the things they want replaced or not. The new parts are pretty much autonomous and complete, they are sufficient to create working Fedora Go packages, they are ready for FPC review.
If someone wants to extract the ready non discussion parts of the old page and get them approved I can work with him to merge both elements
I didn't want to write about the old page, but since you put it on the table.
It is full of digressions and elements of no evident applied value while packaging Go software. It's not an operational "how to do stuff" document it's a "here are various things you may like to know about Go that may or may not help you create Fedora Go packages, if they do not help you forget about it and run gofed". There are too many WIP non operational bits in the old pages to allow merging while in an approval process. And I'm sure any attempt to strip the WIP bits from my side will be met with energetic protests.
People, if you want that page to be ever approved by FPC make it more focused and accurate. Remove anything that does not explain to a Fedora Go packager how to write a Fedora Go spec file and what to ask upstream Go projects in a Fedora packager role. And make sure what remains is succinct, easy to read, and applicable without undocumented holes or gofed magic.
I know that's a lot of non-fun work. I did this work on the new page. I don't particularly *like* writing documentation. For every line I put in the new page, I had to ask myself "is the value to the Fedora packager sufficient to justify the time spend writing the text, formatting it, linking in the right place, proofing the result". I didn't put in stuff I could not justify cleaning up myself. If it was too much work to document it was either of insufficient value or better automated rather than explained in a manual process.
Any part of the text that can not be finished or serves another role has no place in a guideline submitted to FPC. It should be nuked or moved to a separate wiki page. All the half-finished and non operational stuff in there is the reason the draft has been stuck in pre-approval stage for four years. Just put yourself in FPC's place, its mission is to confirm a guideline is ready to be used by the average Fedora packager, not to produce it from half-finished half-related messy notes of the domain experts.
As his proposal doesn't cover at least two major use cases, i.e. separate packaging of tests(they have no place in devel package as they artificially inflate build root size)
At this point of time my mind is pretty much set. I won't do separate packaging of tests because: 1. that raises complex dependency handling questions 2. the average Go project test code is full of crufty junk 3. the average Go test depends on assets not tracked within Go 4. Go is not designed to separate elements shipped in the same directory 5. no one could answer when I asked the *three* Fedora lists what they were using the existing test packages for 6. from what I've seen many of the existing test packages simply can not work because they fail to include elements they need 7. even core Go people refuse to touch the subject with a 10 foot pole. Sam Boyer tried to demo a very small part of what would be necessary this week end at FOSDEM and this part of his demo *failed*. I don't have the level of Go understanding Sam Boyer Has.
I am ready to work with people who want to make separate test packages and I tried very hard to make them possible in the future but I won't spend time trying to ship half-baked stuff that does not work and that no one has a need for.
The proposed patterns strip test code from devel packages so build root sizes are safe (except for the original package build root, since I *do* execute unit tests in check, and they pull their deps in the build root).
You want to test one of my packages, fine, just rebuild it. That will run unit tests *and* build the project binaries (which are also a form of code testing, and a very thorough one at that).
and shipping pre-built shared libraries.
The pre-build shared libraries the old guidelines refuse to build or document? Why exactly are you stating it's a major use case, Fedora is not doing it today.
"By default, the standard golang compiler produces static libraries. There is little value in shipping these prebuilt, especially since these libraries are very specifically tied to the exact minor release of the golang compiler."
"Presently the shared object libraries produced by GCC-Go are not usable"
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Monday, February 5, 2018 4:48:31 PM Subject: Re: Proposed Fedora packaging guideline: More Go packaging
----- Mail original ----- De: "Jakub Cajka"
Hi Jakub
I think that it would be best if Nicolas could fold his proposal in to the original draft as optional part for simple library/binary packages.
Frankly, that's a lot of work and churn, I don't want the new parts to be refused because of something in the old parts, and I certainly do not want to spend weeks replacing bits only to put them back in and so on while people made up their mind on the things they want replaced or not. The new parts are pretty much autonomous and complete, they are sufficient to create working Fedora Go packages, they are ready for FPC review.
If someone wants to extract the ready non discussion parts of the old page and get them approved I can work with him to merge both elements
I didn't want to write about the old page, but since you put it on the table.
It is full of digressions and elements of no evident applied value while packaging Go software. It's not an operational "how to do stuff" document it's a "here are various things you may like to know about Go that may or may not help you create Fedora Go packages, if they do not help you forget about it and run gofed". There are too many WIP non operational bits in the old pages to allow merging while in an approval process. And I'm sure any attempt to strip the WIP bits from my side will be met with energetic protests.
Have you tried that? What make you assume that? I'm sure that if you do it in constructive way, they will be accepted.
People, if you want that page to be ever approved by FPC make it more focused and accurate. Remove anything that does not explain to a Fedora Go packager how to write a Fedora Go spec file and what to ask upstream Go projects in a Fedora packager role. And make sure what remains is succinct, easy to read, and applicable without undocumented holes or gofed magic.
I believe that technically exhausting document is needed as Go packaging is far from trivial. Sure it would be great to have (trivial) quick start guide. I think that your proposal fits that bill more than full documentation.
I know that's a lot of non-fun work. I did this work on the new page. I don't particularly *like* writing documentation. For every line I put in the new page, I had to ask myself "is the value to the Fedora packager sufficient to justify the time spend writing the text, formatting it, linking in the right place, proofing the result". I didn't put in stuff I could not justify cleaning up myself. If it was too much work to document it was either of insufficient value or better automated rather than explained in a manual process.
Any part of the text that can not be finished or serves another role has no place in a guideline submitted to FPC. It should be nuked or moved to a separate wiki page. All the half-finished and non operational stuff in there is the reason the draft has been stuck in pre-approval stage for four years. Just put yourself in FPC's place, its mission is to confirm a guideline is ready to be used by the average Fedora packager, not to produce it from half-finished half-related messy notes of the domain experts.
I believe I can see myself in their shoes(sure not 100% accurately) and I don't see anything that you are describing from their POV, it would be good to have the guidelines sooner, but I will have, as you, rather something complete than something half baked and semi broken. Please note that they are WIP and everybody is welcomed to contribute in constructive way. I don't think that there is any request for FPC to create the guidelines on their own and honestly it would rather bad idea IMHO.
As his proposal doesn't cover at least two major use cases, i.e. separate packaging of tests(they have no place in devel package as they artificially inflate build root size)
At this point of time my mind is pretty much set. I won't do separate packaging of tests because:
- that raises complex dependency handling questions
- the average Go project test code is full of crufty junk
- the average Go test depends on assets not tracked within Go
- Go is not designed to separate elements shipped in the same directory
- no one could answer when I asked the *three* Fedora lists what they were
using the existing test packages for
They are used primarily to minimize build root size, by reducing the deps size and code size.
- from what I've seen many of the existing test packages simply can not work
because they fail to include elements they need
Please file bugs or pull requests. We need to work together to improve the experience. I'm sure that you are including all the dependencies in your devel sub-package(that contain the tests) to be able to run them locally.
- even core Go people refuse to touch the subject with a 10 foot pole. Sam
Boyer tried to demo a very small part of what would be necessary this week end at FOSDEM and this part of his demo *failed*. I don't have the level of Go understanding Sam Boyer Has.
What has been demoed?? Unfortunately I could not make it there. Have you met with Jan there?
I am ready to work with people who want to make separate test packages and I tried very hard to make them possible in the future but I won't spend time trying to ship half-baked stuff that does not work and that no one has a need for.
The proposed patterns strip test code from devel packages so build root sizes are safe (except for the original package build root, since I *do* execute unit tests in check, and they pull their deps in the build root).
Your packaging proposal seems fairly monolithic and disregarding any prior art, so extensions will be very painful.
You want to test one of my packages, fine, just rebuild it. That will run unit tests *and* build the project binaries (which are also a form of code testing, and a very thorough one at that).
I don't think that is great user experience nor packagers/maintainers experience.
and shipping pre-built shared libraries.
The pre-build shared libraries the old guidelines refuse to build or document? Why exactly are you stating it's a major use case, Fedora is not doing it today.
?? I'm bit confused. Yes, it is not mentioned there yet as Jan have been working on stabilizing APIs, so it will be feasible and added there.
"By default, the standard golang compiler produces static libraries. There is little value in shipping these prebuilt, especially since these libraries are very specifically tied to the exact minor release of the golang compiler."
"Presently the shared object libraries produced by GCC-Go are not usable"
You have mentioned that the API breakage is not that big issue, so this led me to conclude that shared lib/bin packaging is now feasible. I believe that any Go packaging guideline that doesn't implement this side in working fashion is not FPC ready IMNHO.
JC
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
----- Mail original ----- De: "Jakub Cajka"
Hi Jakub,
And I'm sure any attempt to strip the WIP bits from my side will be met with energetic protests.
Have you tried that? What make you assume that? I'm sure that if you do it in constructive way, they will be accepted.
My experience with humans is that they get attached to their bits of text even when they have no time to finish them. But yes, I haven't tried, I don't want to see a wiki page for a month after writing and formatting and revising two separate packaging drafts.
I believe that technically exhausting document is needed as Go packaging is far from trivial. Sure it would be great to have (trivial) quick start guide. I think that your proposal fits that bill more than full documentation.
IMHO that's exactly what FPC expects: a quick start document, not an in-depth encyclopædia. In-depth is too hardcore to be validated by domain laymen, is useless to onboard new packagers (which is the primary objective of Fedora guidelines), is never finished because the state of the art changes, and can't be applied to check whether a spec is good or not because in-depth concerns *complex* cases where the "right" choice is not obvious, depends on many factors that change over time, and usually requires asking the domain experts directly.
So you have the simple common cases, which are sufficient for most packagers, and can be addressed by a formal document FPC can review and enforce, and "other things" which are valuable but can not ever attain the formalness and strictness level expected of Fedora guidelines, require continuous freedom to edit, and are hosted in other Fedora wiki pages. At least that's how real-life law is written (law text + expert commentaries) and how other Fedora packaging SIGs function and got their guidelines approved.
They are used primarily to minimize build root size, by reducing the deps size and code size.
Stripping example material from gopath installation filelists achieves the same objective without all the complexity of managing packages that share directories Go considers indivisible.
- even core Go people refuse to touch the subject with a 10 foot pole. Sam
Boyer tried to demo a very small part of what would be necessary this week end at FOSDEM and this part of his demo *failed*. I don't have the level of Go understanding Sam Boyer Has.
What has been demoed?? Unfortunately I could not make it there
Sam Boyer presented the current state of his go dep prototype, and explained his design choices (for example no nested projects which means the go import path root becomes a core concept and it's useless to invest in splitting projects as that usually requires nesting). He ended up stating go dep will ignore tests by default, tried to demo optional test handling, and that failed. He had another conference the next day I couldn't get to as I was stuck in another part of the campus.
Have you met with Jan there?
Unfortunately FOSDEM was its usual crazy stuff with multiple interesting conferences at any given time and no hope of being admitted in a room if you're late. So I've just been running from conf room to conf room without any hope of meeting anyone :(
Your packaging proposal seems fairly monolithic and disregarding any prior art, so extensions will be very painful.
Well, the creation of the proposal was a long suite of extending to cover more cases (that's why some macros are rather long and convoluted), so I'm quite sure extension is possible. Of course I may miss something, if you identify a roadblock please point it out.
You want to test one of my packages, fine, just rebuild it. That will run unit tests *and* build the project binaries (which are also a form of code testing, and a very thorough one at that).
I don't think that is great user experience nor packagers/maintainers experience.
That's pretty much how rpm prefers it to be (use %check and Builddeps) and also how Go prefers it to be (no provision for splitting directories). Of course anyone interested can try to fight the design of both of them to achieve something else, and would try to help, but I'm not interested in being this person.
and shipping pre-built shared libraries.
The pre-build shared libraries the old guidelines refuse to build or document? Why exactly are you stating it's a major use case, Fedora is not doing it today.
?? I'm bit confused. Yes, it is not mentioned there yet as Jan have been working on stabilizing APIs, so it will be feasible and added there.
My maybe naïve POW is that shared libraries are just another container for the elements currently installed in %gopath/src, so doing them is mostly adding a separate stage in %build and making %goinstall install the result instead of installing raw files.
The hardest part is to get compatible code bases, defining what needs stripping in production, and defining shared libraries numbering.
The proposal helps for the two first points and I have no idea or opinion on the third.
The rest is macro plumbing that should not require a lot of spec surgery once specs are mostly free of custom shell code.
You have mentioned that the API breakage is not that big issue, so this led me to conclude that shared lib/bin packaging is now feasible.
API breakage is not a showstopper with the static linking Go prefers because careful BuildDep construction can ensure only one (and the right one) dependency code state ends up in the buildroot, and BuildDeps of the next app can select another code state. So it's mostly a mater of creating the necessary -devel compat packages and defining BuildDep constrains.
It is a showstopper for shared libs because you can't install several versions of the same shared lib without separate identifiers/release ids to disambiguate them, Go upstreams do not provide those identifiers today and I doubt creating a Fedora-specific identifier namespace at the lib level would be a good idea. Though, the few projects that version their import paths, and make sure to only extend their APIS within a major release, could be libified today (theoretically, I've not checked how to do it). I'm not sure there are enough of them to be worth multiple packaging style
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Tuesday, February 6, 2018 7:34:03 PM Subject: Re: Proposed Fedora packaging guideline: More Go packaging
----- Mail original ----- De: "Jakub Cajka"
Hi Jakub,
And I'm sure any attempt to strip the WIP bits from my side will be met with energetic protests.
Have you tried that? What make you assume that? I'm sure that if you do it in constructive way, they will be accepted.
My experience with humans is that they get attached to their bits of text even when they have no time to finish them. But yes, I haven't tried, I don't want to see a wiki page for a month after writing and formatting and revising two separate packaging drafts.
I believe that technically exhausting document is needed as Go packaging is far from trivial. Sure it would be great to have (trivial) quick start guide. I think that your proposal fits that bill more than full documentation.
IMHO that's exactly what FPC expects: a quick start document, not an in-depth encyclopædia. In-depth is too hardcore to be validated by domain laymen, is useless to onboard new packagers (which is the primary objective of Fedora guidelines), is never finished because the state of the art changes, and can't be applied to check whether a spec is good or not because in-depth concerns *complex* cases where the "right" choice is not obvious, depends on many factors that change over time, and usually requires asking the domain experts directly.
So you have the simple common cases, which are sufficient for most packagers, and can be addressed by a formal document FPC can review and enforce, and "other things" which are valuable but can not ever attain the formalness and strictness level expected of Fedora guidelines, require continuous freedom to edit, and are hosted in other Fedora wiki pages. At least that's how real-life law is written (law text + expert commentaries) and how other Fedora packaging SIGs function and got their guidelines approved.
They are used primarily to minimize build root size, by reducing the deps size and code size.
Stripping example material from gopath installation filelists achieves the same objective without all the complexity of managing packages that share directories Go considers indivisible.
??, I have been talking about tests. From Go perspective source files and test source file, testdata are two distinctive sets. From my POV if your code depends for regular builds on tests code it means that there is something horribly wrong with your project(regardless of language used).
Docs and examples would be great too, it should be in guidelines that packager must use reasonable effort to create subpackages with them.
- even core Go people refuse to touch the subject with a 10 foot pole.
Sam Boyer tried to demo a very small part of what would be necessary this week end at FOSDEM and this part of his demo *failed*. I don't have the level of Go understanding Sam Boyer Has.
What has been demoed?? Unfortunately I could not make it there
Sam Boyer presented the current state of his go dep prototype, and explained his design choices (for example no nested projects which means the go import path root becomes a core concept and it's useless to invest in splitting projects as that usually requires nesting). He ended up stating go dep will ignore tests by default, tried to demo optional test handling, and that failed. He had another conference the next day I couldn't get to as I was stuck in another part of the campus.
Cool, there is recording for anyone interested https://fosdem.org/2018/schedule/event/dep/.
Have you met with Jan there?
Unfortunately FOSDEM was its usual crazy stuff with multiple interesting conferences at any given time and no hope of being admitted in a room if you're late. So I've just been running from conf room to conf room without any hope of meeting anyone :(
Your packaging proposal seems fairly monolithic and disregarding any prior art, so extensions will be very painful.
Well, the creation of the proposal was a long suite of extending to cover more cases (that's why some macros are rather long and convoluted), so I'm quite sure extension is possible. Of course I may miss something, if you identify a roadblock please point it out.
You want to test one of my packages, fine, just rebuild it. That will run unit tests *and* build the project binaries (which are also a form of code testing, and a very thorough one at that).
I don't think that is great user experience nor packagers/maintainers experience.
That's pretty much how rpm prefers it to be (use %check and Builddeps) and also how Go prefers it to be (no provision for splitting directories). Of course anyone interested can try to fight the design of both of them to achieve something else, and would try to help, but I'm not interested in being this person.
and shipping pre-built shared libraries.
The pre-build shared libraries the old guidelines refuse to build or document? Why exactly are you stating it's a major use case, Fedora is not doing it today.
?? I'm bit confused. Yes, it is not mentioned there yet as Jan have been working on stabilizing APIs, so it will be feasible and added there.
My maybe naïve POW is that shared libraries are just another container for the elements currently installed in %gopath/src, so doing them is mostly adding a separate stage in %build and making %goinstall install the result instead of installing raw files.
The hardest part is to get compatible code bases, defining what needs stripping in production, and defining shared libraries numbering.
The proposal helps for the two first points and I have no idea or opinion on the third.
The rest is macro plumbing that should not require a lot of spec surgery once specs are mostly free of custom shell code.
You have mentioned that the API breakage is not that big issue, so this led me to conclude that shared lib/bin packaging is now feasible.
API breakage is not a showstopper with the static linking Go prefers because careful BuildDep construction can ensure only one (and the right one) dependency code state ends up in the buildroot, and BuildDeps of the next app can select another code state. So it's mostly a mater of creating the necessary -devel compat packages and defining BuildDep constrains.
Not really, mostly with static linking you are fine if you change deps as long as you don't touch basic runtime(glibc,...) and binaries will keep on working. In dynamic linking when you change the deps in incompatible way(break API/ABI) you break the runtime for all built binaries and you need to do rebuild to just keep distro working.
It is a showstopper for shared libs because you can't install several versions of the same shared lib without separate identifiers/release ids to disambiguate them, Go upstreams do not provide those identifiers today and I doubt creating a Fedora-specific identifier namespace at the lib level would be a good idea. Though, the few projects that version their import paths, and make sure to only extend their APIS within a major release, could be libified today (theoretically, I've not checked how to do it). I'm not sure there are enough of them to be worth multiple packaging style
Regards,
-- Nicolas Mailhot
You can version the so's arbitrarily for example and have explicit non general require on the devel, I think this really minor implementation detail from packaging discussion POV.
JC
golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
Hi Nicolas,
before I start arguing, I appreciate all your hard work on improving the Go guidelines, spending so much time explaining reasoning behind your decision. Also appreciating the effort on making the life easier for packagers, not just in the Go land, but throughout the distribution as well. There is a lot of great ideas in your Go proposal I would like to see implemented and widely used. The more transparent and easy-to-use spec files, the better for all maintainers. Let's keep moving forward, improving the infrastructure for other folks and pushing new ideas and improvements into practical solutions.
On 12/17/2017 08:11 AM, nicolas.mailhot@laposte.net wrote:
Hi,
I am proposing for inclusion a set of rpm technical files aimed at automating the packaging of forge-hosted projects.
- Packaging draft:https://fedoraproject.org/wiki/More_Go_packaging
-https://pagure.io/packaging-committee/issue/734
- go-srpm-macros RFE with the technical files:https://bugzilla.redhat.com/show_bug.cgi?id=1526721
This proposal is integrated with and depends on thehttps://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft It builds on the hard work of the Go SIG and reuses the rpm automation ofhttps://fedoraproject.org/wiki/PackagingDrafts/Go when it exists, and produces compatible packages.
Can you describe what you mean by "compatible packages"? I mentioned a list of things that you did not answer fully. Most important to me: - your macros do not generate build-time dependencies, which I see as one of the drawbacks. Do you plan to ignore them? - support for more subpackages. You describe how to specify which files go to which packages (https://fedoraproject.org/wiki/More_Go_packaging#Separate_code_packages) but you don't say how list of provided packages and a list of dependencies are generated for the subpackages. - reproducible evaluation of macros. Either for running automatic analysis over spec files or for debugging purposes. I would like the macros to be built in top of binaries I can run separately. With Jakub (jcajka@redhat.com) we were discussing many times we could provide a minimal rpm built on top of gofedlib that would extract all the necessary pieces, including the naming convention so everyone can import provided library guided by the same rules as the guidelines enforce. I would like to see us synchronizing on this topic at some point.
What it does:
- drastically shorter spec files, up to 90% in some cases, often removing hundreds of lines per spec.
+1 here, the shorter the spec file the better as long as it is still clear and transparent By better I mean: - easier to read, easier to understand, easier to adopt - less places to change and look if a spec file change is needed By clear I mean: - it is obvious what the spec file is actually declaring to do - it is easy to customize various parts (e.g. list of tests, list of dependencies)
- simple, packager-friendly spec syntax
+1 as long as each macro provides a single functionality. E.g. - generate a list of provided packages - generate a list of tests - generate a list of dependencies
+1 to the %goname, %gourl, %gosource, %goinstall, %__goprovides, %_gorequires and other usefull macros as long as they are simple and transparent
As long as the macros are optional and user can use any of them if it is suitable (suitable meant per a packager judgment).
- automated package naming derived from the native identifier (import path). No more packages names without any relation with current upstream naming.
Can you provide a list of package names that diverge from this convention?
For historical reasons there are some packages that does not fall into the naming schema. Either because a go project got migrated to a different repo or because it make/made sense to combine some projects together and ship them in a single rpm. I don't mention Kubernetes, Docker and other popular projects as you already mention exception for them.
- working Go autoprovides. No forgotten provides anymore.
Sometimes it make sense to skip some provided package. E.g. some packages provide Windows specific code which has no use on Linux. So the claim is not completely valid.
- working Go autorequires. No forgotten requires anymore.
Depending on how you generate the list you can non-intentionally require more than is needed. Which causes installation of unneeded trees of rpms. E.g. for building purposes there is no need to install dependencies of tests So the claim is not completely valid.
(Valid for both requires/provides): Sometimes you only need to install/provide a subset of a Go project. There are not many projects that imports every package another project provides. So to save time packaging dependencies of unimported packages (and their following maintenance), it is beneficial to generate only partial list of required/provided packages.
Plus, in case CGO, there is no 1:1 mapping of C libraries to their corresponding rpms.
- strict automated directory ownership (used by autorequires and autoprovides).
- centralized computation of source URLs (via Forge-hosted projects packaging automation). No more packages lacking guidelines. No more broken guidelines no one notices.
In other words to use %gourl macro?
- easy switch between commits, tags and releases (via Forge-hosted projects packaging automation). No more packages stuck on commits when upstream starts releasing.
The issue is not about upstream releasing, it's about projects that actually import snapshots of other projects instead of particular releases. Given by the nature of the Go land.
- guidelines-compliant automated snapshot naming, including snapshot timestamps (via Forge-hosted projects packaging automation). No more packages stuck in 2014.
The issue is not that a package is stuck and not updated for more than 6 months. The issue is API backwards incompatibility that does not allow to easily update to a newer version. And a lack of man power to perform an update.
- guidelines-compliant bootstraping.
- systematic use of the Go switches defined by the Go maintainer. Easy to do changes followed by a mass rebuild.
- flexibility, do the right thing transparently by default, leave room for special cases and overrides.
- no bundling (a.k.a. vendoring) due to the pain of packaging one more Go dependency.
Can you more elaborate on that? Cause this is unavoidable no matter how great any Go guidelines get. Choice of bundeling is not about how many more Go dependencies you need to package. It's due to API backwards incompatibility reasons.
- centralized Go macros that can be audited and enhanced over time.
+1 as long as they are kept on a reasonable granularity level
- aggressive leverage of upstream unit tests to detect quickly broken code.
Will not work in general due to API backwards incompatibility issues. We need to move the tests out of the spec files if we want to seriously test Go projects. Saying that, we can run subset of tests to at least have some kind of smoke tests.
no reliance on external utilities to compute code requirements. No more dependencies that do not match the shipped Go code.
Please, rephrase or remove the point as for the CGO it is useful to have external utility that can list all imported C header files and try to find a set of packages that provide the header files (and corresponding libraries).
Please consult packaging draft for full information.
The proposal has been tested in Rawhide and EL7 over a set of ~ 140 Go packages. This set is a mix of current Fedora packages, bumped to a more recent version, rewrites of Fedora packages, and completely new packages.
I hope posting the second part of the automation will answer some questions people had on thehttps://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft
Regards,
I am surprised you are able to do that without any API issues. Have you tried to de-bundle and build Kubernetes, Docker, Etcd, Prometheus and other big projects from the same set of Go packages? At some point you have/had to switch to compat packages or bundling.
========================================= From the guidelines:
https://fedoraproject.org/wiki/More_Go_packaging#Limitations
Totally agree with the release discipline in many Go projects.
gives up on many conventions of current Fedora Go packaging, as they were an obstacle to the target packaging efficiency.
(Still part of the Limitations): Can you make a list of conventions you consider as obstacles? I would like to comment on each of them.
https://fedoraproject.org/wiki/More_Go_packaging#Excluding_testing_in_specif... %gochecks . transport/http transport/grpc transport option internal integration-tests/storage examples
The excluding mechanism is not intuitive, my first understand would be to test all the listed directories. Maybe use %gocheck -v the same way is grep does?
https://fedoraproject.org/wiki/More_Go_packaging#Handling_API_changes API changes may require the creation of Go code compatibility packages
I would not recommended doing that unless you have an army of packagers. It not just about creating one compat package. It is not surprising API incompatible change goes hand to hand with updating of dependencies. So instead of creating one compat package you will end up creating a tree of compat packages. I have already mentioned that at https://pagure.io/packaging-committee/issue/382#comment-147649. Plus, multiply the number of compat packages to take care with the active Fedora branches + epel7. On paper it looks like a very good solution but in reality it is not. As long as we don't know how much the Go projects suffer the API incompatibilities, I don't want to spawn thousand compat rpms and maintain them.
a packager, that identifies an API change in his Go package, MUST notify golang@fedoraproject.org at least a week before pushing his changes to any release (and those releases SHOULD include fedora-devel). This grace period can be waived by FPC for security reasons.
From the point of view of updating popular projects like Etcd, Kubernetes, Prometheus, etc. this is highly impractical. Usually, there are more dependencies to update. If I will have to wait one week to update etcd, than another week to push its update to stable branch, then I will not really be effective in updating.
a packager, that identifies an unannounced API change in another Go package, MUST notify golang@fedoraproject.org at once. those notices SHOULD be copied to the maintainers of packages affected by the change, when they are known.
This is going to be a regular thing unless a tooling detecting this changes is available.
a packager, that identifies that the code he packages, is broken by API changes in another project, SHOULD notify politely the upstream of the broken project of the API change, if he can not ascertain it is already aware of the change.
Some upstream Go projects are responsive, some are not. A lot of Go projects are forked, abandoned or inactive. Or the upstream is a single person that does not have time to deal with distribution specific guidelines. But I appreciate the effort to spread the good practices. However, sometimes the API change is intentional cause there is no other way.
=========================================
Usual workflow of updating Go packages:
1. choose a commit to update to 2. update a list of provided packages 3. update a list of build/run-time dependencies (if needed create spec files for new Go projects and open review requests) 4. update a list of tests (skip some tests that fail on specific architectures + open upstream issues, resp. PR with a fix) 5. optionally, update %install/%build section 6. perform scratch-build (may require to override some rpms) 7. perform push and raw build 8. create package update in Bodhi 9. push updates into stable if there is not enough karmas
## Some issues one can encounter with when performing a Go package update - update is backward incompatible (it may remove ability of some package Go projects to get built) - 5 (arbitrary number here) or more new Go projects need to be packaged into distribution
## What we need - automatic updates of Go packages (each time there is a new release or on demand to a specific commit) - spread maintenance among a group of packagers - run unit-tests and integration tests (if available) in CI/CD (e.g. after each update, on periodic basis) - automatic every routine job we do so we can concentrate on "real" and "interesting" problems in the Go land
The updated (and fairly improved/extended) guidelines cover points 2), 3), 4), 5) to some extend. Comparing the new way with the current way (listing only some +/- that popped in my head during interpretation): + provided packages, runtime dependencies are automatically generated + tests are automatically generated + spec files are much smaller (more transparent, a lot of complexity hidden in macros) + guidelines looks awesome, more descriptive in what needs to be done (up to some parts mentioning many times the current spec files sucks a lot)
- no build-time dependencies (we can not discover missing dependencies before installation) - always generating all provided packages that are available (I don't want to provide all packages always, e.g. need just a subset to build my project, the remaining provided packages need new Go projects which adds unnecessary complexity) - as jcajka pointed out, it is not easy to debug the lua based scripts - combination of bundled and un-bundled dependencies (sometimes I just need to use one or two bundled dependencies because without generating a tune of compat packages)
In all cases, the new approach is strongly focused on specfile simplification. The API backwards incompatibility problems are conveniently hidden in the creation of Go code compatibility packages. I have already mentions this approach in the https://pagure.io/packaging-committee/issue/382 and the reasons why it is not a way to go.
Even if we create a lot of compat packages we still need a mechanism that can tell us when a given compat package is no longer needed. I am working on a tooling (in my free time) that will provide that information (https://github.com/gofed/symbols-extractor, feel free to contribute to the project with ideas who we need to detect or extract from the code to make more spec file pieces generic and macro-replaceable).
At the end, even if we do "create as many compat Go packages as needed" or "let's use vendored packages because we can", we will still need to maintain entire ecosystem in some reasonable way.
From my point of view, the new Go packaging guidelines make it easier for fresh Go packager to get familiar with Go packaging practices and to quickly package his/her Go project. However, from the point of view of entire Go ecosystem (or any micro ecosystem = I have more projects building from the same dependencies), it is harder to tune individual packages for the mutual benefit.
=========================================
In overall, I don't like you degrading the current Go packaging guidelines (although it is still a draft). All the decisions made have historical reasons. By the time of applying the guidelines, there were only a few go packages and the severity and impact of the practices were not eminent nor obvious. Now we know it is not sufficient and there are things we need to improve. Indeed, 100% agree. So please, pay at least some respect to all who spent their time and dreamless nights coming with quick solutions. Cause a lot of us do it in our free-time so we are limited by time and capacity. What you are trying to solve now was known two years ago. We already knew about that. For that reason my and my colleagues started building tooling to detect at least some of the known problems like API backwards incompatibilities or automation around updating spec files.
I also suggest to remove every comparison with the Go guidelines draft and the current state of all Go spec files. Optionally, you can put all the mentions under "Best practices" or "Thing to avoid" section instead.
There are other parts of the new guidelines I could contradict. However, doing that we can end up in endless discussion, annoying each other and burning time that we can use to do important things instead.
So let's divide the workload. If you want to take care of the packaging guidelines, feel free to do it. Want to improve the spec file? I will be more than happy to witness that. Do you want to make the Go packaging in Fedora (and maybe other distributions) more user friendly? Man, that is awesome and noble. I have no intention to stop you from doing that. But please, keep in mind, you are not the only one involved in this effort. Some of us have families, some of us has other things/folks to take care of. We only do what we can in the time given :). I have/had my use case I am/was driven, you have yours. That's completely fine and appreciated. The more use cases we can fine, the more ideas and solutions we can combine. Let's not degrade results of our efforts, it leads nowhere. Let's combine the results, our experience. Let's open a google document (or different collaboration solution if you have one in mind) where we collect all our requirements. Let's ask other Go (or wannabe) packagers what are their requirement, what they see as drawbacks of the current and new guidelines, who is willing to help with the packaging and maintenance, let's ask more questions.
I am happy to take the tooling side. As I mentioned in the ticket (https://pagure.io/packaging-committee/issue/382), I shifted my focus on building tooling that do a lot of analysis and detection which are out of scope of any guidelines. Even if we agree on the final form of the guidelines, we still need tooling that will help us with the packaging. E.g. detection of API backwards incompatibilities, spec file generators (enhancing and improving over time), detection of go packages that are no longer needed, running CI/CD, etc.
Regards Jan
De: "Jan Chaloupka"
Hi Jan
Apologies for the delayed answer, I was beating the PR into shape to address review comments.
Let's keep moving forward, improving the infrastructure for other folks and pushing new ideas and improvements into practical solutions.
Let's move forward, I'm sick on sitting on a private spec stash and I see people trying to package stuff already in my list with the old guidelines.
On 12/17/2017 08:11 AM, nicolas.mailhot wrote:
It builds on the hard work of the Go SIG and reuses the rpm automation ofhttps://fedoraproject.org/wiki/PackagingDrafts/Go when it exists, and produces compatible packages.
Can you describe what you mean by "compatible packages"?
That mostly means the resulting packages can use "old-style" packages and "old-style" packages can use "new-style" packages. The effective result is similar enough there is no need for a rupture. Now, as I wrote before "old-style" packages will often be too old or not rigorous enough to be useful to "new-style" packages.
I mentioned a list of things that you did not answer fully. Most important to me:
- your macros do not generate build-time dependencies, which I see as
one of the drawbacks.
Generating BuildRequires dynamically needs changes in rpm tooling (mostly mock, I think). So it can not be done today if I'm not wrong. If you have a tool that outputs BuildRequires outside rpm, you can still use it. That's not an absolute showstopper as the go compiler is pretty explicit on what's missing, so BuildRequires are easily added during the first %build or %check run (though that's tedious and annoying).
Do you plan to ignore them?
Generating BuildRequires would be quite easy if rpm tooling provided an entry point at the end of %prep. So, that's a mid-term objective and requires sending an RFE mock-side first.
Requires generation is automatic and complete, so a Go -devel package will pull in all it needs without manual intervention (except for version constrains, that needs some maturing go dep side first)
- support for more subpackages. You describe how to specify which files
go to which packages (https://fedoraproject.org/wiki/More_Go_packaging#Separate_code_packages) but you don't say how list of provided packages and a list of dependencies are generated for the subpackages.
It is fully automated and transparent, rpm does not care why you split your Go code, it will compute provides and requires for every directory installed and owned under %{gopath}/src/%{goipath}, and attribute the result to the (sub)package owning the the directory. So just choosing what to install in each %goinstall invocation is sufficient to compute the corresponding requires/provides and attach them to the correct subpackage. If you don't want your package to pull a particular dep, just don't install the code that calls it.
- reproducible evaluation of macros. Either for running automatic
analysis over spec files or for debugging purposes. I would like the macros to be built in top of binaries I can run separately.
That would be quite difficult to do as most macros depend on rpm variables and other context defined during the build run, and can themselves alter rpm state dynamically via variable setting and other calls. rpm is not a dumb archive format it includes all kinds of package building facilities that the macros take advantage of, it ships with an embedded lua engine, and so on.
For "simple" macros that only read a specific variable set, and only perform "simple" actions (or set a specific set of variables) you could certainly rewrite them as separate scripts or binaries (for example the naming code is pretty trivial to rewrite in any language, and takes only one input and one output). But the more extensive the integration with rpm is, the more rpm glue you'd need to interface the result, and in some cases the glue is likely to be a significant part of the existing code. Also, you'd lose quite a lot of flexibility, and complexify your build root requirements.
I'm not saying it can not be done, or it is not worthwhile to do, but it's a lot of analysis and rewriting work and I'm not sure it's worth it (I'm certainly not interested to do it myself).
Note that autodeps are already effectively separate scripts, you just need to pass them a few arguments and pipe them Go code directory names to see them working. It may feel weird but that's how the autodep rpm API is defined. Other ecosystems packaged as rpms sometimes use separate binaries there, sometimes with light shell gluing. You have my blessing to rewrite them in Go if you want to.
- automated package naming derived from the native identifier (import path). No more packages names without any relation with current upstream naming.
Can you provide a list of package names that diverge from this convention?
For example, all the golang/x/foo stuff, which is a pity as it is used right and left.
For historical reasons there are some packages that does not fall into the naming schema. Either because a go project got migrated to a different repo
I understand, but the packages should be renamed, and not carry on the previous previous previous upstream name.
or because it make/made sense to combine some projects together and ship them in a single rpm.
Combining does not really work from a lifecycle POW, won't be easy to do with the new automation, or with native Go tooling. You really need to give up on it, sorry, it has never been a good idea for Go or for other languages, all the tooling expects *one* project not many, even when you can somewhat force it it never ends well.
- working Go autoprovides. No forgotten provides anymore.
Sometimes it make sense to skip some provided package. E.g. some packages provide Windows specific code
Then don't install the windows code and it won't pull corresponding deps.
- working Go autorequires. No forgotten requires anymore.
Depending on how you generate the list you can non-intentionally require more than is needed. Which causes installation of unneeded trees of rpms.
Again, the solution is to not install problem code, instead of shipping non-working code to others.
E.g. for building purposes there is no need to install dependencies of tests
I don't install test code so it does not pollute other projects with its dependencies
So the claim is not completely valid.
Yes it is. You're just too used to installing everything without filtering, and trying to workaround the resulting problems.
(Valid for both requires/provides): Sometimes you only need to install/provide a subset of a Go project.
If you only install part of a project, rpm will only generate requires/provides for the partial install. Note, however, than this is not the direction go dep is going towards. A lot of projects are going to need to structure their code better in a go dep world (for exactly the same reasons rpm requires better structuring: tools are binary, there is no convenient gray messy zone like when dealing with humans)
There are not many projects that imports every package another project provides. So to save time packaging dependencies of unimported packages (and their following maintenance),
Again, just don't install unneeded code, it won't pull any unneeded dep. Much simpler for everyone involved.
- strict automated directory ownership (used by autorequires and autoprovides).
- centralized computation of source URLs (via Forge-hosted projects packaging automation). No more packages lacking guidelines. No more broken guidelines no one notices.
In other words to use %gourl macro?
Actually, you need %gosource more. %gourl is a pretty side effect.
- easy switch between commits, tags and releases (via Forge-hosted projects packaging automation). No more packages stuck on commits when upstream starts releasing.
The issue is not about upstream releasing, it's about projects that actually import snapshots of other projects instead of particular releases. Given by the nature of the Go land
Try to convert one old-style Go spec from commit to release, or the reverse, you'll see it's not quite so simple there is a lot of tricky boilerplate to change. I've seen quite a few packages, that used commits that corresponded to releases, because redoing the spec as a release was too much hassle.
- guidelines-compliant automated snapshot naming, including snapshot timestamps (via Forge-hosted projects packaging automation). No more packages stuck in 2014.
The issue is not that a package is stuck and not updated for more than 6 months. The issue is API backwards incompatibility that does not allow to easily update to a newer version. And a lack of man power to perform an update.
You can hope for more manpower or you can make changes less time-consuming. Both result in more work done, one is less realistic than the other.
- no bundling (a.k.a. vendoring) due to the pain of packaging one more Go dependency.
Can you more elaborate on that? Cause this is unavoidable no matter how great any Go guidelines get. Choice of bundeling is not about how many more Go dependencies you need to package. It's due to API backwards incompatibility reasons.
You can address API incompatibilities with compat packages, instead of hiding them in bundled libs
- aggressive leverage of upstream unit tests to detect quickly broken code.
Will not work in general due to API backwards incompatibility issues.
I have some doubt that code that fails unit tests is actually something that can be used (and if it can, not be used why are you shipping it). Sure, the unit test may be wrong, but that's not the usual case. Better to detect, check if another package does not need updating, and report upstream otherwise. We had quite a lot of complex builds that were failing, and got unstuck when someone tried to run unit tests, and updated the components they were pointing to.
no reliance on external utilities to compute code requirements. No more dependencies that do not match the shipped Go code.
Please, rephrase or remove the point as for the CGO it is useful to have external utility that can list all imported C header files and try to find a set of packages that provide the header files (and corresponding libraries).
I'm not saying other things are not useful, but I *have* seen quite a few Fedora packages that were declaring garbage deps, because the external tool was run on another code state and the requirements were never corrected.
I am surprised you are able to do that without any API issues. Have you tried to de-bundle and build Kubernetes, Docker, Etcd, Prometheus and other big projects from the same set of Go packages? At some point you have/had to switch to compat packages or bundling.
Given the state of upstream projects debundling absolutely requires some compat packages. Which is why I documented how to create them easily, even if Jakub disapproves. I'll take compat packages over stealth bundling any day.
gives up on many conventions of current Fedora Go packaging, as they were an obstacle to the target packaging efficiency.
(Still part of the Limitations): Can you make a list of conventions you consider as obstacles? I would like to comment on each of them.
All the very complex layers upon layers of variables, ifdefing and so on that will completely drain your energy before doing any serious change and often differ in subtle ways from the conventions of the rest of the distro. I tried to keep the amount of boilerplate, variables and conditionals minimal, and to apply official Fedora patterns.
https://fedoraproject.org/wiki/More_Go_packaging#Excluding_testing_in_specif... %gochecks . transport/http transport/grpc transport option internal integration-tests/storage examples
The excluding mechanism is not intuitive,
opt-out is not intuitive but if you don't do opt-out people will select a few directories to test, give up as soon as the list is too long, and certainly never refresh on update opt-out is more robust in presence of humain limitations. And the default should be that everything is tested.
As soon as you integrate the opt-out nature of the macro, it's really easy to understand and use. And you certainly do not need to inflict yourself and others one or more exclusion switches that will need to be used every time.
https://fedoraproject.org/wiki/More_Go_packaging#Handling_API_changes API changes may require the creation of Go code compatibility packages
I would not recommended doing that unless you have an army of packagers.
As you wrote yourself, it's that or bundling. And bundling is the same misery without the help of rpm tooling and without putting the result in a package that can be reused in other projects that encountered the same API break.
a packager, that identifies an API change in his Go package, MUST notify golang@fedoraproject.org at least a week before pushing his changes to any release (and those releases SHOULD include fedora-devel). This grace period can be waived by FPC for security reasons.
From the point of view of updating popular projects like Etcd, Kubernetes, Prometheus, etc. this is highly impractical. Usually, there are more dependencies to update. If I will have to wait one week to update etcd, than another week to push its update to stable branch, then I will not really be effective in updating.
I'm not wed to this rule but removing it means instant breakage for all the stuff that relied on the old version. Which is not nice in a community project. Now if all Fedora Go packagers can live with it, I won't push it more.
a packager, that identifies an unannounced API change in another Go package, MUST notify golang@fedoraproject.org at once. those notices SHOULD be copied to the maintainers of packages affected by the change, when they are known.
This is going to be a regular thing unless a tooling detecting this changes is available.
Is it better to silently wait for the rush of mass rebuild breakage? Not That I would't appreciate some tooling (automated rebuild of dependant packages + notification of breakage would be nice)
a packager, that identifies that the code he packages, is broken by API changes in another project, SHOULD notify politely the upstream of the broken project of the API change, if he can not ascertain it is already aware of the change.
Some upstream Go projects are responsive, some are not. A lot of Go projects are forked, abandoned or inactive. Or the upstream is a single person that does not have time to deal with distribution specific guidelines. But I appreciate the effort to spread the good practices. However, sometimes the API change is intentional cause there is no other way.
Note that the guideline write to notify the project that was broken by the change, not the reverse. For non-dead projects they'll have to deal with the breakage sooner or later, better notify them as soon as possible so they can plan how to manage the break.
Usual workflow of updating Go packages:
Pretty much what I understood
## What we need
- automatic updates of Go packages (each time there is a new release or
on demand to a specific commit)
That should be quite easy to do with the new style of packaging as long as there is no huge project restructuring. The hard part is compat packages. BuildRequires could also be annoying as long as they are not fully automated (but, it is quite easy to scrap the build log for 'could not find foo package' messages and add the corresponding BR if already packaged)
- no build-time dependencies (we can not discover missing dependencies
before installation)
You can not discover deps before the package is fully built but once built it does declare all the stuff it needs and it is possible to check that against repo metadata.
- always generating all provided packages that are available (I don't
want to provide all packages always, e.g. need just a subset to build my project, the remaining provided packages need new Go projects which adds unnecessary complexity)
More accurately, you can not package Go code, and publish it to others, without the dep it needs. You may choose to package more deps, or to publish less code. You can rely on the code other packaged having the required dependencies (at least at the import level, version constrain level need progress upstream on the go dep front)
- as jcajka pointed out, it is not easy to debug the lua based scripts
Yes debugging lua is not the most pleasant thing, on the other hand the language is so limited, once code is written and worked once it is very unlikely to break in the future due to unrelated updates. It is much more robust towards changes than shell or other macro code.
- combination of bundled and un-bundled dependencies (sometimes I just
need to use one or two bundled dependencies because without generating a tune of compat packages)
That certainly works today (for whatever definition of "works" that encompasses bundling in general)
In all cases, the new approach is strongly focused on specfile simplification.
That was its primary objective.
Even if we create a lot of compat packages we still need a mechanism that can tell us when a given compat package is no longer needed.
Sure, however it is trivial to switch from compat to tip in buildrequires and try the result.
I am working on a tooling (in my free time) that will provide that information (https://github.com/gofed/symbols-extractor, feel free to contribute to the project with ideas who we need to detect or extract from the code to make more spec file pieces generic and macro-replaceable).
Thank you, added to the long list of things I need to look at.
At the end, even if we do "create as many compat Go packages as needed" or "let's use vendored packages because we can", we will still need to maintain entire ecosystem in some reasonable way.
They are two faces of the same problem as long as upostreams are not careful API-wise and we lack the manpower to fix every Go project every time its deps change.
From my point of view, the new Go packaging guidelines make it easier for fresh Go packager to get familiar with Go packaging practices and to quickly package his/her Go project.
I hope that will help new people to join, there are two few Fedora Go packagers for the amount of code that would be interesting to package and maintain.
However, from the point of view of entire Go ecosystem (or any micro ecosystem = I have more projects building from the same dependencies), it is harder to tune individual packages for the mutual benefit.
I'm not sure why you feel that way, the new guidelines make dependency breakage evident, they do not create it.
So please, pay at least some respect to all who spent their time and dreamless nights coming with quick solutions.
I don't think I've ever criticized anyone? I do criticize the current status but that has nothing to do with the people involved, given the amount of needed work and the ridiculously small Fedora Go packager pool.
I can't justify all the proposed the changes, or the huge amount of work they required, or the amount of work it will take to convert the existing package pool, by pretending the current status is good. I quite like Fedora but I have better ways to spend my work or free time if there is nothing to fix.
I also suggest to remove every comparison with the Go guidelines draft and the current state of all Go spec files.
Why exactly should FPC spend a lot of gruelling review time on the new guidelines if they improve nothing compared to the current state? Their time is not free either. The pros/cons section is just there to explain FPC why we are sucking their precious time, I don't see the point or use of keeping it post-approval. I must say, I don't feel FPC very willing to look at Go in Fedora, if they're not promised some changes.
I understand it is not pleasant to see some of those things written down, but not writing them will not cause them to cease to exist. Getting this stuff implemented, approved, and applied, *will* cause a lot of them to cease to exist. And convincing FPC there are significant gains to be had is a way to get the approval faster, and make all of this a thing of the past.
And of course, it's just a step, someone else will find other things to fix, and his solution will be better than this one, because it will build on this one, and that will be perfectly fine, and I won't insist this state can not be improved.
There are other parts of the new guidelines I could contradict. However, doing that we can end up in endless discussion, annoying each other and burning time that we can use to do important things instead.
Improving documentation is useful, but I'm quite sick of it right now, let's move to other things :)
But please, keep in mind, you are not the only one involved in this effort.
I certainly hope so, there would be little point spending all this time answering questions and writing wiki pages otherwise!
We only do what we can in the time given :).
That is so true… I don't think I can continue stretching it very long either.
Let's combine the results, our experience. Let's open a google document (or different collaboration solution if you have one in mind) where we collect all our requirements.
Let's ask other Go (or wannabe) packagers what are their requirement, what they see as drawbacks of the current and new guidelines, who is willing to help with the packaging and maintenance, let's ask more questions.
Well you won't entice me to write even more things near term, and given how few people replied to this thread, even though it was copied everywhere, I'm not sure a lot of people would be interested either. I'd say they probably need to see some visible advancement to commit their efforts there. I certainly need some visible advancement to continue.
I'd be delighted to be wrong of course.
I am happy to take the tooling side.
More tooling is always good!
However there will be a need for a general package baseline refresh or it will keep not interesting many people.
As I mentioned in the ticket (https://pagure.io/packaging-committee/issue/382), I shifted my focus on building tooling that do a lot of analysis and detection which are out of scope of any guidelines. Even if we agree on the final form of the guidelines, we still need tooling that will help us with the packaging. E.g. detection of API backwards incompatibilities, spec file generators (enhancing and improving over time), detection of go packages that are no longer needed, running CI/CD, etc.
The nice thing about fixing one problem, is that it makes fixing other problems possible.
Best regards
De: "nicolas mailhot" À: "Jan Chaloupka"
I mentioned a list of things that you did not answer fully. Most important to me:
- your macros do not generate build-time dependencies, which I see as
one of the drawbacks.
Generating BuildRequires dynamically needs changes in rpm tooling (mostly mock, I think). So it can not be done today if I'm not wrong. If you have a tool that outputs BuildRequires outside rpm, you can still use it. That's not an absolute showstopper as the go compiler is pretty explicit on what's missing, so BuildRequires are easily added during the first %build or %check run (though that's tedious and annoying).
Do you plan to ignore them?
Generating BuildRequires would be quite easy if rpm tooling provided an entry point at the end of %prep. So, that's a mid-term objective and requires sending an RFE mock-side first.
And that part is done:
https://github.com/rpm-software-management/mock/issues/160
Feel free to complete/correct/support the request so it has a change to be taken into account and we can start working on integrated Go BuildRequires
Regards
golang@lists.fedoraproject.org