Hi,
I am proposing for inclusion a set of rpm technical files aimed at automating the packaging of forge-hosted projects.
- Packaging draft: https://fedoraproject.org/wiki/More_Go_packaging - https://pagure.io/packaging-committee/issue/734 - go-srpm-macros RFE with the technical files: https://bugzilla.redhat.com/show_bug.cgi?id=1526721
This proposal is integrated with and depends on the https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft It builds on the hard work of the Go SIG and reuses the rpm automation of https://fedoraproject.org/wiki/PackagingDrafts/Go when it exists, and produces compatible packages.
What it does:
- drastically shorter spec files, up to 90% in some cases, often removing hundreds of lines per spec. - simple, packager-friendly spec syntax - automated package naming derived from the native identifier (import path). No more packages names without any relation with current upstream naming. - working Go autoprovides. No forgotten provides anymore. - working Go autorequires. No forgotten requires anymore. - strict automated directory ownership (used by autorequires and autoprovides). - centralized computation of source URLs (via Forge-hosted projects packaging automation). No more packages lacking guidelines. No more broken guidelines no one notices. - easy switch between commits, tags and releases (via Forge-hosted projects packaging automation). No more packages stuck on commits when upstream starts releasing. - guidelines-compliant automated snapshot naming, including snapshot timestamps (via Forge-hosted projects packaging automation). No more packages stuck in 2014. - guidelines-compliant bootstraping. - systematic use of the Go switches defined by the Go maintainer. Easy to do changes followed by a mass rebuild. - flexibility, do the right thing transparently by default, leave room for special cases and overrides. - no bundling (a.k.a. vendoring) due to the pain of packaging one more Go dependency. - centralized Go macros that can be audited and enhanced over time. - aggressive leverage of upstream unit tests to detect quickly broken code.
Please consult packaging draft for full information.
The proposal has been tested in Rawhide and EL7 over a set of ~ 140 Go packages. This set is a mix of current Fedora packages, bumped to a more recent version, rewrites of Fedora packages, and completely new packages.
I hope posting the second part of the automation will answer some questions people had on the https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft
Regards,
On Sun, Dec 17, 2017 at 2:11 AM, nicolas.mailhot@laposte.net wrote:
Hi,
I am proposing for inclusion a set of rpm technical files aimed at automating the packaging of forge-hosted projects.
- Packaging draft: https://fedoraproject.org/wiki/More_Go_packaging
- https://pagure.io/packaging-committee/issue/734
- go-srpm-macros RFE with the technical files: https://bugzilla.redhat.com/show_bug.cgi?id=1526721
This proposal is integrated with and depends on the https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft It builds on the hard work of the Go SIG and reuses the rpm automation of https://fedoraproject.org/wiki/PackagingDrafts/Go when it exists, and produces compatible packages.
What it does:
- drastically shorter spec files, up to 90% in some cases, often removing hundreds of lines per spec.
- simple, packager-friendly spec syntax
- automated package naming derived from the native identifier (import path). No more packages names without any relation with current upstream naming.
- working Go autoprovides. No forgotten provides anymore.
- working Go autorequires. No forgotten requires anymore.
- strict automated directory ownership (used by autorequires and autoprovides).
- centralized computation of source URLs (via Forge-hosted projects packaging automation). No more packages lacking guidelines. No more broken guidelines no one notices.
- easy switch between commits, tags and releases (via Forge-hosted projects packaging automation). No more packages stuck on commits when upstream starts releasing.
- guidelines-compliant automated snapshot naming, including snapshot timestamps (via Forge-hosted projects packaging automation). No more packages stuck in 2014.
- guidelines-compliant bootstraping.
- systematic use of the Go switches defined by the Go maintainer. Easy to do changes followed by a mass rebuild.
- flexibility, do the right thing transparently by default, leave room for special cases and overrides.
- no bundling (a.k.a. vendoring) due to the pain of packaging one more Go dependency.
- centralized Go macros that can be audited and enhanced over time.
- aggressive leverage of upstream unit tests to detect quickly broken code.
Please consult packaging draft for full information.
The proposal has been tested in Rawhide and EL7 over a set of ~ 140 Go packages. This set is a mix of current Fedora packages, bumped to a more recent version, rewrites of Fedora packages, and completely new packages.
I hope posting the second part of the automation will answer some questions people had on the https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft
I really do like this. There are only two issues I have with it:
1. This seems to mandate that all packages must be named by their import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
2. Mandating a forge is going to be tricky for self-hosted stuff, or people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
I really do like this. There are only two issues I have with it:
- This seems to mandate that all packages must be named by their
import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
- Mandating a forge is going to be tricky for self-hosted stuff, or
people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
By not using the macros for packages not fitting the model?
I think this is very helpful especially when it's the common practice, and I certainly won't blame anyone doing proper releases and not just a git tag with github releases notes ;)
Regarding naming, I think python packages must be prefixed with python[23]- and can Provides: the upstream project name. On the other hand we have packages like docker that are clearly named after upstream's name, so I don't think that would be a problem for snapd. (and maybe an exception needs to be granted?)
Dridi
On Mon, Jan 22, 2018 at 8:33 AM, Dridi Boukelmoune dridi.boukelmoune@gmail.com wrote:
I really do like this. There are only two issues I have with it:
- This seems to mandate that all packages must be named by their
import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
- Mandating a forge is going to be tricky for self-hosted stuff, or
people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
By not using the macros for packages not fitting the model?
The issue is that the new Go macros are tightly wound into the forge macros. I just want to be sure that we can leverage things like the dependency generators without all the other stuff.
I think this is very helpful especially when it's the common practice, and I certainly won't blame anyone doing proper releases and not just a git tag with github releases notes ;)
Regarding naming, I think python packages must be prefixed with python[23]- and can Provides: the upstream project name. On the other hand we have packages like docker that are clearly named after upstream's name, so I don't think that would be a problem for snapd. (and maybe an exception needs to be granted?)
This rule only applies to Python packages that have modules that are designed to be imported by other Python code. Otherwise, this is not necessary.
On Mon, Jan 22, 2018 at 2:45 PM, Neal Gompa ngompa13@gmail.com wrote:
On Mon, Jan 22, 2018 at 8:33 AM, Dridi Boukelmoune dridi.boukelmoune@gmail.com wrote:
I really do like this. There are only two issues I have with it:
- This seems to mandate that all packages must be named by their
import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
- Mandating a forge is going to be tricky for self-hosted stuff, or
people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
By not using the macros for packages not fitting the model?
The issue is that the new Go macros are tightly wound into the forge macros. I just want to be sure that we can leverage things like the dependency generators without all the other stuff.
I think this is very helpful especially when it's the common practice, and I certainly won't blame anyone doing proper releases and not just a git tag with github releases notes ;)
Regarding naming, I think python packages must be prefixed with python[23]- and can Provides: the upstream project name.
I'm not sure if this matters in this discussion but an example Python3 part of a spec file https://fedoraproject.org/wiki/Packaging:Python to accommodate also EPEL (which on CentOS7 prefixes Python3 packages with python34 and not python3) would look like:
%package -n python%{python3_pkgversion}-%{pname} %{?python_provide:%python_provide python%{python3_pkgversion}-%{pname}}
Macin
On the
other hand we have packages like docker that are clearly named after upstream's name, so I don't think that would be a problem for snapd. (and maybe an exception needs to be granted?)
This rule only applies to Python packages that have modules that are designed to be imported by other Python code. Otherwise, this is not necessary.
-- 真実はいつも一つ!/ Always, there's only one truth! _______________________________________________ packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org
----- Original Message -----
From: "Marcin Dulak" marcin.dulak@gmail.com To: "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Cc: golang@lists.fedoraproject.org, "Development discussions related to Fedora" devel@lists.fedoraproject.org Sent: Monday, January 22, 2018 4:04:19 PM Subject: Re: [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
On Mon, Jan 22, 2018 at 2:45 PM, Neal Gompa < ngompa13@gmail.com > wrote:
On Mon, Jan 22, 2018 at 8:33 AM, Dridi Boukelmoune < dridi.boukelmoune@gmail.com > wrote:
I really do like this. There are only two issues I have with it:
- This seems to mandate that all packages must be named by their
import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
- Mandating a forge is going to be tricky for self-hosted stuff, or
people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
By not using the macros for packages not fitting the model?
The issue is that the new Go macros are tightly wound into the forge macros. I just want to be sure that we can leverage things like the dependency generators without all the other stuff.
I think this is very helpful especially when it's the common practice, and I certainly won't blame anyone doing proper releases and not just a git tag with github releases notes ;)
Regarding naming, I think python packages must be prefixed with python[23]- and can Provides: the upstream project name.
I'm not sure if this matters in this discussion but an example Python3 part of a spec file https://fedoraproject.org/wiki/Packaging:Python to accommodate also EPEL (which on CentOS7 prefixes Python3 packages with python34 and not python3) would look like:
%package -n python%{python3_pkgversion}-%{pname} %{?python_provide:%python_provide python%{python3_pkgversion}-%{pname}}
Macin
Hopefully something like this will never happen as generally I'm strongly against shipping multiple versions(of one implementation) of Go concurrently.
JC
On the
other hand we have packages like docker that are clearly named after upstream's name, so I don't think that would be a problem for snapd. (and maybe an exception needs to be granted?)
This rule only applies to Python packages that have modules that are designed to be imported by other Python code. Otherwise, this is not necessary.
-- 真実はいつも一つ!/ Always, there's only one truth! _______________________________________________ packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
----- Mail original ----- De: "Neal Gompa"
On Mon, Jan 22, 2018 at 8:33 AM, Dridi Boukelmoune
I really do like this. There are only two issues I have with it:
- This seems to mandate that all packages must be named by their
import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
- Mandating a forge is going to be tricky for self-hosted stuff, or
people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
By not using the macros for packages not fitting the model?
The issue is that the new Go macros are tightly wound into the forge macros. I just want to be sure that we can leverage things like the dependency generators without all the other stuff.
Hi Neal,
I should probably not let this pass without clarifying:
1. if your concern is that the *forge* macros will block packaging of projects not hosted on known forges, they won't: — they include code to disable transparently and without exploding the associated bits if the hosting site is unknown — the packager can pre-define the variables that would have been computed if the site was known and all the rest of the automation will just work — baring that the packager can ignore the site-related computations and use whatever he wants instead in Source: and *setup, which are the only parts that depend on hosting structure knowledge,
2. if your concern is that the *forge* macros are defective somewhere I'd be curious where as you'd be the first to report an actual technical problem. I've used them intensively in rawhide and el7 with many different rpm tools and they are rock solid for me
3. if your concern is that they won't ever be merged due to some people non-constructive obstruction, well I share it to some point but I won't spend more time arguing unicorns and what-if-maybe I-don't-like-commits-let's-make-them-awful-to-use. The people blocking there are free to take up the maintenance of my ~400 specs and redo them as they want, we'll see how they love to waste tens of minutes per spec to position setup args manually when they have to do it for hundreds of them, every time there is a package update or a forge relayout.
The main difficulty with the proposal is not the *forge part, it's that automated autodeps are well, automated. As every automation they are strict and unyielding and do not take approximations well. They will fail spectacularly in the following cases:
1. the packager does not package all the deps of its code → the resulting package is uninstallable because autodeps add requires on missing bits
2. the packager packages code with garbage imports (very common in example or test code) → the resulting package is uninstallable because autodeps add require
3. the packager does not own properly the directories where its Go code is installed → given that Go deps depend on directories, Go autodeps also depend on proper directory ownership. Positioning autodeps on Go files themselves would make package build times increase many many times as rpm would try to compute the same autodeps for every single .go file in a directory. (and Go autodeps are already sloooow as snails because invoking the go command is slow)
All those things are mitigated by the use of %goinstall that filters usual example/test/vendor dirs so they don't trigger autodeps, and tries very hard to own all relevant directories.
For those reasons I don't propose to activate autodeps in old-style golang packages. They need conversion (and review by a human to check no problem code is deployed) first.
Regards,
On Jan 23, 2018 12:41, nicolas.mailhot@laposte.net wrote:
----- Mail original ----- De: "Neal Gompa"
On Mon, Jan 22, 2018 at 8:33 AM, Dridi Boukelmoune
I really do like this. There are only two issues I have with it:
- This seems to mandate that all packages must be named by their
import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
- Mandating a forge is going to be tricky for self-hosted stuff, or
people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
By not using the macros for packages not fitting the model?
The issue is that the new Go macros are tightly wound into the forge macros. I just want to be sure that we can leverage things like the dependency generators without all the other stuff.
Hi Neal,
I should probably not let this pass without clarifying:
1. if your concern is that the *forge* macros will block packaging of projects not hosted on known forges, they won't: — they include code to disable transparently and without exploding the associated bits if the hosting site is unknown — the packager can pre-define the variables that would have been computed if the site was known and all the rest of the automation will just work — baring that the packager can ignore the site-related computations and use whatever he wants instead in Source: and *setup, which are the only parts that depend on hosting structure knowledge,
2. if your concern is that the *forge* macros are defective somewhere I'd be curious where as you'd be the first to report an actual technical problem. I've used them intensively in rawhide and el7 with many different rpm tools and they are rock solid for me
3. if your concern is that they won't ever be merged due to some people non-constructive obstruction, well I share it to some point but I won't spend more time arguing unicorns and what-if-maybe I-don't-like-commits-let's-make-them-awful-to-use. The people blocking there are free to take up the maintenance of my ~400 specs and redo them as they want, we'll see how they love to waste tens of minutes per spec to position setup args manually when they have to do it for hundreds of them, every time there is a package update or a forge relayout.
The main difficulty with the proposal is not the *forge part, it's that automated autodeps are well, automated. As every automation they are strict and unyielding and do not take approximations well. They will fail spectacularly in the following cases:
1. the packager does not package all the deps of its code → the resulting package is uninstallable because autodeps add requires on missing bits
2. the packager packages code with garbage imports (very common in example or test code) → the resulting package is uninstallable because autodeps add require
3. the packager does not own properly the directories where its Go code is installed → given that Go deps depend on directories, Go autodeps also depend on proper directory ownership. Positioning autodeps on Go files themselves would make package build times increase many many times as rpm would try to compute the same autodeps for every single .go file in a directory. (and Go autodeps are already sloooow as snails because invoking the go command is slow)
All those things are mitigated by the use of %goinstall that filters usual example/test/vendor dirs so they don't trigger autodeps, and tries very hard to own all relevant directories.
For those reasons I don't propose to activate autodeps in old-style golang packages. They need conversion (and review by a human to check no problem code is deployed) first.
So, if I understand correctly, both the forge stuff and the new macros for go packaging are completely opt-in? If that's correct, this looks like the best solution to me - as old packages can then be converted one at a time (which I am looking forward to doing for my go packages, btw.).
Fabio
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
----- Mail original ----- De: "Fabio Valentini"
So, if I understand correctly, both the forge stuff and the new macros for go packaging are completely opt-in? If that's correct, this looks like the best solution to me - as old packages can then be converted one at a time (which I am looking forward to doing for my go packages, btw.).
It is fully opt-in. You can also use some parts and ignore others (though it's probably more complex than just using all of it unless you understand the new parts really well).
However I should warn that packaging "new-style" something is likely to force conversion to "new-style" at least some of its deps."Old-style" manual declaration of provides often forgets elements and autodeps have no mercy on missing provides. But, some of the "new-style" specs I did depend on existing Fedora packages I didn't have to touch.
Now that the non-Go part in redhat-rpm-macros is merged in devel I'll try to do a clean PR on go-srpm-macros.
Then once Jan or Jakub accepts it it will be possible to play with the automation in devel and I'll be able to share my specs somewhere (not that the work is finished, but some parts *are* finished and I'd rather have the people who use those packages to review my conversions rather than redo part of them without sharing existing work)
Regards,
----- Mail original ----- De: "nicolas mailhot"
Now that the non-Go part in redhat-rpm-macros is merged in devel I'll try to do a clean PR on go-srpm-macros.
Then once Jan or Jakub accepts it it will be possible to play with the automation in devel and I'll be able to share my specs somewhere (not that the work is finished, but some parts *are* finished and I'd rather have the people who use those packages to review my conversions rather than redo part of them without sharing existing work)
And the PR is done and sent:
https://src.fedoraproject.org/rpms/go-srpm-macros/pull-request/1
Now I only need to revisit https://fedoraproject.org/wiki/More_Go_packaging
and enhance it with the packaging patterns that emerged after applying the proposal to the last few hundreds of Go spec files.
Regards,
De: "Neal Gompa"
The issue is that the new Go macros are tightly wound into the forge macros. I just want to be sure that we can leverage things like the dependency generators without all the other stuff.
Hi Neal,
I should probably not let this pass without clarifying: 3. if your concern is that they won't ever be merged
And the forge macros are now available since redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream renaming the file). Heartfelt thanks to Jason Tibbitts !
I'll focus on pushing the Go parts to go-srpm-macros now so everyone can play with them
Regards,
I wish this message wasn't crossposted everywhere, but I don't want to lose any discussion by trimming the CC list. Sorry if replies generate bounces for some.
"nm" == nicolas mailhot nicolas.mailhot@laposte.net writes:
nm> And the forge macros are now available since nm> redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream nm> renaming the file). Heartfelt thanks to Jason Tibbitts !
Please don't forget to let me know when it's time to start thinking about pushing this down to F27. And maybe F26. And as far as I can tell it should work with only minor modification in EPEL7 (via epel-rpm-macros). I don't know about EPEL6, but we really should look at it given some of the other discussions about specfile compatibility. Some packagers wouldn't ever use it if it doesn't work everywhere.
Finally, we should also talk about whether there is any integration or automation possible between fedpkg and specfiles configured with these macros.
- J<
----- Mail original ----- De: "Jason L Tibbitts III"
"nm" == nicolas mailhot nicolas.mailhot@laposte.net writes:
nm> And the forge macros are now available since nm> redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream nm> renaming the file). Heartfelt thanks to Jason Tibbitts !
Please don't forget to let me know when it's time to start thinking about pushing this down to F27. And maybe F26. And as far as I can tell it should work with only minor modification in EPEL7 (via epel-rpm-macros).
I don't know about EPEL6, but we use it as-is in EL7 and it works just as well (except maybe for the %autosetup bits but IIRC that's autosetup which is broken in EL7). In fact it has probably been used more heavily in EL7 than in fedora-devel so far. Maybe it also works in EPEL 6 but I've never tried it. I guess it depends mostly on the level of lua support in EL6 rpm and rpm-related tools now the forge macro code is lua only. I'm pretty sure many of the problems in the early versions of the macro were due to non-lua code and its interactions with lua code once the lua-ification started.
It's a good idea to let people play with it in fedora-devel maybe a month, in case I missed something, but from a technical POW I'm prety sure it could be merged up to EL7 now. I'll submit fedora-devel specific tweaks later (just like I submitted bitkeeper.org support today), right now the code is distro-agnostic. For my part I doubt I'll ever use it in EL6 since I did it for Go and the EL6 Go stack is really too old for a merge to be interesting. Anyway I'll certainly let you know when I feel the time is right (but do not block on me!)
Finally, we should also talk about whether there is any integration or automation possible between fedpkg and specfiles configured with these macros.
I'm afraid my knowledge of recent fedpkg enhancements is too sparse to be of any use there. Though I'm not opposed to the idea at all.
Thanks again!
"nm" == nicolas mailhot nicolas.mailhot@laposte.net writes:
nm> I don't know about EPEL6, but we use it as-is in EL7 and it works nm> just as well (except maybe for the %autosetup bits but IIRC that's nm> autosetup which is broken in EL7).
I had ported autosetup to EPEL6 and then at the next release the macros showed up (without any discussion) in base RHEL6. So they should be there. I'm not entirely sure what's actually broken about %autosetup in EL7, though; I hadn't heard about breakage before this. epel-rpm-macros could conceivably carry %epel_autosetup or something which contains fixes. I mostly understand the internals of %autosetup so maybe there's something I can do.
nm> Maybe it also works in EPEL 6 but I've never tried it. I guess it nm> depends mostly on the level of lua support in EL6 rpm and nm> rpm-related tools now the forge macro code is lua only.
I think it would be worth a try. In my experience not all that much has changed in the Lua interfaces since RHEL6 and even RHEL5. (EL5 just didn't have sources and patches in the Lua namespace.)
nm> For my part I doubt I'll ever use it in EL6 since I did it for Go nm> and the EL6 Go stack is really too old for a merge to be nm> interesting.
Well, sure, Go on EL6 is probably out but these macros were at least presented as being far more general, and I'm sure there are plenty of EPEL6 packages which could benefit. Potential anything that packages a git snapshot of something.
nm> I'm afraid my knowledge of recent fedpkg enhancements is too sparse nm> to be of any use there. Though I'm not opposed to the idea at all.
It's just worth a brainstorm, I think. I can imagine that loads of people would love it if fedpkg could just auto-bump the bits necessary to update a package to today's git head or some specific commit or something like that. Before we didn't really have a good standard way to format a spec when you're using a checkout. Now....
Doesn't necessarily have to start in fedpkg, either. A standalone utility for doing a couple of things like that could be useful as a prototype.
- J<
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: "Jason L Tibbitts III" tibbs@math.uh.edu Cc: golang@lists.fedoraproject.org, "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Tuesday, January 23, 2018 9:28:15 PM Subject: Re: Proposed Fedora packaging guideline: More Go packaging
----- Mail original ----- De: "Jason L Tibbitts III"
"nm" == nicolas mailhot nicolas.mailhot@laposte.net writes:
nm> And the forge macros are now available since nm> redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream nm> renaming the file). Heartfelt thanks to Jason Tibbitts !
Please don't forget to let me know when it's time to start thinking about pushing this down to F27. And maybe F26. And as far as I can tell it should work with only minor modification in EPEL7 (via epel-rpm-macros).
I don't know about EPEL6, but we use it as-is in EL7 and it works just as well (except maybe for the %autosetup bits but IIRC that's autosetup which is broken in EL7). In fact it has probably been used more heavily in EL7 than in fedora-devel so far. Maybe it also works in EPEL 6 but I've never tried it. I guess it depends mostly on the level of lua support in EL6 rpm and rpm-related tools now the forge macro code is lua only. I'm pretty sure many of the problems in the early versions of the macro were due to non-lua code and its interactions with lua code once the lua-ification started.
It's a good idea to let people play with it in fedora-devel maybe a month, in case I missed something, but from a technical POW I'm prety sure it could be merged up to EL7 now. I'll submit fedora-devel specific tweaks later (just like I submitted bitkeeper.org support today), right now the code is distro-agnostic. For my part I doubt I'll ever use it in EL6 since I did it for Go and the EL6 Go stack is really too old for a merge to be interesting. Anyway I'll certainly let you know when I feel the time is right (but do not block on me!)
If we are talking about EPEL6 stack, it is fairly fresh(1.9.2) and stable(it will be on 1.9 for whole of its upstream support), although Go packaging macros are missing.
JC
Finally, we should also talk about whether there is any integration or automation possible between fedpkg and specfiles configured with these macros.
I'm afraid my knowledge of recent fedpkg enhancements is too sparse to be of any use there. Though I'm not opposed to the idea at all.
Thanks again!
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
----- Mail original ----- De: "Jakub Cajka"
Hi Jakub,
For my part I doubt I'll ever use it in EL6 since I did it for Go and the EL6 Go stack is really too old for a merge to be interesting. Anyway I'll certainly let you know when I feel the time is right (but do not block on me!)
If we are talking about EPEL6 stack, it is fairly fresh(1.9.2) and stable(it will be on 1.9 for whole of its upstream support), although Go packaging macros are missing.
Yes the core golang package is there but first the other Go macros are not available in EL6 and second the software level of many common Go packages is very old in EL6.
So, assuming the forge macros just work in EL6, because the little bit of rpm lua they need works the same as in EL7 and devel, updating EL6 to the level needed for a Go spec to be shared mostly unchanged between EL6/EL7 and devel would require:
— making the other Go macros available in EL6 (and I'm far from sure they would work unchanged in EL6, the forge part is easy since it's 100% lua but the other parts exercise the shell and rpm and other stuff that may behave in a slightly different way in such an ancient codebase. Not necessarily a show-stopper but definitely something that requires testing and adjusting time by someone)
— updating all the common Go software packages to the same level as in devel. I don't say it can't be done (after all I'm doing it for EL7 on hundreds of packages) but that's a *lot* more work than just updating a few macro files. Especially if it hits bootstraping issues not existing in EL7 and devel.
Therefore, it requires someone with time and motivation, and should really not be attempted before EL7 and devel are done and things have settled a bit, and by that point will there still be enough interest in upgrading the Go state in EL6 for it to be worth the pain?
Of course one could limit oneself to making the macros available (which still requires some testing and eventually some code porting), that would enable using the same Go spec coding style in EL6 EL7 and devel, if not sharing the spec themselves, but the interest of autodeps is severely limited, if you do not have a large baseline of packages providing the deps (and the dep version) other packages need to build.
That's why I wrote the forge part could be made available in EL6, it depends on little except the lua built in rpm, and can be useful as-is, while the Go part is something else entirely, as its utility is directly linked to the quantity and freshness of Go software packages in the distro.
Regards,
Hi Jason,
"nm" == nicolas mailhot wrote:
Jason L Tibbitts wrote:
nm> And the forge macros are now available since nm> redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream nm> renaming the file). Heartfelt thanks to Jason Tibbitts ! Please don't forget to let me know when it's time to start thinking about pushing this down to F27. And maybe F26. And as far as I can tell it should work with only minor modification in EPEL7 (via epel-rpm-macros).
I don't know about EPEL6, but we use it as-is in EL7 and it works just as well (except maybe for the %autosetup bits but IIRC that's autosetup which is broken in EL7).
It's a good idea to let people play with it in fedora-devel maybe a month, in case I missed something, but from a technical POW I'm prety sure it could be merged up to EL7 now.
Now that the appropriate observation time has passed, and people have not find new problems (in fact they found scenarii that worked I had not thought about) can you please merge
https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/18
and push it to stable?
The Go macros need it and Fedora Go maintainers now want to fork forgemeta if redhat-rpm-config does not make forgemeta available in the Fedora stable versions they target, which would all make a huge mess. I guess it's been too successful ;).
Regards,
----- Original Message -----
From: "Jason L Tibbitts III" tibbs@math.uh.edu To: "nicolas mailhot" nicolas.mailhot@laposte.net Cc: golang@lists.fedoraproject.org, "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Tuesday, January 23, 2018 6:00:24 PM Subject: Re: Proposed Fedora packaging guideline: More Go packaging
I wish this message wasn't crossposted everywhere, but I don't want to lose any discussion by trimming the CC list. Sorry if replies generate bounces for some.
"nm" == nicolas mailhot nicolas.mailhot@laposte.net writes:
nm> And the forge macros are now available since nm> redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream nm> renaming the file). Heartfelt thanks to Jason Tibbitts !
Please don't forget to let me know when it's time to start thinking about pushing this down to F27. And maybe F26. And as far as I can tell it should work with only minor modification in EPEL7 (via epel-rpm-macros). I don't know about EPEL6, but we really should look at it given some of the other discussions about specfile compatibility. Some packagers wouldn't ever use it if it doesn't work everywhere.
I think that it would be great to land it also in the EPEL6/7.
JC
Finally, we should also talk about whether there is any integration or automation possible between fedpkg and specfiles configured with these macros.
- J<
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On Tue, Jan 23, 2018 at 5:45 AM, nicolas.mailhot@laposte.net wrote:
----- Mail original ----- De: "Neal Gompa"
On Mon, Jan 22, 2018 at 8:33 AM, Dridi Boukelmoune
I really do like this. There are only two issues I have with it:
- This seems to mandate that all packages must be named by their
import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
- Mandating a forge is going to be tricky for self-hosted stuff, or
people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
By not using the macros for packages not fitting the model?
The issue is that the new Go macros are tightly wound into the forge macros. I just want to be sure that we can leverage things like the dependency generators without all the other stuff.
Hi Neal,
I should probably not let this pass without clarifying:
- if your concern is that the *forge* macros will block packaging of projects not hosted on known forges, they won't:
— they include code to disable transparently and without exploding the associated bits if the hosting site is unknown — the packager can pre-define the variables that would have been computed if the site was known and all the rest of the automation will just work — baring that the packager can ignore the site-related computations and use whatever he wants instead in Source: and *setup, which are the only parts that depend on hosting structure knowledge,
This was my main concern. There are plenty of people who don't like any of the known forges. And Go ecosystem incompetence for mandating URI-style deps and build paths doesn't really stop people from wanting to do that.
- if your concern is that the *forge* macros are defective somewhere I'd be curious where as you'd be the first to report an actual technical problem. I've used them intensively in rawhide and el7 with many different rpm tools and they are rock solid for me
I don't consider them defective. They make me a bit squeamish because I don't see the spec preamble and such because they're autogenerated, but you've mentioned that it's possible to selectively use these things with Go and forge macros.
SUSE does something similar with Python for their "singlespec" part, and it's fantastic[1][2]. But I'm slightly uncomfortable with the idea that there's no preamble *at all* in the spec using the Go macros and it's generated fully by macros.
[1]: https://github.com/openSUSE/python-rpm-macros [2]: https://build.opensuse.org/package/view_file/openSUSE:Factory/python-inipars...
- if your concern is that they won't ever be merged due to some people non-constructive obstruction, well I share it to some point but I won't spend more time arguing unicorns and what-if-maybe I-don't-like-commits-let's-make-them-awful-to-use. The people blocking there are free to take up the maintenance of my ~400 specs and redo them as they want, we'll see how they love to waste tens of minutes per spec to position setup args manually when they have to do it for hundreds of them, every time there is a package update or a forge relayout.
Technically, no one has to "accept" anything for the macros to be available. It's whether the macros would be broadly used. I could, in fact, tomorrow, make various extra useful macros that I use available in a package for people to pull in. It doesn't mean anyone has to use them.
The main difficulty with the proposal is not the *forge part, it's that automated autodeps are well, automated. As every automation they are strict and unyielding and do not take approximations well. They will fail spectacularly in the following cases:
the packager does not package all the deps of its code → the resulting package is uninstallable because autodeps add requires on missing bits
the packager packages code with garbage imports (very common in example or test code) → the resulting package is uninstallable because autodeps add require
the packager does not own properly the directories where its Go code is installed → given that Go deps depend on directories, Go autodeps also depend on proper directory ownership. Positioning autodeps on Go files themselves would make package build times increase many many times as rpm would try to compute the same autodeps for every single .go file in a directory. (and Go autodeps are already sloooow as snails because invoking the go command is slow)
All those things are mitigated by the use of %goinstall that filters usual example/test/vendor dirs so they don't trigger autodeps, and tries very hard to own all relevant directories.
For those reasons I don't propose to activate autodeps in old-style golang packages. They need conversion (and review by a human to check no problem code is deployed) first.
As long as I can do Obsoletes/Provides for the old name for the devel, unit-test, etc. stuff so the old names work, it won't bother me too much. I want people to be able to do "snapd-devel" or whatnot and get the right thing. Though I'm not sure why package name enforcement is required for autodeps, since that's based on file lists passed to a dep generator...
----- Mail original ----- De: "Neal Gompa"
- if your concern is that the *forge* macros are defective somewhere I'd be curious where as you'd be the
first to report an actual technical problem. I've used them intensively in rawhide and el7 with many different rpm tools and they are rock solid for me
I don't consider them defective. They make me a bit squeamish because I don't see the spec preamble and such because they're autogenerated, but you've mentioned that it's possible to selectively use these things with Go and forge macros.
I'm curious, what are you missing in the preamble ? As far as I can see it's all there (even though some values set to variables %gometa precomputes). I had it's right autogenerated some parts of it in the past but it's all converted to variable use now to avoid packager surprise and permit customization.
For example, what are you missing there?
%<--- %global goipath github.com/hashicorp/consul Version: 1.0.2
%gometa
%global common_description %{expand: Consul is a tool for service discovery […]}
Name: consul Release: 5%{?dist} Summary: A tool for easy service discovery, monitoring and configuration License: MPLv2.0 URL: https://www.consul.io/ Source: %{gosource} %<---
Or here ?
%<--- %global goipath github.com/hashicorp/go-sockaddr %global commit 9b4c5fa5b10a683339a270d664474b9f4aee62fc
%gometa
%global common_description %{expand: Socket address convenience functions for Go. go-sockaddr is a convenience library…}
Name: %{goname} Version: 0 Release: 0.14%{?dist} Summary: A Go convenience library to manipulate IP Addresses and UNIX sockets License: MPLv2.0 URL: %{gourl} Source: %{gosource} %<---
Regards,
On Tue, Jan 23, 2018 at 8:54 AM, nicolas.mailhot@laposte.net wrote:
----- Mail original ----- De: "Neal Gompa"
- if your concern is that the *forge* macros are defective somewhere I'd be curious where as you'd be the
first to report an actual technical problem. I've used them intensively in rawhide and el7 with many different rpm tools and they are rock solid for me
I don't consider them defective. They make me a bit squeamish because I don't see the spec preamble and such because they're autogenerated, but you've mentioned that it's possible to selectively use these things with Go and forge macros.
I'm curious, what are you missing in the preamble ? As far as I can see it's all there (even though some values set to variables %gometa precomputes). I had it's right autogenerated some parts of it in the past but it's all converted to variable use now to avoid packager surprise and permit customization.
For example, what are you missing there?
%<--- %global goipath github.com/hashicorp/consul Version: 1.0.2
%gometa
%global common_description %{expand: Consul is a tool for service discovery […]}
Name: consul Release: 5%{?dist} Summary: A tool for easy service discovery, monitoring and configuration License: MPLv2.0 URL: https://www.consul.io/ Source: %{gosource} %<---
Or here ?
%<--- %global goipath github.com/hashicorp/go-sockaddr %global commit 9b4c5fa5b10a683339a270d664474b9f4aee62fc
%gometa
%global common_description %{expand: Socket address convenience functions for Go. go-sockaddr is a convenience library…}
Name: %{goname} Version: 0 Release: 0.14%{?dist} Summary: A Go convenience library to manipulate IP Addresses and UNIX sockets License: MPLv2.0 URL: %{gourl} Source: %{gosource} %<---
Actually, this is fine with me. This is what disturbed me a bit: https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation#Pa...
----- Mail original ----- De: "Neal Gompa"
I'm curious, what are you missing in the preamble ? As far as I can see it's all there (even though some values set to variables %gometa precomputes). I had it's right autogenerated some parts of it in the past but it's all converted to variable use now to avoid packager surprise and permit customization.
Actually, this is fine with me. This is what disturbed me a bit: https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation#Pa...
Ah, yes the examples snip the lines that do not need specific changes for readability Is the … not clear enough? I fear that putting a full preamble would be even more confusing for some readers. Though if people disagree, I'l try to find some time to add the skipped lines back in
On Tue, Jan 23, 2018 at 9:40 AM, nicolas.mailhot@laposte.net wrote:
----- Mail original ----- De: "Neal Gompa"
I'm curious, what are you missing in the preamble ? As far as I can see it's all there (even though some values set to variables %gometa precomputes). I had it's right autogenerated some parts of it in the past but it's all converted to variable use now to avoid packager surprise and permit customization.
Actually, this is fine with me. This is what disturbed me a bit: https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation#Pa...
Ah, yes the examples snip the lines that do not need specific changes for readability Is the … not clear enough? I fear that putting a full preamble would be even more confusing for some readers. Though if people disagree, I'l try to find some time to add the skipped lines back in
For snipping, use "[...]" notation to indicate skipped stuff. It's hard to tell otherwise.
----- Mail original ----- De: "Neal Gompa"
For snipping, use "[...]" notation to indicate skipped stuff. It's hard to tell otherwise.
Ok, that was easy to fix :)
----- Mail original ----- De: "Neal Gompa"
As long as I can do Obsoletes/Provides for the old name for the devel, unit-test,
BTW is anyone using the unit-test packages? Right now I do not generate them, I don't need them, and making them work with autodeps would be hairy (deploying without autodeps should be trivial however)
To be honest, given all the parts current packages fail to install, I'd expect many of the current unit test packages to fail in mysterious ways, so I'm curious: what use has been found for them?
Regards,
On Tue, Jan 23, 2018 at 9:00 AM, nicolas.mailhot@laposte.net wrote:
----- Mail original ----- De: "Neal Gompa"
As long as I can do Obsoletes/Provides for the old name for the devel, unit-test,
BTW is anyone using the unit-test packages? Right now I do not generate them, I don't need them, and making them work with autodeps would be hairy (deploying without autodeps should be trivial however)
To be honest, given all the parts current packages fail to install, I'd expect many of the current unit test packages to fail in mysterious ways, so I'm curious: what use has been found for them?
No idea, but I just checked, and I don't even have the unit-test subpackage enabled on snapd, so I don't particularly care too much there...
----- Mail original ----- De: "Neal Gompa"
Hi,
Thanks for the review !
I really do like this. There are only two issues I have with it:
- This seems to mandate that all packages must be named by their
import path. My golang package (snapd) is not, intentionally so. I don't want to change this.
Well that was not the intent, I probably need to clarify things a bit more. The *devel* package must absolutely be named after the import path (for sanity sake). The core package can be named something else (usually because it's the packaging of an app)
For example in my etcd spec
Name: etcd ← well-known app name %package -n %{goname}-devel ← go naming for go devel stuff
- Mandating a forge is going to be tricky for self-hosted stuff, or
people who release Go code as tarballs (it's rare, but it happens). How do you deal with that?
The proposal automates integration with forges, it does not mandate it
You can declare manually archivename archiveurl and forgesetup before calling the %gometa macro and you should benefit from the rest of the automation even on an hosting site it does not know anything about. The whole code tries very hard to let the packager pre-define everything it may guess wrong or not know about to avoid giving up on all the automation just because a little part is wrong.
And even after that you can still declare Source manually and pass arguments to setup the old way it that's what you want
Not calling %gometa at all will kill stuff like goname which is kind of mandatory for consistency.
Regards,
On 12/17/2017 01:11 AM, nicolas.mailhot@laposte.net wrote:
Hi,
I am proposing for inclusion a set of rpm technical files aimed at automating the packaging of forge-hosted projects.
- Packaging draft: https://fedoraproject.org/wiki/More_Go_packaging
- https://pagure.io/packaging-committee/issue/734
- go-srpm-macros RFE with the technical files: https://bugzilla.redhat.com/show_bug.cgi?id=1526721
Hi,
This looks pretty cool! One thing I notice in the limitations section of your draft is a lot of "we can't do XXX due to lack of release discipline..."
Do you have any recommendations for Go programmers on how to structure their software so that it is easy to package?
Thanks, -Mat
----- Mail original ----- De: "Mátyás Selmeci"
Hi,
This looks pretty cool!
Thanks for the feedback!
One thing I notice in the limitations section of your draft is a lot of "we can't do XXX due to lack of release discipline..."
Do you have any recommendations for Go programmers on how to structure their software so that it is easy to package?
If there is an interest I can add a section on how to make a Go project easier to package, sure.
It won't be earth-shattering, just the Go declination of basic common sense rules that are needed in any coding language, that many Go projects already apply (unfortunately not all of them):
— do not change your import path every other month — do not make your code accessible through multiple import paths – only use smallcaps in your import path (I know some systems are case insensitive. Many others are NOT) – communicate clearly the canonical import path of the project at the top of your README.md — if you absolutely need to change your import path fix your code to use the new import path do not rely on http redirections – that includes testing and example code — do not add a .git suffix to your import path
— use _testdata for all the material needed by unit test – put your example code in _examples (with subdirectories if you ship several examples). Do not use creative unusual names such as tutorial. – do not pepper your subdirectories with .md files. Keep documentation in the project root or in a docs root subdirectory if there is too much of it — add a one-line summary and a least a § describing what your project does at the top of your README.md
— choose licenses already vetted by Fedora or Debian https://fedoraproject.org/wiki/Licensing:Main?rd=Licensing#Good_Licenses – add a licensing file named LICENSE — use unmodified plaintext canonical licensing texts, that state the LICENSE name at the top of the file — if you absolutely want to add an extension to make Windows happy, use LICENSE.txt — if you absolutely want to name your licensing file something else, please do not use something.md — do not hide your licensing terms in README.md, do not refer to a license by name without providing its text
– do releases — do releases, even for minor fixes. If you haven't felt the need to touch a project for months its latest state should be released! — do releases, even for projects that can only be called through another project. Do not rely on this other project to set code state through vendoring (that's easy to do, just propagate the tip project version to the ancillary projects at release time, like kubernetes does) – use semver for your releases https://semver.org/ Distributors and godep will thank you – if your project is in git, use a different branch for every major version of your project — if your project is in git, tag your release x.y.z as x.y.z, or vx.y.z, never as vx.y.zprettybeta. Versions and version tags are not the right place to document a project maturity. — numbers are cheap, never reuse the same number for a pre-release and a final release, increase the minor version! – please version your import paths with each major release (gopkg.in is good for that)
– use releases of the projects you depend on — never depend on a project that already depends on you (otherwise software dependencies stop being a nice directed acyclic graph and you get into dependency loop hell. That's a nasic software engineering golden rule, not respecting it will bite you sooner or later with or without linux distros, despite vendoring) – if for some reason, one of the projects you depend on does not release, please ask nicely it to do so – if for some reason you need a code state in tip which is not in a release, please inform the origin you'd like it to do a release, and switch to this release as soon as it is available – never depend on a commit state somewhere between two releases — document the major versions of the stuff you depend on somewhere easy to find. If a major version is only usable past as specific minor version, document it – add a unit test that detects if the project you depend on is missing the part that requires being after this minor version
– if your project provides wrappers, connexion glue or anything similar to many many many other projects keep the code for each other project in a separate subdirectory (Go package) so it can be desactivated without impacting the rest of your code if the other project ode has a problem.
— never add changes to the projects you reuse, submit the changes to those projects — if you absolutely need to change a project you reuse, fork it cleanly with a new import path so distributors do not accidentally reinject the original project — don't forget to rebase your code on the original project if you don't have the energy to maintain your own fork – rebase to the latest minor version of every project you depend on at release time. Do not let changes accumulate till rebasing becomes a major endeavor.
— do not hide vendor subdirectories deep inside your project tree. Only use a toplevel directory – identify clearly all the bits scrounged from other projects and not located in vendor in your README.md file – if part of your project is generated code, document cleanly how to remove what you already generated and how to regenerate it from scratch
– put the code that needs to be built into a binary in cmd/binary-name
– do not test projects that depend on you from your project. Contribute the testing code to those projects (otherwise software dependencies stop being a nice directed acyclic graph and you get into dependency loop hell) – do not ship not-working testing code (people will thing their build is broken) — do not test that a project you depend on has a specific import path, it may rename itself in the future – do not ship tests that depend on special parameters, go test should always just work — do not ship tests that rely on being in a specific directory, as long as the test code in in GOPATH it should be testable regardless of the location — do not ship tests that depend on specific timings, distributor build farms may be heavily loaded, causing the tests to fail. – if a test depends on a special preparation, ship a shell script that describes this preparation. Do not rely on Ubuntu docker images. – if a test depends on network access, make it fail gracefully when this access is blocked (with PASS not FAIL) — never do go get from tests — never stomp on GOPATH in tests
Anyway, you get the idea. This is probably not exhaustive, it needs to be maintained in a wiki page if people find it useful
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: "Development discussions related to Fedora" devel@lists.fedoraproject.org Sent: Tuesday, January 23, 2018 7:45:06 PM Subject: Re: <DKIM> Re: Proposed Fedora packaging guideline: More Go packaging
----- Mail original ----- De: "Mátyás Selmeci"
Hi,
This looks pretty cool!
Thanks for the feedback!
One thing I notice in the limitations section of your draft is a lot of "we can't do XXX due to lack of release discipline..."
Do you have any recommendations for Go programmers on how to structure their software so that it is easy to package?
If there is an interest I can add a section on how to make a Go project easier to package, sure.
It won't be earth-shattering, just the Go declination of basic common sense rules that are needed in any coding language, that many Go projects already apply (unfortunately not all of them):
— do not change your import path every other month — do not make your code accessible through multiple import paths – only use smallcaps in your import path (I know some systems are case insensitive. Many others are NOT) – communicate clearly the canonical import path of the project at the top of your README.md — if you absolutely need to change your import path fix your code to use the new import path do not rely on http redirections – that includes testing and example code — do not add a .git suffix to your import path
— use _testdata for all the material needed by unit test
- make sure that all tests and exclusive test dependencies are really only in _test.go files
– put your example code in _examples (with subdirectories if you ship several examples). Do not use creative unusual names such as tutorial. – do not pepper your subdirectories with .md files. Keep documentation in the project root or in a docs root subdirectory if there is too much of it — add a one-line summary and a least a § describing what your project does at the top of your README.md
— choose licenses already vetted by Fedora or Debian https://fedoraproject.org/wiki/Licensing:Main?rd=Licensing#Good_Licenses – add a licensing file named LICENSE — use unmodified plaintext canonical licensing texts, that state the LICENSE name at the top of the file — if you absolutely want to add an extension to make Windows happy, use LICENSE.txt — if you absolutely want to name your licensing file something else, please do not use something.md — do not hide your licensing terms in README.md, do not refer to a license by name without providing its text
– do releases — do releases, even for minor fixes. If you haven't felt the need to touch a project for months its latest state should be released! — do releases, even for projects that can only be called through another project. Do not rely on this other project to set code state through vendoring (that's easy to do, just propagate the tip project version to the ancillary projects at release time, like kubernetes does) – use semver for your releases https://semver.org/ Distributors and godep will thank you – if your project is in git, use a different branch for every major version of your project — if your project is in git, tag your release x.y.z as x.y.z, or vx.y.z, never as vx.y.zprettybeta. Versions and version tags are not the right place to document a project maturity. — numbers are cheap, never reuse the same number for a pre-release and a final release, increase the minor version! – please version your import paths with each major release (gopkg.in is good for that)
- ideally do major releases in separate branches, so you can do minor fixes/releases easily even for older major releases(look at GC) - do release whenever you alter your API, extending it, modifying it, removing some and document it in the release notes
– use releases of the projects you depend on — never depend on a project that already depends on you (otherwise software dependencies stop being a nice directed acyclic graph and you get into dependency loop hell. That's a nasic software engineering golden rule, not respecting it will bite you sooner or later with or without linux distros, despite vendoring) – if for some reason, one of the projects you depend on does not release, please ask nicely it to do so – if for some reason you need a code state in tip which is not in a release, please inform the origin you'd like it to do a release, and switch to this release as soon as it is available – never depend on a commit state somewhere between two releases — document the major versions of the stuff you depend on somewhere easy to find. If a major version is only usable past as specific minor version, document it – add a unit test that detects if the project you depend on is missing the part that requires being after this minor version
– if your project provides wrappers, connexion glue or anything similar to many many many other projects keep the code for each other project in a separate subdirectory (Go package) so it can be desactivated without impacting the rest of your code if the other project ode has a problem.
— never add changes to the projects you reuse, submit the changes to those projects — if you absolutely need to change a project you reuse, fork it cleanly with a new import path so distributors do not accidentally reinject the original project — don't forget to rebase your code on the original project if you don't have the energy to maintain your own fork – rebase to the latest minor version of every project you depend on at release time. Do not let changes accumulate till rebasing becomes a major endeavor.
- if you ever have to use vendor-ing, first don't use it, secondly if you really really have too - pull in the vendor-ed deps, fresh in your build infrastructure, don't pollute the repo with (mostly) outdated rotting code
— do not hide vendor subdirectories deep inside your project tree. Only use a toplevel directory – identify clearly all the bits scrounged from other projects and not located in vendor in your README.md file – if part of your project is generated code, document cleanly how to remove what you already generated and how to regenerate it from scratch
– put the code that needs to be built into a binary in cmd/binary-name
– do not test projects that depend on you from your project. Contribute the testing code to those projects (otherwise software dependencies stop being a nice directed acyclic graph and you get into dependency loop hell) – do not ship not-working testing code (people will thing their build is broken) — do not test that a project you depend on has a specific import path, it may rename itself in the future – do not ship tests that depend on special parameters, go test should always just work
- if you have some special needs in testing(networking, special HW,...), detect those, if not met skip(and log)
— do not ship tests that rely on being in a specific directory, as long as the test code in in GOPATH it should be testable regardless of the location — do not ship tests that depend on specific timings, distributor build farms may be heavily loaded, causing the tests to fail. – if a test depends on a special preparation, ship a shell script that describes this preparation. Do not rely on Ubuntu docker images. – if a test depends on network access, make it fail gracefully when this access is blocked (with PASS not FAIL) — never do go get from tests — never stomp on GOPATH in tests
Anyway, you get the idea. This is probably not exhaustive, it needs to be maintained in a wiki page if people find it useful
Regards,
-- Nicolas Mailhot
Very nice list, it would be nice to have it as sub-wiki page of guidelines. I have took liberty to add few points.
JC
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
De: "Jakub Cajka"
Very nice list, it would be nice to have it as sub-wiki page of guidelines. I have took liberty to add few points.
Ok, I put it here so people have a place to work on it https://fedoraproject.org/wiki/More_Go_packaging#Go_developer_guidance:_maki...
It can be moved later wherever you feel is more appropriate
Regards,
Hi,
Now the technical PR is submitted https://src.fedoraproject.org/rpms/go-srpm-macros/pull-request/1
and waiting for action from the go-srpm-macros maintainers, I took (quite a long) time to refresh and flesh out the corresponding packaging guidelines proposal. It should be fairly complete now:
https://fedoraproject.org/wiki/More_Go_packaging
I'd appreciate review to check if I have forgotten an important case, if people understand the text, if they have enhancements or corrections to propose, and so on.
Then I will push it FPC side again.
Actual practice should be fairly simple and self-explanatory, the proposal length can be scary but that's because it documents all kinds of corner cases that required digging through specs and mailing lists to find resolution examples before. The basic Go packaging skeleton will be sufficient is most cases without requiring to read any documentation.
Regards,
On Tue, Jan 30, 2018 at 10:11 AM, nicolas.mailhot@laposte.net wrote:
Hi,
Now the technical PR is submitted https://src.fedoraproject.org/rpms/go-srpm-macros/pull-request/1
and waiting for action from the go-srpm-macros maintainers, I took (quite a long) time to refresh and flesh out the corresponding packaging guidelines proposal. It should be fairly complete now:
https://fedoraproject.org/wiki/More_Go_packaging
I'd appreciate review to check if I have forgotten an important case, if people understand the text, if they have enhancements or corrections to propose, and so on.
Then I will push it FPC side again.
Actual practice should be fairly simple and self-explanatory, the proposal length can be scary but that's because it documents all kinds of corner cases that required digging through specs and mailing lists to find resolution examples before. The basic Go packaging skeleton will be sufficient is most cases without requiring to read any documentation.
The only thing I see that might be missing is autogenerating bundled(golang()) Provides when a vendor tree exists (with the appropriate automatic filters on Requires).
De: "Neal Gompa"
The only thing I see that might be missing is autogenerating bundled(golang()) Provides when a vendor tree exists (with the appropriate automatic filters on Requires).
I had though a little about doing it but first, as many Go elements, vendoring relies on conventions not standards. The nasty thing about conventions is that they are not applied 100% the same way by everyone, making automation a PITA. And second interactions with autodeps can be nasty: you can filter out provides, but do you filter out requires? What about all the junk code many projects ship as testing and examples and which is vendored with the rest?
I don't say it can't be done, or that it would be difficult to do once the rest is merged, but I'll live it to someone that absolutely want to ship a Go package with vendored parts.
Right now we sidestep the issue in our packages by rm -fr ing anything that looks like vendored code in %prep. Unbundling takes time but it has a positive cumulative effect: the more you unbundle the less you need to worry about in other packages with the same requirements. And unbundling reveals code/legal problems that it would take about as much work to solve by auditing the vendored code manually.
Regards,
Hi Nicolas,
Is there a guide for Fedora packagers about how to handle unbundling for golang packages? The draft guidelines don't seem to go into any details. I've looked at packaging a few golang packages unbundled, and have immediately run into:
A) lots of unpackaged dependencies B) dependencies that are packaged in Fedora with different, often much older versions
A) is a pretty known quantity for any type of package, but I found B) more intimidating. It seems like to package the new package, I have to get the maintainer of the library to upgrade the version, and someone needs to rebuild everything that depends on the rebuilt library and test that the rebuilt packages work.
Some tutorial showing a practical example of packaging a golang package for the first time I think would be very helpful.
Owen
On Wed, Jan 31, 2018 at 5:30 AM, nicolas.mailhot@laposte.net wrote:
De: "Neal Gompa"
The only thing I see that might be missing is autogenerating bundled(golang()) Provides when a vendor tree exists (with the appropriate automatic filters on Requires).
I had though a little about doing it but first, as many Go elements, vendoring relies on conventions not standards. The nasty thing about conventions is that they are not applied 100% the same way by everyone, making automation a PITA. And second interactions with autodeps can be nasty: you can filter out provides, but do you filter out requires? What about all the junk code many projects ship as testing and examples and which is vendored with the rest?
I don't say it can't be done, or that it would be difficult to do once the rest is merged, but I'll live it to someone that absolutely want to ship a Go package with vendored parts.
Right now we sidestep the issue in our packages by rm -fr ing anything that looks like vendored code in %prep. Unbundling takes time but it has a positive cumulative effect: the more you unbundle the less you need to worry about in other packages with the same requirements. And unbundling reveals code/legal problems that it would take about as much work to solve by auditing the vendored code manually.
Regards,
-- Nicolas mAilhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
----- Original Message -----
From: "Owen Taylor" otaylor@redhat.com To: "Development discussions related to Fedora" devel@lists.fedoraproject.org Cc: golang@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Wednesday, January 31, 2018 6:50:21 PM Subject: Re: <DKIM> [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
Hi Nicolas,
Is there a guide for Fedora packagers about how to handle unbundling for golang packages? The draft guidelines don't seem to go into any details. I've looked at packaging a few golang packages unbundled, and have immediately run into:
A) lots of unpackaged dependencies B) dependencies that are packaged in Fedora with different, often much older versions
A) is a pretty known quantity for any type of package, but I found B) more intimidating. It seems like to package the new package, I have to get the maintainer of the library to upgrade the version, and someone needs to rebuild everything that depends on the rebuilt library and test that the rebuilt packages work.
Some tutorial showing a practical example of packaging a golang package for the first time I think would be very helpful.
Owen
Unfortunately this is how the situation stand thanks to the wide spread use of vendoring and generally no release and API stability in Go projects. You have to package all not packaged deps and work with other maintainers(which mostly means with Jan Chaloupka, cc'ed) to get other already packaged deps to commonly acceptable version/commit level, although it might not be even possible thanks to API breaks between commits. So you have to stick to bundling in some cases, if you really want to package that project.
JC
On Wed, Jan 31, 2018 at 5:30 AM, < nicolas.mailhot@laposte.net > wrote:
De: "Neal Gompa"
The only thing I see that might be missing is autogenerating bundled(golang()) Provides when a vendor tree exists (with the appropriate automatic filters on Requires).
I had though a little about doing it but first, as many Go elements, vendoring relies on conventions not standards. The nasty thing about conventions is that they are not applied 100% the same way by everyone, making automation a PITA. And second interactions with autodeps can be nasty: you can filter out provides, but do you filter out requires? What about all the junk code many projects ship as testing and examples and which is vendored with the rest?
I don't say it can't be done, or that it would be difficult to do once the rest is merged, but I'll live it to someone that absolutely want to ship a Go package with vendored parts.
Right now we sidestep the issue in our packages by rm -fr ing anything that looks like vendored code in %prep. Unbundling takes time but it has a positive cumulative effect: the more you unbundle the less you need to worry about in other packages with the same requirements. And unbundling reveals code/legal problems that it would take about as much work to solve by auditing the vendored code manually.
Regards,
-- Nicolas mAilhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
----- Mail original ----- De: "Owen Taylor"
Hi Owen,
Is there a guide for Fedora packagers about how to handle unbundling for golang packages? The draft guidelines don't seem to go into any details.
I don't think there is, nor that it is necessarily needed. The posted guidelines should be sufficient technically, there are no magic I know of I didn't document, the rest is just a lot of work (but see ↓)
I've looked at packaging a few golang packages unbundled, and have immediately run into:
A) lots of unpackaged dependencies B) dependencies that are packaged in Fedora with different, often much older versions
Yes the state of Go packages in Fedora is pretty sad right now. I wouldn't expect anyone to be able to package anything but the most trivial app in unbundled mode. Too many common parts are missing, when they are not missing they are too old, trying to update an existing Go package is an exercise in frustration (too much package-specific shell code, that is difficult to understand and does not really work with new code versions), and trying to update or add missing parts just reveals more breakage and work to be done.
However, accepting to package complex Go apps in bundled mode is how Fedora got to this state in the first place. In plain speak, bundling (vendoring in Go linguo) just does not scale. You need an awful lot of manpower to audit the hundreds of other projects each app bundles, bundling removes all the package tooling that may help you to do so, and the result is not shared with any other package, or with other versions of the same package, so you get zero positive network effects. Worse, big bundled apps that do not actually try to work in unbundled mode, do not actually test the code they export (they are bundling, remember) so the result is toxic to small Go packages that try to work with them.
So bundling parts is a direct road to bundling everything, bundling everything is a direct road to bundling everything blindly, bundling everything blindly is error-prone and dangerous (because upstreams are only human and do make lots of mistakes), and pretty much removes any value Fedora can add to end users, to other parts of Fedora that would like to integrate with Go software, or to the upstream projects themselves (no Fedora QA, no stream of Fedora-originated fixes, no Fedora pressure to stabilize parts when upstream is lost in tunnel effect mode and does not realize that it is wasting everyone's time starting with its own).
Therefore, trying to get all this it a better state, requires the following steps IMHO:
1. completely refactor our Go packaging style it's less painful to update Go packages and they do not need a packager with deep knowledge of package-specific shell glue. That takes documentation, and factoring out common Go packaging functions in shared rpm code (macros and autodeps) → that's what I posted
2. using the new documentation and tooling to clean up years of Fedora technical debt, and create a new set of up-to-date Go packages that can serve as new baseline → I have hundreds of specs that I'm waiting for step 1 to complete to submit. They won't constitute a full baseline by themselves, they're not all 100% done, it's too much work to do alone but working with others requires the common macros and documentation to be merged and adopted. This step is going to be painful I'm afraid, Fedora dug itself a deep hole, leaving it is going to be hard.
3. hopefully, the result will be streamlined enough it won't be too painful to keep up to date, having an up to date baseline will help attract new Go packagers like you, everyone will benefit and be happy. We had to package some ostree bits for example to create this baseline, and I'm pretty sure ostree people within Fedora would prefer to maintain those themselves, if the rest of the Fedora go universe didn't make it too hard
B) more intimidating. It seems like to package the new package, I have to get the maintainer of the library to upgrade the version,
The guidelines and automation aim at making upgrading easy, and avoid one package or packager blocking others
and someone needs to rebuild everything that depends on the rebuilt library and test that the rebuilt packages work.
I hope that normalizing Fedora Go packages means it will be possible to automate QA tests such as "try to rebuild all the packages that depend on golang(foo) every time you see the package providing golang(foo) changing, and report what broke"
That would be expensive computing-side but a lot less expensive and long than expecting each Go packager to do it himself with his own means. And that is certainly not overkill, given how lax Go projects are about maintaining API stability.
And then in case of breakage, revert or create a compat package. That's why there is a long chapter dedicated to compat package creation in the proposed guidelines.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Thursday, February 1, 2018 11:21:59 AM Subject: Re: <DKIM> [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
----- Mail original ----- De: "Owen Taylor"
Hi Owen,
Is there a guide for Fedora packagers about how to handle unbundling for golang packages? The draft guidelines don't seem to go into any details.
I don't think there is, nor that it is necessarily needed. The posted guidelines should be sufficient technically, there are no magic I know of I didn't document, the rest is just a lot of work (but see ↓)
I've looked at packaging a few golang packages unbundled, and have immediately run into:
A) lots of unpackaged dependencies B) dependencies that are packaged in Fedora with different, often much older versions
Yes the state of Go packages in Fedora is pretty sad right now. I wouldn't expect anyone to be able to package anything but the most trivial app in unbundled mode. Too many common parts are missing, when they are not missing they are too old, trying to update an existing Go package is an exercise in frustration (too much package-specific shell code, that is difficult to understand and does not really work with new code versions), and trying to update or add missing parts just reveals more breakage and work to be done.
It depends (as everything) on available manpower, if you are willing to own your dependencies you can package anything and everything debundled.
However, accepting to package complex Go apps in bundled mode is how Fedora got to this state in the first place. In plain speak, bundling (vendoring in Go linguo) just does not scale. You need an awful lot of manpower to audit the hundreds of other projects each app bundles, bundling removes all the package tooling that may help you to do so, and the result is not shared with any other package, or with other versions of the same package, so you get zero positive network effects. Worse, big bundled apps that do not actually try to work in unbundled mode, do not actually test the code they export (they are bundling, remember) so the result is toxic to small Go packages that try to work with them.
On contrary, to this state you dislike(or seems to) Fedora got because of effort of many to de-bundle Go based projects.
So bundling parts is a direct road to bundling everything, bundling everything is a direct road to bundling everything blindly, bundling everything blindly is error-prone and dangerous (because upstreams are only human and do make lots of mistakes), and pretty much removes any value Fedora can add to end users, to other parts of Fedora that would like to integrate with Go software, or to the upstream projects themselves (no Fedora QA, no stream of Fedora-originated fixes, no Fedora pressure to stabilize parts when upstream is lost in tunnel effect mode and does not realize that it is wasting everyone's time starting with its own).
Therefore, trying to get all this it a better state, requires the following steps IMHO:
- completely refactor our Go packaging style it's less painful to update Go
packages and they do not need a packager with deep knowledge of package-specific shell glue. That takes documentation, and factoring out common Go packaging functions in shared rpm code (macros and autodeps) → that's what I posted
I don't see way how it makes it less painful(you have still the whole distro issue). It makes specs more abstract and streamlined when it works, but more opaque and hard to read when its not(and you have to debug some arcane srpm/lua macros).
- using the new documentation and tooling to clean up years of Fedora
technical debt, and create a new set of up-to-date Go packages that can serve as new baseline → I have hundreds of specs that I'm waiting for step 1 to complete to submit. They won't constitute a full baseline by themselves, they're not all 100% done, it's too much work to do alone but working with others requires the common macros and documentation to be merged and adopted. This step is going to be painful I'm afraid, Fedora dug itself a deep hole, leaving it is going to be hard.
On contrary Fedora is trying to fill the hole that upstream Go projects dug them selves in to. IMNHO Go have traded any subjectively perceived "RPM/deb hell" for even deeper levels of "Go (vendor) hell".
- hopefully, the result will be streamlined enough it won't be too painful
to keep up to date, having an up to date baseline will help attract new Go packagers like you, everyone will benefit and be happy. We had to package some ostree bits for example to create this baseline, and I'm pretty sure ostree people within Fedora would prefer to maintain those themselves, if the rest of the Fedora go universe didn't make it too hard
Pain will not be eased in any significant way as you will still have to carefully evaluate every change so you don't break any dependent package.
B) more intimidating. It seems like to package the new package, I have to get the maintainer of the library to upgrade the version,
The guidelines and automation aim at making upgrading easy, and avoid one package or packager blocking others
This is not really and automation per se. It is minor re-factoring and spec streamlining. It might or might not enable easier implementation of packaging automation.
and someone needs to rebuild everything that depends on the rebuilt library and test that the rebuilt packages work.
I hope that normalizing Fedora Go packages means it will be possible to automate QA tests such as "try to rebuild all the packages that depend on golang(foo) every time you see the package providing golang(foo) changing, and report what broke"
That would be expensive computing-side but a lot less expensive and long than expecting each Go packager to do it himself with his own means. And that is certainly not overkill, given how lax Go projects are about maintaining API stability.
Until upstreams will be working in true CI/CD manners(when they do not make any releases). Fedora can't really fix this. Currently AFAIK we have GoFed and accompanying projects that provide you with possibility to gather such data.
And then in case of breakage, revert or create a compat package. That's why there is a long chapter dedicated to compat package creation in the proposed guidelines.
Compat packages do not scale. Imagine that you will have in worst case maintain 10s of different versions of same package for every release, doing security backports, etc....
JC
Regards,
-- Nicolas Mailhot _______________________________________________ golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
De: "Jakub Cajka"
Hi Jakub,
It depends (as everything) on available manpower, if you are willing to own your dependencies you can package anything and everything debundled.
Sure, but available manpower depends on how high the bar you put for people wanting to join, and right now this bar is pretty high.
On contrary, to this state you dislike(or seems to) Fedora got because of effort of many to de bundle Go based projects.
I'm *not* disparaging the past efforts of many. They did a lot of good. Unfortunately, the result still falls short of the mark. The baseline is not complete enough, the baseline is old and aging fast, would-be Go packagers like Owen are ready to give up when they start getting an idea of the Go state in Fedora.
It's time to invest in tooling and simplification so the few Go Fedora packagers there is, can do more with less, so casual Go packagers can contribute their little bit instead of being repulsed by complexity that can be avoided and factored out, so people already familiar with Fedora packaging can package the Go bits they need without needing a lengthy formation on Go specifics.
There *are* lots of people that would like to do Go packaging. Many Go software projects currently make headlines. That there are so few Go packagers in Fedora, at the time Go is in the limelight, is a pretty good indicator we are making it too hard.
I don't see way how it makes it less painful(you have still the whole distro issue). It makes specs more abstract and streamlined when it works, but more opaque and hard to read when its not(and you have to debug some arcane srpm/lua macros).
It makes it less painful because there is no package-specific magic in each and every Go spec file.
You do not need to debug code in each spec because code that already just works for hundreds of packages is unlikely to not work on the next spec file.
You do not need to comb each and every line of a Go spec file wondering if the line is cut and pasting of a common template (multiply by all past revisions of common templates) or if it is package-specific. The common parts are in common macros not spread right and left.
You do not need to worry wether common code is complete or not because factoring out common code removes the temptation to document 90% of what's needed, leaving others to fill in the missing 10%. 10% amounts to an astonishing level of work when you multiply it by the number of Go packagers and the number of Go packages we would ideally need.
Plus, when you do find a bug in common code, the fix is shared with the whole distribution, not hidden in a single spec file.
That's what successful factoring of common code earns you.
And given I have more than 470 Go spec files that just work with the proposed macros with no special package-specific adjustment needed, and most time I compare one of those to the Fedora equivalent I'm executing more unit tests and building more binaries, with drastically shorter spec code, I'm pretty sure the factoring out *is* successful.
This step is going to be painful I'm afraid, Fedora dug itself a deep hole, leaving it is going to be hard.
On contrary Fedora is trying to fill the hole that upstream Go projects dug them selves in to.
Filling upstream holes is pretty much the definition of an integrator/distributor role. In Go they are particularly numerous and deep, but Fedora users do want their docker and kubernetes (and Kubernetes, BTW, is astonishingly free of the problems that plague many Go projects, proving it *is* possible to do good release engineering in Go).
Fedora did dug itself a hole by trying the bundling shortcut around EL7 time. Where are the EL7 Go packages? Why are Fedora Atomic people not spearheading Go packaging in Fedora? Waiting for bundling to be solved upstream-side made Fedora worse not better.
IMNHO Go have traded any subjectively perceived "RPM/deb hell" for even deeper levels of "Go vendor) hell".
I fully agree. What I meant to convey is that replicating upstream vendoring Fedora-side only makes the situation worse for users, for upstreams, and for Fedora.
Pain will not be eased in any significant way as you will still have to carefully evaluate every change so you don't break any dependent package.
Pain *will* be eased in a significant way because we will have the tooling to detect breakage automatically, and because fixes once they're done will be packaged and shared.
The guidelines and automation aim at making upgrading easy, and avoid one package or packager blocking others
This is not really and automation per se. It is minor re-factoring and spec streamlining. It might or might not enable easier implementation of packaging automation.
If it's minor let's do it. I'm quite sure you will be pleasantly surprised by the level of pain those minor changes remove.
That would be expensive computing-side but a lot less expensive and long than expecting each Go packager to do it himself with his own means. And that is certainly not overkill, given how lax Go projects are about maintaining API stability.
Until upstreams will be working in true CI/CD manners(when they do not make any releases). Fedora can't really fix this.
Fedora can detect the breakage which is the first step to fix it. It does not take an army of Go experts, just unbundled packages that can be rebuilt automatically to identify incompatibilities. Go experts are better employed working on fixes than doing rebuild checks that can be automated.
Currently AFAIK we have GoFed and accompanying projects that provide you with possibility to gather such data.
And nothing stops you for continuing to use those. Except, the golden test will always be to actually try to build the code, which is something that can be automated reliably once the dependencies exported to Fedora tools are themselves complete and reliable, and Go packages are exported as rpm packages, not hidden in private vendor trees.
And then in case of breakage, revert or create a compat package. That's why there is a long chapter dedicated to compat package creation in the proposed guidelines.
Compat packages do not scale.
They scale a lot better than bundling, when an API break results in dozens of projects choosing to vendor a pre-break commit. With each of them choosing a slightly different state, resulting in ten times the amount of code to audit and maintain.
Imagine that you will have in worst case maintain 10s of different versions of same package for every release, doing security backports, etc....
When you do not unbundle you are still shipping the vendored code, you are still responsible for its bugs (security or not). Compat packages do not augment the code surface you have to care about, on the contrary they offer you the possibility to *reduce* the number of code states in the distribution.
Plus, they are the occasion, to notify upstreams they should move to a more recent code state. And reduce the number of code states you have to care about yet more. Not all upstreams are dead set against updating their vendoring. A lot of them are nice and responsible and just didn't see they had an API break to manage yet because their use of vendoring hid the problem.
Of course that supposes that creating a compat package does not add a significant package-creation burden. Which I *did* try to avoid. Organization-wise Go will still need a form of exemption to create many many packages easily, because Go code is spread over many many projects, and not consolidated in a few big libraries like in the C world.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Thursday, February 1, 2018 2:51:13 PM Subject: Re: <DKIM> [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
De: "Jakub Cajka"
Hi Jakub,
It depends (as everything) on available manpower, if you are willing to own your dependencies you can package anything and everything debundled.
Sure, but available manpower depends on how high the bar you put for people wanting to join, and right now this bar is pretty high.
On contrary, to this state you dislike(or seems to) Fedora got because of effort of many to de bundle Go based projects.
I'm *not* disparaging the past efforts of many. They did a lot of good. Unfortunately, the result still falls short of the mark. The baseline is not complete enough, the baseline is old and aging fast, would-be Go packagers like Owen are ready to give up when they start getting an idea of the Go state in Fedora.
It's time to invest in tooling and simplification so the few Go Fedora packagers there is, can do more with less, so casual Go packagers can contribute their little bit instead of being repulsed by complexity that can be avoided and factored out, so people already familiar with Fedora packaging can package the Go bits they need without needing a lengthy formation on Go specifics.
There *are* lots of people that would like to do Go packaging. Many Go software projects currently make headlines. That there are so few Go packagers in Fedora, at the time Go is in the limelight, is a pretty good indicator we are making it too hard.
I don't see way how it makes it less painful(you have still the whole distro issue). It makes specs more abstract and streamlined when it works, but more opaque and hard to read when its not(and you have to debug some arcane srpm/lua macros).
It makes it less painful because there is no package-specific magic in each and every Go spec file.
You do not need to debug code in each spec because code that already just works for hundreds of packages is unlikely to not work on the next spec file.
You do not need to comb each and every line of a Go spec file wondering if the line is cut and pasting of a common template (multiply by all past revisions of common templates) or if it is package-specific. The common parts are in common macros not spread right and left.
You do not need to worry wether common code is complete or not because factoring out common code removes the temptation to document 90% of what's needed, leaving others to fill in the missing 10%. 10% amounts to an astonishing level of work when you multiply it by the number of Go packagers and the number of Go packages we would ideally need.
Plus, when you do find a bug in common code, the fix is shared with the whole distribution, not hidden in a single spec file.
That's what successful factoring of common code earns you.
And given I have more than 470 Go spec files that just work with the proposed macros with no special package-specific adjustment needed, and most time I compare one of those to the Fedora equivalent I'm executing more unit tests and building more binaries, with drastically shorter spec code, I'm pretty sure the factoring out *is* successful.
This step is going to be painful I'm afraid, Fedora dug itself a deep hole, leaving it is going to be hard.
On contrary Fedora is trying to fill the hole that upstream Go projects dug them selves in to.
Filling upstream holes is pretty much the definition of an integrator/distributor role. In Go they are particularly numerous and deep, but Fedora users do want their docker and kubernetes (and Kubernetes, BTW, is astonishingly free of the problems that plague many Go projects, proving it *is* possible to do good release engineering in Go).
??
Fedora did dug itself a hole by trying the bundling shortcut around EL7 time. Where are the EL7 Go packages? Why are Fedora Atomic people not spearheading Go packaging in Fedora? Waiting for bundling to be solved upstream-side made Fedora worse not better.
I'm not sure that Fedora. Go ask them don't ask me or other regular Fedora maintainers. Fedora haven't been bundling and has been doing as much as possible to un-bundle. Fedora can't solve others non related problems.
IMNHO Go have traded any subjectively perceived "RPM/deb hell" for even deeper levels of "Go vendor) hell".
I fully agree. What I meant to convey is that replicating upstream vendoring Fedora-side only makes the situation worse for users, for upstreams, and for Fedora.
Pain will not be eased in any significant way as you will still have to carefully evaluate every change so you don't break any dependent package.
Pain *will* be eased in a significant way because we will have the tooling to detect breakage automatically, and because fixes once they're done will be packaged and shared.
Your proposal doesn't provide this kind of tooling, if I'm not mistaken.
The guidelines and automation aim at making upgrading easy, and avoid one package or packager blocking others
This is not really and automation per se. It is minor re-factoring and spec streamlining. It might or might not enable easier implementation of packaging automation.
If it's minor let's do it. I'm quite sure you will be pleasantly surprised by the level of pain those minor changes remove.
?? I'm in no way opposing your proposal per se.
That would be expensive computing-side but a lot less expensive and long than expecting each Go packager to do it himself with his own means. And that is certainly not overkill, given how lax Go projects are about maintaining API stability.
Until upstreams will be working in true CI/CD manners(when they do not make any releases). Fedora can't really fix this.
Fedora can detect the breakage which is the first step to fix it. It does not take an army of Go experts, just unbundled packages that can be rebuilt automatically to identify incompatibilities. Go experts are better employed working on fixes than doing rebuild checks that can be automated.
Rebuilds could be automated, analysis of them not easily.
Currently AFAIK we have GoFed and accompanying projects that provide you with possibility to gather such data.
And nothing stops you for continuing to use those. Except, the golden test will always be to actually try to build the code, which is something that can be automated reliably once the dependencies exported to Fedora tools are themselves complete and reliable, and Go packages are exported as rpm packages, not hidden in private vendor trees.
?? I bit don't understand you. If you can, please help Jan to improve them.
And then in case of breakage, revert or create a compat package. That's why there is a long chapter dedicated to compat package creation in the proposed guidelines.
Compat packages do not scale.
They scale a lot better than bundling, when an API break results in dozens of projects choosing to vendor a pre-break commit. With each of them choosing a slightly different state, resulting in ten times the amount of code to audit and maintain.
Imagine that you will have in worst case maintain 10s of different versions of same package for every release, doing security backports, etc....
When you do not unbundle you are still shipping the vendored code, you are still responsible for its bugs (security or not). Compat packages do not augment the code surface you have to care about, on the contrary they offer you the possibility to *reduce* the number of code states in the distribution.
Please don't put words in to my mouth. I haven't been talking about not un-bundling, just compat packages.
JC
Plus, they are the occasion, to notify upstreams they should move to a more recent code state. And reduce the number of code states you have to care about yet more. Not all upstreams are dead set against updating their vendoring. A lot of them are nice and responsible and just didn't see they had an API break to manage yet because their use of vendoring hid the problem.
Of course that supposes that creating a compat package does not add a significant package-creation burden. Which I *did* try to avoid. Organization-wise Go will still need a form of exemption to create many many packages easily, because Go code is spread over many many projects, and not consolidated in a few big libraries like in the C world.
Regards,
-- Nicolas Mailhot _______________________________________________ golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
De: "Jakub Cajka"
Filling upstream holes is pretty much the definition of an integrator/distributor role. In Go they are particularly numerous and deep, but Fedora users do want their docker and kubernetes (and Kubernetes, BTW, is astonishingly free of the problems that plague many Go projects, proving it *is* possible to do good release engineering in Go).
??
Just sending some kuddos Kubernetes-side. All the Kubernetes parts we had to work with so far were astonishingly free of API breakage and dependency cycles because Kubernetes people chose their deps carefully and do version their code states. It was a vacation from packaging other Go code.
Pain will not be eased in any significant way as you will still have to carefully evaluate every change so you don't break any dependent package.
Pain *will* be eased in a significant way because we will have the tooling to detect breakage automatically, and because fixes once they're done will be packaged and shared.
Your proposal doesn't provide this kind of tooling, if I'm not mistaken.
Not directly. It does provide the means to easily rev a spec to a new code state (version tag or commit), and it makes deps systematic (so Fedora tooling can accurately detect what is likely to be impacted by a change). Together, those open the way to use generic Fedora testing tools to automate build tests.
If it's minor let's do it. I'm quite sure you will be pleasantly surprised by the level of pain those minor changes remove.
?? I'm in no way opposing your proposal per se.
Thank you
Fedora can detect the breakage which is the first step to fix it. It does not take an army of Go experts, just unbundled packages that can be rebuilt automatically to identify incompatibilities. Go experts are better employed working on fixes than doing rebuild checks that can be automated.
Rebuilds could be automated, analysis of them not easily.
Sure, but at that point 1. you know if the new API is compatible or not with other packages 2. you've saved human packagers the time and effort to do the same rebuild check manually
So that's still a net win, no ?
And nothing stops you for continuing to use those. Except, the golden test will always be to actually try to build the code, which is something that can be automated reliably once the dependencies exported to Fedora tools are themselves complete and reliable, and Go packages are exported as rpm packages, not hidden in private vendor trees.
?? I bit don't understand you. If you can, please help Jan to improve them.
What I meant is that: 1. there are many ways to try to divine if a code state will build, but the final arbiter is the Go compiler 2. to use Fedora-wide QA tools, code objects need to be exported as packages. Every missing test dependency, which is not packaged as a separate Go rpm, is something Fedora QA won't be able to run.
When you do not unbundle you are still shipping the vendored code, you are still responsible for its bugs (security or not). Compat packages do not augment the code surface you have to care about, on the contrary they offer you the possibility to *reduce* the number of code states in the distribution.
Please don't put words in to my mouth. I haven't been talking about not un-bundling, just compat packages.
Sorry if I misunderstood you:(
However, I do think compat packages are a natural effect of unbundling. One does need something to fill in during the time upstream or Fedora work on the effects of an API break. Other Fedora language ecosystems do not manage to avoid them. You can strive to limit compat packages to the minimum amount, and the minimum period, but you can not fully avoid them, once an ecosystem has grown enough it is too big to be perfectly coordinated all the time.
So making compat creation easy and fast, is required to manage a huge amount of interdependent packages, which are not too careful about API stability. Even if one has little love for compat packages as such.
Regards,
On Thu, Feb 1, 2018, at 10:24 AM, nicolas.mailhot@laposte.net wrote:
Not directly. It does provide the means to easily rev a spec to a new code state (version tag or commit), and it makes deps systematic (so Fedora tooling can accurately detect what is likely to be impacted by a change). Together, those open the way to use generic Fedora testing tools to automate build tests.
I appreciate the work you're doing here, but I think the right path for golang (indeed for most other language ecosystems) is to autogenerate specs. Like this: https://github.com/clearlinux/autospec plus language-specific plugins basically.
Also having all of the golang -devel packages in a single git repository with subdirectories so it's easy to have a single PR affect multiple packages at once. (Of course, this model crosses other things beyond golang).
Reference: https://github.com/coreos/coreos-overlay for the latter.
De: "Colin Walters"
I appreciate the work you're doing here,
Thank you
but I think the right path for golang (indeed for most other language ecosystems) is to autogenerate specs.
You're, of course, are entitled to your opinion, but I do not share it :) I think rpm has been an incredibly successful technology. It is used today to package software no one could imagine when rpm was designed, in languages that didn't even exist at that time, and yet one can open spec files from last millenium and still recognize most of the parts.
And during all that time, people have been writing one spec generator after another, that all failed to get serious traction, and make any serious impact.
I think that the reason for rpm success on one hand, and the serial flops of spec generators on the other, it that spec generators are inherently inflexible. They do not deal well with special cases. They do not deal well with mixed projects that require combining several packaging patterns. They are very slow to adapt to rpm enhancements. It's so easy to generate yet another line of code there is no incentive to refactor what's generated to remove parts no longer needed, to strive for succinctness, efficiency and maintainability.
By and large generators generate very verbose low-quality spec code, which can of course, be manually adapted, but is not designed to be adapted by humans, making adaptations hard and inefficient (that seems to be a common pitfall of code generators of any sort). And, when packaging any large pool of software, you will hit many special cases, because software is written by humans, humans are messy, and expecting every software project to follow conventions with the level of strictness expected by spec generators is, IMHO illusory.
spec file syntax, warts and all, seems to hit the sweet spot between the wish to automate as much as possible, and the need to add a few lines of custom code whenever a project needs a little dose of custom handling.
I could, quite easily I think, implement something similar to autospec for Fedora Go packages. All it would take would be to define a variable name for each parameter of the autospec manifest, then call a giant macro that generates the rest of the spec file. I didn't do it not because I don't know how to, nor because it wouldn't work as is for many Go projects, but because it would suffer from the usual generator inflexibility, and inability to handle special cases, and projects that deviate slightly from the norm. I *chose* to split the Go automation in separate rpm macros, that can be combined by packagers as needed in different ways, that can be mixed with the other macros implemented by Fedora, that can be completed by custom spec code when there is a need (that is BTW a balance I didn't get quite right when I automated fonts packaging a few years ago, with hindsight I wish I had designed something a little more adaptable, with more flexibility).
Basically I feel that providing powerful spec verbs, is a more powerful model than expecting to capture every need in rigid autogenerated templates, no matter the amount of switches bolted on later, to manage all the deviations not envisioned when defining the original "simple" generation.
That being said rpm is not perfect and will need to continue to evolve to stay relevant. For example, with the horizontal compilation model of Go, the number of BuildRequires in specs is getting out of hand. So rpm and mock will probably need to evolve towards a model, where BuildRequires are not frozen till the end of %prep. That would allow to infer at least some BuildRequires from what's in the project archive, instead of requiring packagers to declare every single BuildRequires manually, because the build root is setup before the archive is even opened. However I see such things more like a natural evolution of the rpm model, and certainly nothing that requires dumping rpm, or "fixing" it with a generation layer.
Also having all of the golang -devel packages in a single git repository with subdirectories so it's easy to have a single PR affect multiple packages at once.
I can't really disagree there, that's how we work internally. It has proved invaluable every time we hit "damn, that's the twentieth spec that needs this exception, maybe it's time to fold it in common macro code to simplify those specs and avoid writing the exception another time, let's refactors all this in a single commit (or commit series)"
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Thursday, February 1, 2018 4:24:52 PM Subject: Re: <DKIM> [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
De: "Jakub Cajka"
Filling upstream holes is pretty much the definition of an integrator/distributor role. In Go they are particularly numerous and deep, but Fedora users do want their docker and kubernetes (and Kubernetes, BTW, is astonishingly free of the problems that plague many Go projects, proving it *is* possible to do good release engineering in Go).
??
Just sending some kuddos Kubernetes-side. All the Kubernetes parts we had to work with so far were astonishingly free of API breakage and dependency cycles because Kubernetes people chose their deps carefully and do version their code states. It was a vacation from packaging other Go code.
Yes kubernetes is kind of better than most projects, but IMHO still uses lots of not so good practices, although getting bit better over the years.
Pain will not be eased in any significant way as you will still have to carefully evaluate every change so you don't break any dependent package.
Pain *will* be eased in a significant way because we will have the tooling to detect breakage automatically, and because fixes once they're done will be packaged and shared.
Your proposal doesn't provide this kind of tooling, if I'm not mistaken.
Not directly. It does provide the means to easily rev a spec to a new code state (version tag or commit), and it makes deps systematic (so Fedora tooling can accurately detect what is likely to be impacted by a change). Together, those open the way to use generic Fedora testing tools to automate build tests.
It makes it bit easier in some cases, but I wouldn't call it and revolution(as you are trying to sell it) rather than evolution(and omission of some currently mentioned parts), building on top of current practice.
If it's minor let's do it. I'm quite sure you will be pleasantly surprised by the level of pain those minor changes remove.
?? I'm in no way opposing your proposal per se.
Thank you
Fedora can detect the breakage which is the first step to fix it. It does not take an army of Go experts, just unbundled packages that can be rebuilt automatically to identify incompatibilities. Go experts are better employed working on fixes than doing rebuild checks that can be automated.
Rebuilds could be automated, analysis of them not easily.
Sure, but at that point
- you know if the new API is compatible or not with other packages
- you've saved human packagers the time and effort to do the same rebuild
check manually
So that's still a net win, no ?
No really, or not much, until you determine why the builds have failed(I can rebuild of whole distro each day, still I don't have enough spare time to determine and fix all FTBFS issues). You still have to have someone who would analyze those failures.
And nothing stops you for continuing to use those. Except, the golden test will always be to actually try to build the code, which is something that can be automated reliably once the dependencies exported to Fedora tools are themselves complete and reliable, and Go packages are exported as rpm packages, not hidden in private vendor trees.
?? I bit don't understand you. If you can, please help Jan to improve them.
What I meant is that:
- there are many ways to try to divine if a code state will build, but the
final arbiter is the Go compiler 2. to use Fedora-wide QA tools, code objects need to be exported as packages. Every missing test dependency, which is not packaged as a separate Go rpm, is something Fedora QA won't be able to run.
I think that I'm still missing your point. Of course you need to package all your dependencies.
When you do not unbundle you are still shipping the vendored code, you are still responsible for its bugs (security or not). Compat packages do not augment the code surface you have to care about, on the contrary they offer you the possibility to *reduce* the number of code states in the distribution.
Please don't put words in to my mouth. I haven't been talking about not un-bundling, just compat packages.
Sorry if I misunderstood you:(
However, I do think compat packages are a natural effect of unbundling. One does need something to fill in during the time upstream or Fedora work on the effects of an API break. Other Fedora language ecosystems do not manage to avoid them. You can strive to limit compat packages to the minimum amount, and the minimum period, but you can not fully avoid them, once an ecosystem has grown enough it is too big to be perfectly coordinated all the time.
So making compat creation easy and fast, is required to manage a huge amount of interdependent packages, which are not too careful about API stability. Even if one has little love for compat packages as such.
Regards,
-- Nicolas Mailhot _______________________________________________ golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
I'm strongly against general unrestricted practice of compat packages as proposed. If you need compat package you need to work with usptreams on stabilizing the API/project, fork it, or just use COPR as your projects(or its dependencies) are not yet mature enough for Fedora.
JC
De: "Jakub Cajka"
Hi Jakub
I'm strongly against general unrestricted practice of compat packages as proposed. If you need compat package you need to work with usptreams on stabilizing the API/project, fork it, or just use COPR as your projects(or its dependencies) are not yet mature enough for Fedora.
I appreciate the sentiment, and I quite hate compat packages, but the restrictions you want did not work in practice for Fedora.
Making compat package creation hard does not result in faster fixing to use new code. What it actually does is to block the packaging of all the projects that need a more recent package state, because packagers that need the old state for one of their packages, will drag their feet on updating common dependencies till the code of their packages is fixed upstream, or they have the time to fix it themselves.
And blocking packaging of new part means you do not get the new packagers that would join because they're interested in the new parts, and would contribute to the maintenance of common deps (not all of which need compat-ing), and would relieve part of the pressure on the existing packagers, that prevents them from working as much a they'd like to on updating their packages. It's a death spiral. It results in a massively obsolete Go package baselines, full of holes, because all the energy is poured in making existing stuff work, at the expense of onboarding new packages and packagers.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Saturday, February 3, 2018 4:27:36 PM Subject: Re: <DKIM> [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
De: "Jakub Cajka"
Hi Jakub
I'm strongly against general unrestricted practice of compat packages as proposed. If you need compat package you need to work with usptreams on stabilizing the API/project, fork it, or just use COPR as your projects(or its dependencies) are not yet mature enough for Fedora.
I appreciate the sentiment, and I quite hate compat packages, but the restrictions you want did not work in practice for Fedora.
Making compat package creation hard does not result in faster fixing to use new code. What it actually does is to block the packaging of all the projects that need a more recent package state, because packagers that need the old state for one of their packages, will drag their feet on updating common dependencies till the code of their packages is fixed upstream, or they have the time to fix it themselves.
I think one of the main responsibilities of Fedora packager is to work with upstreams, help them mature and generally improve their projects.
And blocking packaging of new part means you do not get the new packagers that would join because they're interested in the new parts, and would contribute to the maintenance of common deps (not all of which need compat-ing), and would relieve part of the pressure on the existing packagers, that prevents them from working as much a they'd like to on updating their packages. It's a death spiral. It results in a massively obsolete Go package baselines, full of holes, because all the energy is poured in making existing stuff work, at the expense of onboarding new packages and packagers.
Regards,
-- Nicolas Mailhot _______________________________________________ golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
Do you have any evidence supporting this claim? From my point of view Jan and other packages has been spending lot of time on on boarding packagers and working on tooling. From my perspective distribution will end up with crazy amount of bitrotting, backport, constant attention requiring package crud..., IMHO basically same as now, apart of it we will have few 1000s of Go based packages with compat names nobody cares about, instead of current 500 or so packages.
JC
----- Mail original ----- De: "Jakub Cajka"
I think one of the main responsibilities of Fedora packager is to work with upstreams, help them mature and generally improve their projects.
Sure but expecting everything to be perfect and consistent before shipping anything just DOES NOT WORK. Reality is not black and white, it's messy with shades of gray
Even the C/C++ guys do not manage to ship a compat-free package ecosystem, and their projects had a few more decades than Go projects to mature, and rpm was pretty much built around C/C++ expectations.
Compat packages are a fact of life in any large software ecosystem, where you can't achieve perfect cohesion. Go is getting *very* large. It needs compat packages. That does not mean there will be hundreds of them because, creating packages is *work*, that does not need they will need maintaining for years, because the aim of each compat package is to get obsoleted as fast as possible, but there will always be some of them at any given time. We are not building a cathedral. Bazaars are messy when full of life.
Do you have any evidence supporting this claim? From my point of view Jan and other packages has been spending lot of time on on boarding packagers and working on tooling.
No one (and certainly not me) is accusing you or Jan not to spend a crazy amount of time and energy trying to make things work. But you did not achieve a satisfactory Go state in Fedora, read again what Owen wrote, I didn't put any word in his mouth, even though I could have written pretty much the same message a few months ago.
Again, no one (and certainly not me) disputes your level of dedication to the "cause". You just made some choices in the past (trying to avoid working within rpm via godep, refusing to include different states of the same Go code in the distro when major Go apps *disagree* on the correct state to include) that didn't work out in practice, with the hindsight of several years of Fedora Go packaging and the Go package state achieved in Fedora after those years.
It's time to adjust those choices a bit.
From my perspective distribution will end up with crazy amount of bitrotting, backport, constant attention requiring package crud..., IMHO basically same as now, apart of it we will have few 1000s of Go based packages with compat names nobody cares about, instead of current 500 or so packages.
Fedora has lots of Go packages no one cares about because they do not permit building the apps people *do* care about.
Building the apps people care about requires a more aggressive update policy, and accepting people will work in parallel on apps, that demand different code states, of common dependencies. And there are not so many of those, I count 18 of them in our repo out of 476 Go packages, even though the work is not completely finished, and finalizing the build of complex tip apps is almost certain to increase the proportion a little. That's awfully *good* and *nice* given the "no Go API compatibility" scarecrows people like to invoke.
You won't *ever* attract a large pool of packagers, to work on a package baseline, that will eventually in some years permit building apps, but never this year, because upstream state is not conflict and compat-free yet.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Monday, February 5, 2018 12:16:14 PM Subject: [Fedora-packaging] Re: <DKIM> Re: Proposed Fedora packaging guideline: More Go packaging
----- Mail original ----- De: "Jakub Cajka"
I think one of the main responsibilities of Fedora packager is to work with upstreams, help them mature and generally improve their projects.
Sure but expecting everything to be perfect and consistent before shipping anything just DOES NOT WORK. Reality is not black and white, it's messy with shades of gray
Even the C/C++ guys do not manage to ship a compat-free package ecosystem, and their projects had a few more decades than Go projects to mature, and rpm was pretty much built around C/C++ expectations.
Compat packages are a fact of life in any large software ecosystem, where you can't achieve perfect cohesion. Go is getting *very* large. It needs compat packages. That does not mean there will be hundreds of them because, creating packages is *work*, that does not need they will need maintaining for years, because the aim of each compat package is to get obsoleted as fast as possible, but there will always be some of them at any given time. We are not building a cathedral. Bazaars are messy when full of life.
Do you have any evidence supporting this claim? From my point of view Jan and other packages has been spending lot of time on on boarding packagers and working on tooling.
No one (and certainly not me) is accusing you or Jan not to spend a crazy amount of time and energy trying to make things work. But you did not achieve a satisfactory Go state in Fedora, read again what Owen wrote, I didn't put any word in his mouth, even though I could have written pretty much the same message a few months ago.
Again, no one (and certainly not me) disputes your level of dedication to the "cause". You just made some choices in the past (trying to avoid working within rpm via godep, refusing to include different states of the same Go code in the distro when major Go apps *disagree* on the correct state to include) that didn't work out in practice, with the hindsight of several years of Fedora Go packaging and the Go package state achieved in Fedora after those years.
It's time to adjust those choices a bit.
From my perspective distribution will end up with crazy amount of bitrotting, backport, constant attention requiring package crud..., IMHO basically same as now, apart of it we will have few 1000s of Go based packages with compat names nobody cares about, instead of current 500 or so packages.
Fedora has lots of Go packages no one cares about because they do not permit building the apps people *do* care about.
Building the apps people care about requires a more aggressive update policy, and accepting people will work in parallel on apps, that demand different code states, of common dependencies. And there are not so many of those, I count 18 of them in our repo out of 476 Go packages, even though the work is not completely finished, and finalizing the build of complex tip apps is almost certain to increase the proportion a little. That's awfully *good* and *nice* given the "no Go API compatibility" scarecrows people like to invoke.
You won't *ever* attract a large pool of packagers, to work on a package baseline, that will eventually in some years permit building apps, but never this year, because upstream state is not conflict and compat-free yet.
Regards,
-- Nicolas Mailhot _______________________________________________ packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org
Our as Fedora or yours company/org? I believe that your contribution of those in to Fedora will be much appreciated. But IMHO even your changes will not change this, if you don't have few dedicated packager around to do all the bidding.
JC
----- Mail original ----- De: "Jakub Cajka"
Our as Fedora or yours company/org? I believe that your contribution of those in to Fedora will be much appreciated.
Our was meaning the set of specs we are preparing for inclusion. Can't really share them before the macros they depend on are merged in rawhide.
You can grab most of those from https://copr.fedorainfracloud.org/coprs/nim/More_Go_Packaging
before I nuke it to launch a new build from scratch (if seems a bit of grpc bootstraping failed this run so that's back to build log analysis, add constrains, scratch, retry, wait 3 days for the result. I soooo hate the yum version in copr, every time there is a dumb decision to make it will make it, it manages to make yum in EL7 look smart)
But IMHO even your changes will not change this, if you don't have few dedicated packager around to do all the bidding.
Sure, the changes will only remove some barriers to new Go packagers, they can't replace those packagers, or the people who take care of the baseline core.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: "Development discussions related to Fedora" devel@lists.fedoraproject.org Cc: golang@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Monday, February 5, 2018 3:27:01 PM Subject: Re: [Fedora-packaging] Re: <DKIM> Re: Proposed Fedora packaging guideline: More Go packaging
----- Mail original ----- De: "Jakub Cajka"
Our as Fedora or yours company/org? I believe that your contribution of those in to Fedora will be much appreciated.
Our was meaning the set of specs we are preparing for inclusion. Can't really share them before the macros they depend on are merged in rawhide.
You can grab most of those from https://copr.fedorainfracloud.org/coprs/nim/More_Go_Packaging
before I nuke it to launch a new build from scratch (if seems a bit of grpc bootstraping failed this run so that's back to build log analysis, add constrains, scratch, retry, wait 3 days for the result. I soooo hate the yum version in copr, every time there is a dumb decision to make it will make it, it manages to make yum in EL7 look smart)
You have to explicit about the specs, TBH I have had more fun with dnf, when it fist came out.
I believe that you can already open pull requests and review requests stating the dependency(so it doesn't get merger too soon) as it will take it some time to merge and details can get ironed out during merging/review. As IMHO your guidelines will require whole distro change/rebuild to make it compliant. I feel like you haven't been engaging with current maintainers much.
JC
But IMHO even your changes will not change this, if you don't have few dedicated packager around to do all the bidding.
Sure, the changes will only remove some barriers to new Go packagers, they can't replace those packagers, or the people who take care of the baseline core.
Regards,
-- Nicolas Mailhot _______________________________________________ golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
On 02/01/2018 05:49 AM, Jakub Cajka wrote:
On contrary Fedora is trying to fill the hole that upstream Go projects dug them selves in to. IMNHO Go have traded any subjectively perceived "RPM/deb hell" for even deeper levels of "Go (vendor) hell".
This unfortunately became a trend: "the old packaging is dead, long live the new packaging!" (not just go, but also pip/conda/npm/...). I think Fedora is doing a good job getting it under control, leading upstream projects to get their act together---and it demonstrated that this works, and actually makes upstream better.
Hi Nicolas,
I'm embarrassed to admit that before I sent my mail I carefully read over ... the old PackageDrafts/Go :-( My only excuse is that it was in my browser history.
Having actually read the relevant parts of "More Go Packaging", the explanation of compat packages and notification procedures does make the intended operation clearer, though the social and technical barrier to a packager new to Fedora will still be high if packaging their target package requires creating a compat package and fixing multiple other packages. As you say, once Fedora packaging catches up to the modern state, things will be better, because the onus won't be on the new packager to do the work of updating old libraries.
I still worry that Fedora is not big enough to move the status-quo in the Go world - to get the point where Go programs require github.com/foo/bar >= 1.2.3 and actually have been tested with a multiple versions in that range, not exactly the one vendored version shipped with the program. Sp we might end up with lots of Fedora-specific bugs that the upstream doesn't care about. But if the Fedora Go community can get some size and momentum and start pushing fixes upstream, that will certainly be a positive force for reducing Go programs stuck on random old versions of dependencies!
I haven't had time to read through the entire proposal, but it certainly looks like a major step forward!
Owen
On Thu, Feb 1, 2018 at 5:21 AM, nicolas.mailhot@laposte.net wrote:
----- Mail original ----- De: "Owen Taylor"
Hi Owen,
Is there a guide for Fedora packagers about how to handle unbundling for golang packages? The draft guidelines don't seem to go into any details.
I don't think there is, nor that it is necessarily needed. The posted guidelines should be sufficient technically, there are no magic I know of I didn't document, the rest is just a lot of work (but see ↓)
I've looked at packaging a few golang packages unbundled, and have immediately run into:
A) lots of unpackaged dependencies B) dependencies that are packaged in Fedora with different, often much older versions
Yes the state of Go packages in Fedora is pretty sad right now. I wouldn't expect anyone to be able to package anything but the most trivial app in unbundled mode. Too many common parts are missing, when they are not missing they are too old, trying to update an existing Go package is an exercise in frustration (too much package-specific shell code, that is difficult to understand and does not really work with new code versions), and trying to update or add missing parts just reveals more breakage and work to be done.
However, accepting to package complex Go apps in bundled mode is how Fedora got to this state in the first place. In plain speak, bundling (vendoring in Go linguo) just does not scale. You need an awful lot of manpower to audit the hundreds of other projects each app bundles, bundling removes all the package tooling that may help you to do so, and the result is not shared with any other package, or with other versions of the same package, so you get zero positive network effects. Worse, big bundled apps that do not actually try to work in unbundled mode, do not actually test the code they export (they are bundling, remember) so the result is toxic to small Go packages that try to work with them.
So bundling parts is a direct road to bundling everything, bundling everything is a direct road to bundling everything blindly, bundling everything blindly is error-prone and dangerous (because upstreams are only human and do make lots of mistakes), and pretty much removes any value Fedora can add to end users, to other parts of Fedora that would like to integrate with Go software, or to the upstream projects themselves (no Fedora QA, no stream of Fedora-originated fixes, no Fedora pressure to stabilize parts when upstream is lost in tunnel effect mode and does not realize that it is wasting everyone's time starting with its own).
Therefore, trying to get all this it a better state, requires the following steps IMHO:
- completely refactor our Go packaging style it's less painful to update
Go packages and they do not need a packager with deep knowledge of package-specific shell glue. That takes documentation, and factoring out common Go packaging functions in shared rpm code (macros and autodeps) → that's what I posted
- using the new documentation and tooling to clean up years of Fedora
technical debt, and create a new set of up-to-date Go packages that can serve as new baseline → I have hundreds of specs that I'm waiting for step 1 to complete to submit. They won't constitute a full baseline by themselves, they're not all 100% done, it's too much work to do alone but working with others requires the common macros and documentation to be merged and adopted. This step is going to be painful I'm afraid, Fedora dug itself a deep hole, leaving it is going to be hard.
- hopefully, the result will be streamlined enough it won't be too
painful to keep up to date, having an up to date baseline will help attract new Go packagers like you, everyone will benefit and be happy. We had to package some ostree bits for example to create this baseline, and I'm pretty sure ostree people within Fedora would prefer to maintain those themselves, if the rest of the Fedora go universe didn't make it too hard
B) more intimidating. It seems like to package the new package, I have to get the maintainer of the library to upgrade the version,
The guidelines and automation aim at making upgrading easy, and avoid one package or packager blocking others
and someone needs to rebuild everything that depends on the rebuilt library and test that the rebuilt packages work.
I hope that normalizing Fedora Go packages means it will be possible to automate QA tests such as "try to rebuild all the packages that depend on golang(foo) every time you see the package providing golang(foo) changing, and report what broke"
That would be expensive computing-side but a lot less expensive and long than expecting each Go packager to do it himself with his own means. And that is certainly not overkill, given how lax Go projects are about maintaining API stability.
And then in case of breakage, revert or create a compat package. That's why there is a long chapter dedicated to compat package creation in the proposed guidelines.
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
De: "Owen Taylor"
I'm embarrassed to admit that before I sent my mail I carefully read over ... the old PackageDrafts/Go :-( My only excuse is that it was in my browser history.
NP, that gave you some context on where Fedora is today.
Having actually read the relevant parts of "More Go Packaging", the explanation of compat packages and notification procedures does make the intended operation clearer,
Thank you. Do not hesitate to improve the wording in the wiki, or post suggestions here, if you have ideas on how to make it clearer, simpler or more efficient
though the social and technical barrier to a packager new to Fedora will still be high if packaging their target package requires creating a compat package and fixing multiple other packages.
Unfortunately, packaging something in Go is still going to be harder than packaging software in another language in the near term :( I'd love to find more magic bullets.
I still worry that Fedora is not big enough to move the status-quo in the Go world - to get the point where Go programs require github.com/foo/bar >= .2.3 and actually have been tested with a multiple versions in that range, not exactly the one vendored version shipped with the program.
That's a legitimate worry, yes. However given container and cloud people are massively adopting Go, that critical cloud software is now mostly written in Go, I don't think Fedora can afford to pass on Go and still stay relevant server-side. That's even more true for Fedora downstreams and Fedora's main sponsor.
So I guess it all boils down to strategic choices for Fedora and Red Hat: invest in Go packaging, even if it *is* painful, or pass, and lose relevance server-side in the near future. Red Hat certainly has the assets, with Openshift and Coreos, to influence the Go world in a Fedora-positive way. I don't know if it will choose, or manage, to leverage them. It can not fail to notice the many Go projects that propose their software as Ubuntu Docker images today. It can not fail to notice that Jakub and Jan, with all their qualities, are a tad overworked and not really sufficient to pull Fedora Go forward the required amount. The Go ecosystem has just grown too big. Lastly, I understand the temptation to let Fedora and RHEL slip in favor of product lines more profitable in the near term.
On my employer's side we *do* wish to stay relevant in a Cloud world. My Go contributions are an attempt to partner with the Fedora Centos and RHEL communities on that. If the partnership does not bear fruits, and Fedora/Centos/RHEL do not move in a direction we can use, we'll have to invest elsewhere. I'd regret it but this proposal and the spec dump that will follow are just too much work to do on one's free time. And employer time requires results.
I haven't had time to read through the entire proposal, but it certainly looks like a major step forward!
Please finish reading it and propose all the fixes and comments and improvements you want! Your experience is appreciated.
Regards,
On mardi 30 janvier 2018 16:11:49 CET nicolas.mailhot@laposte.net wrote:
Hi,
Now the technical PR is submitted https://src.fedoraproject.org/rpms/go-srpm-macros/pull-request/1
and waiting for action from the go-srpm-macros maintainers, I took (quite a long) time to refresh and flesh out the corresponding packaging guidelines proposal. It should be fairly complete now:
https://fedoraproject.org/wiki/More_Go_packaging
I'd appreciate review to check if I have forgotten an important case, if people understand the text, if they have enhancements or corrections to propose, and so on.
Then I will push it FPC side again.
Actual practice should be fairly simple and self-explanatory, the proposal length can be scary but that's because it documents all kinds of corner cases that required digging through specs and mailing lists to find resolution examples before. The basic Go packaging skeleton will be sufficient is most cases without requiring to read any documentation.
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
Hello Nicolas,
Could you add two full examples, one for binary package, one for library package, like in https://fedoraproject.org/wiki/PackagingDrafts/ Go#Sample_RPM_Spec
Putting it all together (https://fedoraproject.org/wiki/ More_Go_packaging#Putting_it_all_together) is nice but don't understand if it is for a binary or a library.
Also, is there any plan to update Gofed with these new guidelines? If it's not automated like today, we will lose time instead of gaining some in the end.
Best regards,
Robert-André
Hi Robert André
That's an interesting request
I guess you can't figure if the example is for building binaries or Go libs, because there is no hard frontier between both cases in the proposed guidelines
In Go, everything is effectively a code library that can be reused elsewhere. So the approach taken in the proposed guidelines, is that everything is a code library, some of which happen to build binaries, or setup compatibility symlinks, or need bootstrapping
It's a bit of a Lego guideline, you assemble the spec blocs you need, and ignore those you don't need. The example was chosen to include as many blocks as possible, with the walkthrough explaining their respective functions. All the blocks are very short and succinct, don't search for long runs of shell code like in the old guidelines
----- Mail original ----- De: "Nicolas Mailhot"
It's a bit of a Lego guideline, you assemble the spec blocs you need, and ignore those you don't need. The example was chosen to include as many blocks as possible, with the walkthrough explaining their respective functions. All the blocks are very short and succinct, don't search for long runs of shell code like in the old guidelines
That being said if people feel it's useful I can add a barebones lib-only example with none of the optional blocks in. However, I fear that the more ways the page explains the same stuff, the more confusing and difficult to maintain it will be.
So, can people chime in to say whether they feel a supplementary example is useful of counter productive?
Regards,
----- Original Message -----
From: "Robert-André Mauchin" zebob.m@gmail.com To: devel@lists.fedoraproject.org Cc: golang@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Friday, February 2, 2018 4:54:10 PM Subject: Re: Proposed Fedora packaging guideline: More Go packaging
On mardi 30 janvier 2018 16:11:49 CET nicolas.mailhot@laposte.net wrote:
Hi,
Now the technical PR is submitted https://src.fedoraproject.org/rpms/go-srpm-macros/pull-request/1
and waiting for action from the go-srpm-macros maintainers, I took (quite a long) time to refresh and flesh out the corresponding packaging guidelines proposal. It should be fairly complete now:
https://fedoraproject.org/wiki/More_Go_packaging
I'd appreciate review to check if I have forgotten an important case, if people understand the text, if they have enhancements or corrections to propose, and so on.
Then I will push it FPC side again.
Actual practice should be fairly simple and self-explanatory, the proposal length can be scary but that's because it documents all kinds of corner cases that required digging through specs and mailing lists to find resolution examples before. The basic Go packaging skeleton will be sufficient is most cases without requiring to read any documentation.
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
Hello Nicolas,
Could you add two full examples, one for binary package, one for library package, like in https://fedoraproject.org/wiki/PackagingDrafts/ Go#Sample_RPM_Spec
Putting it all together (https://fedoraproject.org/wiki/ More_Go_packaging#Putting_it_all_together) is nice but don't understand if it is for a binary or a library.
Also, is there any plan to update Gofed with these new guidelines? If it's not automated like today, we will lose time instead of gaining some in the end.
Best regards,
Robert-André
I think that it would be best if Nicolas could fold his proposal in to the original draft as optional part for simple library/binary packages.
As his proposal doesn't cover at least two major use cases, i.e. separate packaging of tests(they have no place in devel package as they artificially inflate build root size) and shipping pre-built shared libraries.
Just my opinion,
JC
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
----- Mail original ----- De: "Jakub Cajka"
Hi Jakub
I think that it would be best if Nicolas could fold his proposal in to the original draft as optional part for simple library/binary packages.
Frankly, that's a lot of work and churn, I don't want the new parts to be refused because of something in the old parts, and I certainly do not want to spend weeks replacing bits only to put them back in and so on while people made up their mind on the things they want replaced or not. The new parts are pretty much autonomous and complete, they are sufficient to create working Fedora Go packages, they are ready for FPC review.
If someone wants to extract the ready non discussion parts of the old page and get them approved I can work with him to merge both elements
I didn't want to write about the old page, but since you put it on the table.
It is full of digressions and elements of no evident applied value while packaging Go software. It's not an operational "how to do stuff" document it's a "here are various things you may like to know about Go that may or may not help you create Fedora Go packages, if they do not help you forget about it and run gofed". There are too many WIP non operational bits in the old pages to allow merging while in an approval process. And I'm sure any attempt to strip the WIP bits from my side will be met with energetic protests.
People, if you want that page to be ever approved by FPC make it more focused and accurate. Remove anything that does not explain to a Fedora Go packager how to write a Fedora Go spec file and what to ask upstream Go projects in a Fedora packager role. And make sure what remains is succinct, easy to read, and applicable without undocumented holes or gofed magic.
I know that's a lot of non-fun work. I did this work on the new page. I don't particularly *like* writing documentation. For every line I put in the new page, I had to ask myself "is the value to the Fedora packager sufficient to justify the time spend writing the text, formatting it, linking in the right place, proofing the result". I didn't put in stuff I could not justify cleaning up myself. If it was too much work to document it was either of insufficient value or better automated rather than explained in a manual process.
Any part of the text that can not be finished or serves another role has no place in a guideline submitted to FPC. It should be nuked or moved to a separate wiki page. All the half-finished and non operational stuff in there is the reason the draft has been stuck in pre-approval stage for four years. Just put yourself in FPC's place, its mission is to confirm a guideline is ready to be used by the average Fedora packager, not to produce it from half-finished half-related messy notes of the domain experts.
As his proposal doesn't cover at least two major use cases, i.e. separate packaging of tests(they have no place in devel package as they artificially inflate build root size)
At this point of time my mind is pretty much set. I won't do separate packaging of tests because: 1. that raises complex dependency handling questions 2. the average Go project test code is full of crufty junk 3. the average Go test depends on assets not tracked within Go 4. Go is not designed to separate elements shipped in the same directory 5. no one could answer when I asked the *three* Fedora lists what they were using the existing test packages for 6. from what I've seen many of the existing test packages simply can not work because they fail to include elements they need 7. even core Go people refuse to touch the subject with a 10 foot pole. Sam Boyer tried to demo a very small part of what would be necessary this week end at FOSDEM and this part of his demo *failed*. I don't have the level of Go understanding Sam Boyer Has.
I am ready to work with people who want to make separate test packages and I tried very hard to make them possible in the future but I won't spend time trying to ship half-baked stuff that does not work and that no one has a need for.
The proposed patterns strip test code from devel packages so build root sizes are safe (except for the original package build root, since I *do* execute unit tests in check, and they pull their deps in the build root).
You want to test one of my packages, fine, just rebuild it. That will run unit tests *and* build the project binaries (which are also a form of code testing, and a very thorough one at that).
and shipping pre-built shared libraries.
The pre-build shared libraries the old guidelines refuse to build or document? Why exactly are you stating it's a major use case, Fedora is not doing it today.
"By default, the standard golang compiler produces static libraries. There is little value in shipping these prebuilt, especially since these libraries are very specifically tied to the exact minor release of the golang compiler."
"Presently the shared object libraries produced by GCC-Go are not usable"
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Monday, February 5, 2018 4:48:31 PM Subject: Re: Proposed Fedora packaging guideline: More Go packaging
----- Mail original ----- De: "Jakub Cajka"
Hi Jakub
I think that it would be best if Nicolas could fold his proposal in to the original draft as optional part for simple library/binary packages.
Frankly, that's a lot of work and churn, I don't want the new parts to be refused because of something in the old parts, and I certainly do not want to spend weeks replacing bits only to put them back in and so on while people made up their mind on the things they want replaced or not. The new parts are pretty much autonomous and complete, they are sufficient to create working Fedora Go packages, they are ready for FPC review.
If someone wants to extract the ready non discussion parts of the old page and get them approved I can work with him to merge both elements
I didn't want to write about the old page, but since you put it on the table.
It is full of digressions and elements of no evident applied value while packaging Go software. It's not an operational "how to do stuff" document it's a "here are various things you may like to know about Go that may or may not help you create Fedora Go packages, if they do not help you forget about it and run gofed". There are too many WIP non operational bits in the old pages to allow merging while in an approval process. And I'm sure any attempt to strip the WIP bits from my side will be met with energetic protests.
Have you tried that? What make you assume that? I'm sure that if you do it in constructive way, they will be accepted.
People, if you want that page to be ever approved by FPC make it more focused and accurate. Remove anything that does not explain to a Fedora Go packager how to write a Fedora Go spec file and what to ask upstream Go projects in a Fedora packager role. And make sure what remains is succinct, easy to read, and applicable without undocumented holes or gofed magic.
I believe that technically exhausting document is needed as Go packaging is far from trivial. Sure it would be great to have (trivial) quick start guide. I think that your proposal fits that bill more than full documentation.
I know that's a lot of non-fun work. I did this work on the new page. I don't particularly *like* writing documentation. For every line I put in the new page, I had to ask myself "is the value to the Fedora packager sufficient to justify the time spend writing the text, formatting it, linking in the right place, proofing the result". I didn't put in stuff I could not justify cleaning up myself. If it was too much work to document it was either of insufficient value or better automated rather than explained in a manual process.
Any part of the text that can not be finished or serves another role has no place in a guideline submitted to FPC. It should be nuked or moved to a separate wiki page. All the half-finished and non operational stuff in there is the reason the draft has been stuck in pre-approval stage for four years. Just put yourself in FPC's place, its mission is to confirm a guideline is ready to be used by the average Fedora packager, not to produce it from half-finished half-related messy notes of the domain experts.
I believe I can see myself in their shoes(sure not 100% accurately) and I don't see anything that you are describing from their POV, it would be good to have the guidelines sooner, but I will have, as you, rather something complete than something half baked and semi broken. Please note that they are WIP and everybody is welcomed to contribute in constructive way. I don't think that there is any request for FPC to create the guidelines on their own and honestly it would rather bad idea IMHO.
As his proposal doesn't cover at least two major use cases, i.e. separate packaging of tests(they have no place in devel package as they artificially inflate build root size)
At this point of time my mind is pretty much set. I won't do separate packaging of tests because:
- that raises complex dependency handling questions
- the average Go project test code is full of crufty junk
- the average Go test depends on assets not tracked within Go
- Go is not designed to separate elements shipped in the same directory
- no one could answer when I asked the *three* Fedora lists what they were
using the existing test packages for
They are used primarily to minimize build root size, by reducing the deps size and code size.
- from what I've seen many of the existing test packages simply can not work
because they fail to include elements they need
Please file bugs or pull requests. We need to work together to improve the experience. I'm sure that you are including all the dependencies in your devel sub-package(that contain the tests) to be able to run them locally.
- even core Go people refuse to touch the subject with a 10 foot pole. Sam
Boyer tried to demo a very small part of what would be necessary this week end at FOSDEM and this part of his demo *failed*. I don't have the level of Go understanding Sam Boyer Has.
What has been demoed?? Unfortunately I could not make it there. Have you met with Jan there?
I am ready to work with people who want to make separate test packages and I tried very hard to make them possible in the future but I won't spend time trying to ship half-baked stuff that does not work and that no one has a need for.
The proposed patterns strip test code from devel packages so build root sizes are safe (except for the original package build root, since I *do* execute unit tests in check, and they pull their deps in the build root).
Your packaging proposal seems fairly monolithic and disregarding any prior art, so extensions will be very painful.
You want to test one of my packages, fine, just rebuild it. That will run unit tests *and* build the project binaries (which are also a form of code testing, and a very thorough one at that).
I don't think that is great user experience nor packagers/maintainers experience.
and shipping pre-built shared libraries.
The pre-build shared libraries the old guidelines refuse to build or document? Why exactly are you stating it's a major use case, Fedora is not doing it today.
?? I'm bit confused. Yes, it is not mentioned there yet as Jan have been working on stabilizing APIs, so it will be feasible and added there.
"By default, the standard golang compiler produces static libraries. There is little value in shipping these prebuilt, especially since these libraries are very specifically tied to the exact minor release of the golang compiler."
"Presently the shared object libraries produced by GCC-Go are not usable"
You have mentioned that the API breakage is not that big issue, so this led me to conclude that shared lib/bin packaging is now feasible. I believe that any Go packaging guideline that doesn't implement this side in working fashion is not FPC ready IMNHO.
JC
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
----- Mail original ----- De: "Jakub Cajka"
Hi Jakub,
And I'm sure any attempt to strip the WIP bits from my side will be met with energetic protests.
Have you tried that? What make you assume that? I'm sure that if you do it in constructive way, they will be accepted.
My experience with humans is that they get attached to their bits of text even when they have no time to finish them. But yes, I haven't tried, I don't want to see a wiki page for a month after writing and formatting and revising two separate packaging drafts.
I believe that technically exhausting document is needed as Go packaging is far from trivial. Sure it would be great to have (trivial) quick start guide. I think that your proposal fits that bill more than full documentation.
IMHO that's exactly what FPC expects: a quick start document, not an in-depth encyclopædia. In-depth is too hardcore to be validated by domain laymen, is useless to onboard new packagers (which is the primary objective of Fedora guidelines), is never finished because the state of the art changes, and can't be applied to check whether a spec is good or not because in-depth concerns *complex* cases where the "right" choice is not obvious, depends on many factors that change over time, and usually requires asking the domain experts directly.
So you have the simple common cases, which are sufficient for most packagers, and can be addressed by a formal document FPC can review and enforce, and "other things" which are valuable but can not ever attain the formalness and strictness level expected of Fedora guidelines, require continuous freedom to edit, and are hosted in other Fedora wiki pages. At least that's how real-life law is written (law text + expert commentaries) and how other Fedora packaging SIGs function and got their guidelines approved.
They are used primarily to minimize build root size, by reducing the deps size and code size.
Stripping example material from gopath installation filelists achieves the same objective without all the complexity of managing packages that share directories Go considers indivisible.
- even core Go people refuse to touch the subject with a 10 foot pole. Sam
Boyer tried to demo a very small part of what would be necessary this week end at FOSDEM and this part of his demo *failed*. I don't have the level of Go understanding Sam Boyer Has.
What has been demoed?? Unfortunately I could not make it there
Sam Boyer presented the current state of his go dep prototype, and explained his design choices (for example no nested projects which means the go import path root becomes a core concept and it's useless to invest in splitting projects as that usually requires nesting). He ended up stating go dep will ignore tests by default, tried to demo optional test handling, and that failed. He had another conference the next day I couldn't get to as I was stuck in another part of the campus.
Have you met with Jan there?
Unfortunately FOSDEM was its usual crazy stuff with multiple interesting conferences at any given time and no hope of being admitted in a room if you're late. So I've just been running from conf room to conf room without any hope of meeting anyone :(
Your packaging proposal seems fairly monolithic and disregarding any prior art, so extensions will be very painful.
Well, the creation of the proposal was a long suite of extending to cover more cases (that's why some macros are rather long and convoluted), so I'm quite sure extension is possible. Of course I may miss something, if you identify a roadblock please point it out.
You want to test one of my packages, fine, just rebuild it. That will run unit tests *and* build the project binaries (which are also a form of code testing, and a very thorough one at that).
I don't think that is great user experience nor packagers/maintainers experience.
That's pretty much how rpm prefers it to be (use %check and Builddeps) and also how Go prefers it to be (no provision for splitting directories). Of course anyone interested can try to fight the design of both of them to achieve something else, and would try to help, but I'm not interested in being this person.
and shipping pre-built shared libraries.
The pre-build shared libraries the old guidelines refuse to build or document? Why exactly are you stating it's a major use case, Fedora is not doing it today.
?? I'm bit confused. Yes, it is not mentioned there yet as Jan have been working on stabilizing APIs, so it will be feasible and added there.
My maybe naïve POW is that shared libraries are just another container for the elements currently installed in %gopath/src, so doing them is mostly adding a separate stage in %build and making %goinstall install the result instead of installing raw files.
The hardest part is to get compatible code bases, defining what needs stripping in production, and defining shared libraries numbering.
The proposal helps for the two first points and I have no idea or opinion on the third.
The rest is macro plumbing that should not require a lot of spec surgery once specs are mostly free of custom shell code.
You have mentioned that the API breakage is not that big issue, so this led me to conclude that shared lib/bin packaging is now feasible.
API breakage is not a showstopper with the static linking Go prefers because careful BuildDep construction can ensure only one (and the right one) dependency code state ends up in the buildroot, and BuildDeps of the next app can select another code state. So it's mostly a mater of creating the necessary -devel compat packages and defining BuildDep constrains.
It is a showstopper for shared libs because you can't install several versions of the same shared lib without separate identifiers/release ids to disambiguate them, Go upstreams do not provide those identifiers today and I doubt creating a Fedora-specific identifier namespace at the lib level would be a good idea. Though, the few projects that version their import paths, and make sure to only extend their APIS within a major release, could be libified today (theoretically, I've not checked how to do it). I'm not sure there are enough of them to be worth multiple packaging style
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Tuesday, February 6, 2018 7:34:03 PM Subject: Re: Proposed Fedora packaging guideline: More Go packaging
----- Mail original ----- De: "Jakub Cajka"
Hi Jakub,
And I'm sure any attempt to strip the WIP bits from my side will be met with energetic protests.
Have you tried that? What make you assume that? I'm sure that if you do it in constructive way, they will be accepted.
My experience with humans is that they get attached to their bits of text even when they have no time to finish them. But yes, I haven't tried, I don't want to see a wiki page for a month after writing and formatting and revising two separate packaging drafts.
I believe that technically exhausting document is needed as Go packaging is far from trivial. Sure it would be great to have (trivial) quick start guide. I think that your proposal fits that bill more than full documentation.
IMHO that's exactly what FPC expects: a quick start document, not an in-depth encyclopædia. In-depth is too hardcore to be validated by domain laymen, is useless to onboard new packagers (which is the primary objective of Fedora guidelines), is never finished because the state of the art changes, and can't be applied to check whether a spec is good or not because in-depth concerns *complex* cases where the "right" choice is not obvious, depends on many factors that change over time, and usually requires asking the domain experts directly.
So you have the simple common cases, which are sufficient for most packagers, and can be addressed by a formal document FPC can review and enforce, and "other things" which are valuable but can not ever attain the formalness and strictness level expected of Fedora guidelines, require continuous freedom to edit, and are hosted in other Fedora wiki pages. At least that's how real-life law is written (law text + expert commentaries) and how other Fedora packaging SIGs function and got their guidelines approved.
They are used primarily to minimize build root size, by reducing the deps size and code size.
Stripping example material from gopath installation filelists achieves the same objective without all the complexity of managing packages that share directories Go considers indivisible.
??, I have been talking about tests. From Go perspective source files and test source file, testdata are two distinctive sets. From my POV if your code depends for regular builds on tests code it means that there is something horribly wrong with your project(regardless of language used).
Docs and examples would be great too, it should be in guidelines that packager must use reasonable effort to create subpackages with them.
- even core Go people refuse to touch the subject with a 10 foot pole.
Sam Boyer tried to demo a very small part of what would be necessary this week end at FOSDEM and this part of his demo *failed*. I don't have the level of Go understanding Sam Boyer Has.
What has been demoed?? Unfortunately I could not make it there
Sam Boyer presented the current state of his go dep prototype, and explained his design choices (for example no nested projects which means the go import path root becomes a core concept and it's useless to invest in splitting projects as that usually requires nesting). He ended up stating go dep will ignore tests by default, tried to demo optional test handling, and that failed. He had another conference the next day I couldn't get to as I was stuck in another part of the campus.
Cool, there is recording for anyone interested https://fosdem.org/2018/schedule/event/dep/.
Have you met with Jan there?
Unfortunately FOSDEM was its usual crazy stuff with multiple interesting conferences at any given time and no hope of being admitted in a room if you're late. So I've just been running from conf room to conf room without any hope of meeting anyone :(
Your packaging proposal seems fairly monolithic and disregarding any prior art, so extensions will be very painful.
Well, the creation of the proposal was a long suite of extending to cover more cases (that's why some macros are rather long and convoluted), so I'm quite sure extension is possible. Of course I may miss something, if you identify a roadblock please point it out.
You want to test one of my packages, fine, just rebuild it. That will run unit tests *and* build the project binaries (which are also a form of code testing, and a very thorough one at that).
I don't think that is great user experience nor packagers/maintainers experience.
That's pretty much how rpm prefers it to be (use %check and Builddeps) and also how Go prefers it to be (no provision for splitting directories). Of course anyone interested can try to fight the design of both of them to achieve something else, and would try to help, but I'm not interested in being this person.
and shipping pre-built shared libraries.
The pre-build shared libraries the old guidelines refuse to build or document? Why exactly are you stating it's a major use case, Fedora is not doing it today.
?? I'm bit confused. Yes, it is not mentioned there yet as Jan have been working on stabilizing APIs, so it will be feasible and added there.
My maybe naïve POW is that shared libraries are just another container for the elements currently installed in %gopath/src, so doing them is mostly adding a separate stage in %build and making %goinstall install the result instead of installing raw files.
The hardest part is to get compatible code bases, defining what needs stripping in production, and defining shared libraries numbering.
The proposal helps for the two first points and I have no idea or opinion on the third.
The rest is macro plumbing that should not require a lot of spec surgery once specs are mostly free of custom shell code.
You have mentioned that the API breakage is not that big issue, so this led me to conclude that shared lib/bin packaging is now feasible.
API breakage is not a showstopper with the static linking Go prefers because careful BuildDep construction can ensure only one (and the right one) dependency code state ends up in the buildroot, and BuildDeps of the next app can select another code state. So it's mostly a mater of creating the necessary -devel compat packages and defining BuildDep constrains.
Not really, mostly with static linking you are fine if you change deps as long as you don't touch basic runtime(glibc,...) and binaries will keep on working. In dynamic linking when you change the deps in incompatible way(break API/ABI) you break the runtime for all built binaries and you need to do rebuild to just keep distro working.
It is a showstopper for shared libs because you can't install several versions of the same shared lib without separate identifiers/release ids to disambiguate them, Go upstreams do not provide those identifiers today and I doubt creating a Fedora-specific identifier namespace at the lib level would be a good idea. Though, the few projects that version their import paths, and make sure to only extend their APIS within a major release, could be libified today (theoretically, I've not checked how to do it). I'm not sure there are enough of them to be worth multiple packaging style
Regards,
-- Nicolas Mailhot
You can version the so's arbitrarily for example and have explicit non general require on the devel, I think this really minor implementation detail from packaging discussion POV.
JC
golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
Hi Nicolas,
before I start arguing, I appreciate all your hard work on improving the Go guidelines, spending so much time explaining reasoning behind your decision. Also appreciating the effort on making the life easier for packagers, not just in the Go land, but throughout the distribution as well. There is a lot of great ideas in your Go proposal I would like to see implemented and widely used. The more transparent and easy-to-use spec files, the better for all maintainers. Let's keep moving forward, improving the infrastructure for other folks and pushing new ideas and improvements into practical solutions.
On 12/17/2017 08:11 AM, nicolas.mailhot@laposte.net wrote:
Hi,
I am proposing for inclusion a set of rpm technical files aimed at automating the packaging of forge-hosted projects.
- Packaging draft:https://fedoraproject.org/wiki/More_Go_packaging
-https://pagure.io/packaging-committee/issue/734
- go-srpm-macros RFE with the technical files:https://bugzilla.redhat.com/show_bug.cgi?id=1526721
This proposal is integrated with and depends on thehttps://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft It builds on the hard work of the Go SIG and reuses the rpm automation ofhttps://fedoraproject.org/wiki/PackagingDrafts/Go when it exists, and produces compatible packages.
Can you describe what you mean by "compatible packages"? I mentioned a list of things that you did not answer fully. Most important to me: - your macros do not generate build-time dependencies, which I see as one of the drawbacks. Do you plan to ignore them? - support for more subpackages. You describe how to specify which files go to which packages (https://fedoraproject.org/wiki/More_Go_packaging#Separate_code_packages) but you don't say how list of provided packages and a list of dependencies are generated for the subpackages. - reproducible evaluation of macros. Either for running automatic analysis over spec files or for debugging purposes. I would like the macros to be built in top of binaries I can run separately. With Jakub (jcajka@redhat.com) we were discussing many times we could provide a minimal rpm built on top of gofedlib that would extract all the necessary pieces, including the naming convention so everyone can import provided library guided by the same rules as the guidelines enforce. I would like to see us synchronizing on this topic at some point.
What it does:
- drastically shorter spec files, up to 90% in some cases, often removing hundreds of lines per spec.
+1 here, the shorter the spec file the better as long as it is still clear and transparent By better I mean: - easier to read, easier to understand, easier to adopt - less places to change and look if a spec file change is needed By clear I mean: - it is obvious what the spec file is actually declaring to do - it is easy to customize various parts (e.g. list of tests, list of dependencies)
- simple, packager-friendly spec syntax
+1 as long as each macro provides a single functionality. E.g. - generate a list of provided packages - generate a list of tests - generate a list of dependencies
+1 to the %goname, %gourl, %gosource, %goinstall, %__goprovides, %_gorequires and other usefull macros as long as they are simple and transparent
As long as the macros are optional and user can use any of them if it is suitable (suitable meant per a packager judgment).
- automated package naming derived from the native identifier (import path). No more packages names without any relation with current upstream naming.
Can you provide a list of package names that diverge from this convention?
For historical reasons there are some packages that does not fall into the naming schema. Either because a go project got migrated to a different repo or because it make/made sense to combine some projects together and ship them in a single rpm. I don't mention Kubernetes, Docker and other popular projects as you already mention exception for them.
- working Go autoprovides. No forgotten provides anymore.
Sometimes it make sense to skip some provided package. E.g. some packages provide Windows specific code which has no use on Linux. So the claim is not completely valid.
- working Go autorequires. No forgotten requires anymore.
Depending on how you generate the list you can non-intentionally require more than is needed. Which causes installation of unneeded trees of rpms. E.g. for building purposes there is no need to install dependencies of tests So the claim is not completely valid.
(Valid for both requires/provides): Sometimes you only need to install/provide a subset of a Go project. There are not many projects that imports every package another project provides. So to save time packaging dependencies of unimported packages (and their following maintenance), it is beneficial to generate only partial list of required/provided packages.
Plus, in case CGO, there is no 1:1 mapping of C libraries to their corresponding rpms.
- strict automated directory ownership (used by autorequires and autoprovides).
- centralized computation of source URLs (via Forge-hosted projects packaging automation). No more packages lacking guidelines. No more broken guidelines no one notices.
In other words to use %gourl macro?
- easy switch between commits, tags and releases (via Forge-hosted projects packaging automation). No more packages stuck on commits when upstream starts releasing.
The issue is not about upstream releasing, it's about projects that actually import snapshots of other projects instead of particular releases. Given by the nature of the Go land.
- guidelines-compliant automated snapshot naming, including snapshot timestamps (via Forge-hosted projects packaging automation). No more packages stuck in 2014.
The issue is not that a package is stuck and not updated for more than 6 months. The issue is API backwards incompatibility that does not allow to easily update to a newer version. And a lack of man power to perform an update.
- guidelines-compliant bootstraping.
- systematic use of the Go switches defined by the Go maintainer. Easy to do changes followed by a mass rebuild.
- flexibility, do the right thing transparently by default, leave room for special cases and overrides.
- no bundling (a.k.a. vendoring) due to the pain of packaging one more Go dependency.
Can you more elaborate on that? Cause this is unavoidable no matter how great any Go guidelines get. Choice of bundeling is not about how many more Go dependencies you need to package. It's due to API backwards incompatibility reasons.
- centralized Go macros that can be audited and enhanced over time.
+1 as long as they are kept on a reasonable granularity level
- aggressive leverage of upstream unit tests to detect quickly broken code.
Will not work in general due to API backwards incompatibility issues. We need to move the tests out of the spec files if we want to seriously test Go projects. Saying that, we can run subset of tests to at least have some kind of smoke tests.
no reliance on external utilities to compute code requirements. No more dependencies that do not match the shipped Go code.
Please, rephrase or remove the point as for the CGO it is useful to have external utility that can list all imported C header files and try to find a set of packages that provide the header files (and corresponding libraries).
Please consult packaging draft for full information.
The proposal has been tested in Rawhide and EL7 over a set of ~ 140 Go packages. This set is a mix of current Fedora packages, bumped to a more recent version, rewrites of Fedora packages, and completely new packages.
I hope posting the second part of the automation will answer some questions people had on thehttps://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft
Regards,
I am surprised you are able to do that without any API issues. Have you tried to de-bundle and build Kubernetes, Docker, Etcd, Prometheus and other big projects from the same set of Go packages? At some point you have/had to switch to compat packages or bundling.
========================================= From the guidelines:
https://fedoraproject.org/wiki/More_Go_packaging#Limitations
Totally agree with the release discipline in many Go projects.
gives up on many conventions of current Fedora Go packaging, as they were an obstacle to the target packaging efficiency.
(Still part of the Limitations): Can you make a list of conventions you consider as obstacles? I would like to comment on each of them.
https://fedoraproject.org/wiki/More_Go_packaging#Excluding_testing_in_specif... %gochecks . transport/http transport/grpc transport option internal integration-tests/storage examples
The excluding mechanism is not intuitive, my first understand would be to test all the listed directories. Maybe use %gocheck -v the same way is grep does?
https://fedoraproject.org/wiki/More_Go_packaging#Handling_API_changes API changes may require the creation of Go code compatibility packages
I would not recommended doing that unless you have an army of packagers. It not just about creating one compat package. It is not surprising API incompatible change goes hand to hand with updating of dependencies. So instead of creating one compat package you will end up creating a tree of compat packages. I have already mentioned that at https://pagure.io/packaging-committee/issue/382#comment-147649. Plus, multiply the number of compat packages to take care with the active Fedora branches + epel7. On paper it looks like a very good solution but in reality it is not. As long as we don't know how much the Go projects suffer the API incompatibilities, I don't want to spawn thousand compat rpms and maintain them.
a packager, that identifies an API change in his Go package, MUST notify golang@fedoraproject.org at least a week before pushing his changes to any release (and those releases SHOULD include fedora-devel). This grace period can be waived by FPC for security reasons.
From the point of view of updating popular projects like Etcd, Kubernetes, Prometheus, etc. this is highly impractical. Usually, there are more dependencies to update. If I will have to wait one week to update etcd, than another week to push its update to stable branch, then I will not really be effective in updating.
a packager, that identifies an unannounced API change in another Go package, MUST notify golang@fedoraproject.org at once. those notices SHOULD be copied to the maintainers of packages affected by the change, when they are known.
This is going to be a regular thing unless a tooling detecting this changes is available.
a packager, that identifies that the code he packages, is broken by API changes in another project, SHOULD notify politely the upstream of the broken project of the API change, if he can not ascertain it is already aware of the change.
Some upstream Go projects are responsive, some are not. A lot of Go projects are forked, abandoned or inactive. Or the upstream is a single person that does not have time to deal with distribution specific guidelines. But I appreciate the effort to spread the good practices. However, sometimes the API change is intentional cause there is no other way.
=========================================
Usual workflow of updating Go packages:
1. choose a commit to update to 2. update a list of provided packages 3. update a list of build/run-time dependencies (if needed create spec files for new Go projects and open review requests) 4. update a list of tests (skip some tests that fail on specific architectures + open upstream issues, resp. PR with a fix) 5. optionally, update %install/%build section 6. perform scratch-build (may require to override some rpms) 7. perform push and raw build 8. create package update in Bodhi 9. push updates into stable if there is not enough karmas
## Some issues one can encounter with when performing a Go package update - update is backward incompatible (it may remove ability of some package Go projects to get built) - 5 (arbitrary number here) or more new Go projects need to be packaged into distribution
## What we need - automatic updates of Go packages (each time there is a new release or on demand to a specific commit) - spread maintenance among a group of packagers - run unit-tests and integration tests (if available) in CI/CD (e.g. after each update, on periodic basis) - automatic every routine job we do so we can concentrate on "real" and "interesting" problems in the Go land
The updated (and fairly improved/extended) guidelines cover points 2), 3), 4), 5) to some extend. Comparing the new way with the current way (listing only some +/- that popped in my head during interpretation): + provided packages, runtime dependencies are automatically generated + tests are automatically generated + spec files are much smaller (more transparent, a lot of complexity hidden in macros) + guidelines looks awesome, more descriptive in what needs to be done (up to some parts mentioning many times the current spec files sucks a lot)
- no build-time dependencies (we can not discover missing dependencies before installation) - always generating all provided packages that are available (I don't want to provide all packages always, e.g. need just a subset to build my project, the remaining provided packages need new Go projects which adds unnecessary complexity) - as jcajka pointed out, it is not easy to debug the lua based scripts - combination of bundled and un-bundled dependencies (sometimes I just need to use one or two bundled dependencies because without generating a tune of compat packages)
In all cases, the new approach is strongly focused on specfile simplification. The API backwards incompatibility problems are conveniently hidden in the creation of Go code compatibility packages. I have already mentions this approach in the https://pagure.io/packaging-committee/issue/382 and the reasons why it is not a way to go.
Even if we create a lot of compat packages we still need a mechanism that can tell us when a given compat package is no longer needed. I am working on a tooling (in my free time) that will provide that information (https://github.com/gofed/symbols-extractor, feel free to contribute to the project with ideas who we need to detect or extract from the code to make more spec file pieces generic and macro-replaceable).
At the end, even if we do "create as many compat Go packages as needed" or "let's use vendored packages because we can", we will still need to maintain entire ecosystem in some reasonable way.
From my point of view, the new Go packaging guidelines make it easier for fresh Go packager to get familiar with Go packaging practices and to quickly package his/her Go project. However, from the point of view of entire Go ecosystem (or any micro ecosystem = I have more projects building from the same dependencies), it is harder to tune individual packages for the mutual benefit.
=========================================
In overall, I don't like you degrading the current Go packaging guidelines (although it is still a draft). All the decisions made have historical reasons. By the time of applying the guidelines, there were only a few go packages and the severity and impact of the practices were not eminent nor obvious. Now we know it is not sufficient and there are things we need to improve. Indeed, 100% agree. So please, pay at least some respect to all who spent their time and dreamless nights coming with quick solutions. Cause a lot of us do it in our free-time so we are limited by time and capacity. What you are trying to solve now was known two years ago. We already knew about that. For that reason my and my colleagues started building tooling to detect at least some of the known problems like API backwards incompatibilities or automation around updating spec files.
I also suggest to remove every comparison with the Go guidelines draft and the current state of all Go spec files. Optionally, you can put all the mentions under "Best practices" or "Thing to avoid" section instead.
There are other parts of the new guidelines I could contradict. However, doing that we can end up in endless discussion, annoying each other and burning time that we can use to do important things instead.
So let's divide the workload. If you want to take care of the packaging guidelines, feel free to do it. Want to improve the spec file? I will be more than happy to witness that. Do you want to make the Go packaging in Fedora (and maybe other distributions) more user friendly? Man, that is awesome and noble. I have no intention to stop you from doing that. But please, keep in mind, you are not the only one involved in this effort. Some of us have families, some of us has other things/folks to take care of. We only do what we can in the time given :). I have/had my use case I am/was driven, you have yours. That's completely fine and appreciated. The more use cases we can fine, the more ideas and solutions we can combine. Let's not degrade results of our efforts, it leads nowhere. Let's combine the results, our experience. Let's open a google document (or different collaboration solution if you have one in mind) where we collect all our requirements. Let's ask other Go (or wannabe) packagers what are their requirement, what they see as drawbacks of the current and new guidelines, who is willing to help with the packaging and maintenance, let's ask more questions.
I am happy to take the tooling side. As I mentioned in the ticket (https://pagure.io/packaging-committee/issue/382), I shifted my focus on building tooling that do a lot of analysis and detection which are out of scope of any guidelines. Even if we agree on the final form of the guidelines, we still need tooling that will help us with the packaging. E.g. detection of API backwards incompatibilities, spec file generators (enhancing and improving over time), detection of go packages that are no longer needed, running CI/CD, etc.
Regards Jan
De: "Jan Chaloupka"
Hi Jan
Apologies for the delayed answer, I was beating the PR into shape to address review comments.
Let's keep moving forward, improving the infrastructure for other folks and pushing new ideas and improvements into practical solutions.
Let's move forward, I'm sick on sitting on a private spec stash and I see people trying to package stuff already in my list with the old guidelines.
On 12/17/2017 08:11 AM, nicolas.mailhot wrote:
It builds on the hard work of the Go SIG and reuses the rpm automation ofhttps://fedoraproject.org/wiki/PackagingDrafts/Go when it exists, and produces compatible packages.
Can you describe what you mean by "compatible packages"?
That mostly means the resulting packages can use "old-style" packages and "old-style" packages can use "new-style" packages. The effective result is similar enough there is no need for a rupture. Now, as I wrote before "old-style" packages will often be too old or not rigorous enough to be useful to "new-style" packages.
I mentioned a list of things that you did not answer fully. Most important to me:
- your macros do not generate build-time dependencies, which I see as
one of the drawbacks.
Generating BuildRequires dynamically needs changes in rpm tooling (mostly mock, I think). So it can not be done today if I'm not wrong. If you have a tool that outputs BuildRequires outside rpm, you can still use it. That's not an absolute showstopper as the go compiler is pretty explicit on what's missing, so BuildRequires are easily added during the first %build or %check run (though that's tedious and annoying).
Do you plan to ignore them?
Generating BuildRequires would be quite easy if rpm tooling provided an entry point at the end of %prep. So, that's a mid-term objective and requires sending an RFE mock-side first.
Requires generation is automatic and complete, so a Go -devel package will pull in all it needs without manual intervention (except for version constrains, that needs some maturing go dep side first)
- support for more subpackages. You describe how to specify which files
go to which packages (https://fedoraproject.org/wiki/More_Go_packaging#Separate_code_packages) but you don't say how list of provided packages and a list of dependencies are generated for the subpackages.
It is fully automated and transparent, rpm does not care why you split your Go code, it will compute provides and requires for every directory installed and owned under %{gopath}/src/%{goipath}, and attribute the result to the (sub)package owning the the directory. So just choosing what to install in each %goinstall invocation is sufficient to compute the corresponding requires/provides and attach them to the correct subpackage. If you don't want your package to pull a particular dep, just don't install the code that calls it.
- reproducible evaluation of macros. Either for running automatic
analysis over spec files or for debugging purposes. I would like the macros to be built in top of binaries I can run separately.
That would be quite difficult to do as most macros depend on rpm variables and other context defined during the build run, and can themselves alter rpm state dynamically via variable setting and other calls. rpm is not a dumb archive format it includes all kinds of package building facilities that the macros take advantage of, it ships with an embedded lua engine, and so on.
For "simple" macros that only read a specific variable set, and only perform "simple" actions (or set a specific set of variables) you could certainly rewrite them as separate scripts or binaries (for example the naming code is pretty trivial to rewrite in any language, and takes only one input and one output). But the more extensive the integration with rpm is, the more rpm glue you'd need to interface the result, and in some cases the glue is likely to be a significant part of the existing code. Also, you'd lose quite a lot of flexibility, and complexify your build root requirements.
I'm not saying it can not be done, or it is not worthwhile to do, but it's a lot of analysis and rewriting work and I'm not sure it's worth it (I'm certainly not interested to do it myself).
Note that autodeps are already effectively separate scripts, you just need to pass them a few arguments and pipe them Go code directory names to see them working. It may feel weird but that's how the autodep rpm API is defined. Other ecosystems packaged as rpms sometimes use separate binaries there, sometimes with light shell gluing. You have my blessing to rewrite them in Go if you want to.
- automated package naming derived from the native identifier (import path). No more packages names without any relation with current upstream naming.
Can you provide a list of package names that diverge from this convention?
For example, all the golang/x/foo stuff, which is a pity as it is used right and left.
For historical reasons there are some packages that does not fall into the naming schema. Either because a go project got migrated to a different repo
I understand, but the packages should be renamed, and not carry on the previous previous previous upstream name.
or because it make/made sense to combine some projects together and ship them in a single rpm.
Combining does not really work from a lifecycle POW, won't be easy to do with the new automation, or with native Go tooling. You really need to give up on it, sorry, it has never been a good idea for Go or for other languages, all the tooling expects *one* project not many, even when you can somewhat force it it never ends well.
- working Go autoprovides. No forgotten provides anymore.
Sometimes it make sense to skip some provided package. E.g. some packages provide Windows specific code
Then don't install the windows code and it won't pull corresponding deps.
- working Go autorequires. No forgotten requires anymore.
Depending on how you generate the list you can non-intentionally require more than is needed. Which causes installation of unneeded trees of rpms.
Again, the solution is to not install problem code, instead of shipping non-working code to others.
E.g. for building purposes there is no need to install dependencies of tests
I don't install test code so it does not pollute other projects with its dependencies
So the claim is not completely valid.
Yes it is. You're just too used to installing everything without filtering, and trying to workaround the resulting problems.
(Valid for both requires/provides): Sometimes you only need to install/provide a subset of a Go project.
If you only install part of a project, rpm will only generate requires/provides for the partial install. Note, however, than this is not the direction go dep is going towards. A lot of projects are going to need to structure their code better in a go dep world (for exactly the same reasons rpm requires better structuring: tools are binary, there is no convenient gray messy zone like when dealing with humans)
There are not many projects that imports every package another project provides. So to save time packaging dependencies of unimported packages (and their following maintenance),
Again, just don't install unneeded code, it won't pull any unneeded dep. Much simpler for everyone involved.
- strict automated directory ownership (used by autorequires and autoprovides).
- centralized computation of source URLs (via Forge-hosted projects packaging automation). No more packages lacking guidelines. No more broken guidelines no one notices.
In other words to use %gourl macro?
Actually, you need %gosource more. %gourl is a pretty side effect.
- easy switch between commits, tags and releases (via Forge-hosted projects packaging automation). No more packages stuck on commits when upstream starts releasing.
The issue is not about upstream releasing, it's about projects that actually import snapshots of other projects instead of particular releases. Given by the nature of the Go land
Try to convert one old-style Go spec from commit to release, or the reverse, you'll see it's not quite so simple there is a lot of tricky boilerplate to change. I've seen quite a few packages, that used commits that corresponded to releases, because redoing the spec as a release was too much hassle.
- guidelines-compliant automated snapshot naming, including snapshot timestamps (via Forge-hosted projects packaging automation). No more packages stuck in 2014.
The issue is not that a package is stuck and not updated for more than 6 months. The issue is API backwards incompatibility that does not allow to easily update to a newer version. And a lack of man power to perform an update.
You can hope for more manpower or you can make changes less time-consuming. Both result in more work done, one is less realistic than the other.
- no bundling (a.k.a. vendoring) due to the pain of packaging one more Go dependency.
Can you more elaborate on that? Cause this is unavoidable no matter how great any Go guidelines get. Choice of bundeling is not about how many more Go dependencies you need to package. It's due to API backwards incompatibility reasons.
You can address API incompatibilities with compat packages, instead of hiding them in bundled libs
- aggressive leverage of upstream unit tests to detect quickly broken code.
Will not work in general due to API backwards incompatibility issues.
I have some doubt that code that fails unit tests is actually something that can be used (and if it can, not be used why are you shipping it). Sure, the unit test may be wrong, but that's not the usual case. Better to detect, check if another package does not need updating, and report upstream otherwise. We had quite a lot of complex builds that were failing, and got unstuck when someone tried to run unit tests, and updated the components they were pointing to.
no reliance on external utilities to compute code requirements. No more dependencies that do not match the shipped Go code.
Please, rephrase or remove the point as for the CGO it is useful to have external utility that can list all imported C header files and try to find a set of packages that provide the header files (and corresponding libraries).
I'm not saying other things are not useful, but I *have* seen quite a few Fedora packages that were declaring garbage deps, because the external tool was run on another code state and the requirements were never corrected.
I am surprised you are able to do that without any API issues. Have you tried to de-bundle and build Kubernetes, Docker, Etcd, Prometheus and other big projects from the same set of Go packages? At some point you have/had to switch to compat packages or bundling.
Given the state of upstream projects debundling absolutely requires some compat packages. Which is why I documented how to create them easily, even if Jakub disapproves. I'll take compat packages over stealth bundling any day.
gives up on many conventions of current Fedora Go packaging, as they were an obstacle to the target packaging efficiency.
(Still part of the Limitations): Can you make a list of conventions you consider as obstacles? I would like to comment on each of them.
All the very complex layers upon layers of variables, ifdefing and so on that will completely drain your energy before doing any serious change and often differ in subtle ways from the conventions of the rest of the distro. I tried to keep the amount of boilerplate, variables and conditionals minimal, and to apply official Fedora patterns.
https://fedoraproject.org/wiki/More_Go_packaging#Excluding_testing_in_specif... %gochecks . transport/http transport/grpc transport option internal integration-tests/storage examples
The excluding mechanism is not intuitive,
opt-out is not intuitive but if you don't do opt-out people will select a few directories to test, give up as soon as the list is too long, and certainly never refresh on update opt-out is more robust in presence of humain limitations. And the default should be that everything is tested.
As soon as you integrate the opt-out nature of the macro, it's really easy to understand and use. And you certainly do not need to inflict yourself and others one or more exclusion switches that will need to be used every time.
https://fedoraproject.org/wiki/More_Go_packaging#Handling_API_changes API changes may require the creation of Go code compatibility packages
I would not recommended doing that unless you have an army of packagers.
As you wrote yourself, it's that or bundling. And bundling is the same misery without the help of rpm tooling and without putting the result in a package that can be reused in other projects that encountered the same API break.
a packager, that identifies an API change in his Go package, MUST notify golang@fedoraproject.org at least a week before pushing his changes to any release (and those releases SHOULD include fedora-devel). This grace period can be waived by FPC for security reasons.
From the point of view of updating popular projects like Etcd, Kubernetes, Prometheus, etc. this is highly impractical. Usually, there are more dependencies to update. If I will have to wait one week to update etcd, than another week to push its update to stable branch, then I will not really be effective in updating.
I'm not wed to this rule but removing it means instant breakage for all the stuff that relied on the old version. Which is not nice in a community project. Now if all Fedora Go packagers can live with it, I won't push it more.
a packager, that identifies an unannounced API change in another Go package, MUST notify golang@fedoraproject.org at once. those notices SHOULD be copied to the maintainers of packages affected by the change, when they are known.
This is going to be a regular thing unless a tooling detecting this changes is available.
Is it better to silently wait for the rush of mass rebuild breakage? Not That I would't appreciate some tooling (automated rebuild of dependant packages + notification of breakage would be nice)
a packager, that identifies that the code he packages, is broken by API changes in another project, SHOULD notify politely the upstream of the broken project of the API change, if he can not ascertain it is already aware of the change.
Some upstream Go projects are responsive, some are not. A lot of Go projects are forked, abandoned or inactive. Or the upstream is a single person that does not have time to deal with distribution specific guidelines. But I appreciate the effort to spread the good practices. However, sometimes the API change is intentional cause there is no other way.
Note that the guideline write to notify the project that was broken by the change, not the reverse. For non-dead projects they'll have to deal with the breakage sooner or later, better notify them as soon as possible so they can plan how to manage the break.
Usual workflow of updating Go packages:
Pretty much what I understood
## What we need
- automatic updates of Go packages (each time there is a new release or
on demand to a specific commit)
That should be quite easy to do with the new style of packaging as long as there is no huge project restructuring. The hard part is compat packages. BuildRequires could also be annoying as long as they are not fully automated (but, it is quite easy to scrap the build log for 'could not find foo package' messages and add the corresponding BR if already packaged)
- no build-time dependencies (we can not discover missing dependencies
before installation)
You can not discover deps before the package is fully built but once built it does declare all the stuff it needs and it is possible to check that against repo metadata.
- always generating all provided packages that are available (I don't
want to provide all packages always, e.g. need just a subset to build my project, the remaining provided packages need new Go projects which adds unnecessary complexity)
More accurately, you can not package Go code, and publish it to others, without the dep it needs. You may choose to package more deps, or to publish less code. You can rely on the code other packaged having the required dependencies (at least at the import level, version constrain level need progress upstream on the go dep front)
- as jcajka pointed out, it is not easy to debug the lua based scripts
Yes debugging lua is not the most pleasant thing, on the other hand the language is so limited, once code is written and worked once it is very unlikely to break in the future due to unrelated updates. It is much more robust towards changes than shell or other macro code.
- combination of bundled and un-bundled dependencies (sometimes I just
need to use one or two bundled dependencies because without generating a tune of compat packages)
That certainly works today (for whatever definition of "works" that encompasses bundling in general)
In all cases, the new approach is strongly focused on specfile simplification.
That was its primary objective.
Even if we create a lot of compat packages we still need a mechanism that can tell us when a given compat package is no longer needed.
Sure, however it is trivial to switch from compat to tip in buildrequires and try the result.
I am working on a tooling (in my free time) that will provide that information (https://github.com/gofed/symbols-extractor, feel free to contribute to the project with ideas who we need to detect or extract from the code to make more spec file pieces generic and macro-replaceable).
Thank you, added to the long list of things I need to look at.
At the end, even if we do "create as many compat Go packages as needed" or "let's use vendored packages because we can", we will still need to maintain entire ecosystem in some reasonable way.
They are two faces of the same problem as long as upostreams are not careful API-wise and we lack the manpower to fix every Go project every time its deps change.
From my point of view, the new Go packaging guidelines make it easier for fresh Go packager to get familiar with Go packaging practices and to quickly package his/her Go project.
I hope that will help new people to join, there are two few Fedora Go packagers for the amount of code that would be interesting to package and maintain.
However, from the point of view of entire Go ecosystem (or any micro ecosystem = I have more projects building from the same dependencies), it is harder to tune individual packages for the mutual benefit.
I'm not sure why you feel that way, the new guidelines make dependency breakage evident, they do not create it.
So please, pay at least some respect to all who spent their time and dreamless nights coming with quick solutions.
I don't think I've ever criticized anyone? I do criticize the current status but that has nothing to do with the people involved, given the amount of needed work and the ridiculously small Fedora Go packager pool.
I can't justify all the proposed the changes, or the huge amount of work they required, or the amount of work it will take to convert the existing package pool, by pretending the current status is good. I quite like Fedora but I have better ways to spend my work or free time if there is nothing to fix.
I also suggest to remove every comparison with the Go guidelines draft and the current state of all Go spec files.
Why exactly should FPC spend a lot of gruelling review time on the new guidelines if they improve nothing compared to the current state? Their time is not free either. The pros/cons section is just there to explain FPC why we are sucking their precious time, I don't see the point or use of keeping it post-approval. I must say, I don't feel FPC very willing to look at Go in Fedora, if they're not promised some changes.
I understand it is not pleasant to see some of those things written down, but not writing them will not cause them to cease to exist. Getting this stuff implemented, approved, and applied, *will* cause a lot of them to cease to exist. And convincing FPC there are significant gains to be had is a way to get the approval faster, and make all of this a thing of the past.
And of course, it's just a step, someone else will find other things to fix, and his solution will be better than this one, because it will build on this one, and that will be perfectly fine, and I won't insist this state can not be improved.
There are other parts of the new guidelines I could contradict. However, doing that we can end up in endless discussion, annoying each other and burning time that we can use to do important things instead.
Improving documentation is useful, but I'm quite sick of it right now, let's move to other things :)
But please, keep in mind, you are not the only one involved in this effort.
I certainly hope so, there would be little point spending all this time answering questions and writing wiki pages otherwise!
We only do what we can in the time given :).
That is so true… I don't think I can continue stretching it very long either.
Let's combine the results, our experience. Let's open a google document (or different collaboration solution if you have one in mind) where we collect all our requirements.
Let's ask other Go (or wannabe) packagers what are their requirement, what they see as drawbacks of the current and new guidelines, who is willing to help with the packaging and maintenance, let's ask more questions.
Well you won't entice me to write even more things near term, and given how few people replied to this thread, even though it was copied everywhere, I'm not sure a lot of people would be interested either. I'd say they probably need to see some visible advancement to commit their efforts there. I certainly need some visible advancement to continue.
I'd be delighted to be wrong of course.
I am happy to take the tooling side.
More tooling is always good!
However there will be a need for a general package baseline refresh or it will keep not interesting many people.
As I mentioned in the ticket (https://pagure.io/packaging-committee/issue/382), I shifted my focus on building tooling that do a lot of analysis and detection which are out of scope of any guidelines. Even if we agree on the final form of the guidelines, we still need tooling that will help us with the packaging. E.g. detection of API backwards incompatibilities, spec file generators (enhancing and improving over time), detection of go packages that are no longer needed, running CI/CD, etc.
The nice thing about fixing one problem, is that it makes fixing other problems possible.
Best regards
De: "nicolas mailhot" À: "Jan Chaloupka"
I mentioned a list of things that you did not answer fully. Most important to me:
- your macros do not generate build-time dependencies, which I see as
one of the drawbacks.
Generating BuildRequires dynamically needs changes in rpm tooling (mostly mock, I think). So it can not be done today if I'm not wrong. If you have a tool that outputs BuildRequires outside rpm, you can still use it. That's not an absolute showstopper as the go compiler is pretty explicit on what's missing, so BuildRequires are easily added during the first %build or %check run (though that's tedious and annoying).
Do you plan to ignore them?
Generating BuildRequires would be quite easy if rpm tooling provided an entry point at the end of %prep. So, that's a mid-term objective and requires sending an RFE mock-side first.
And that part is done:
https://github.com/rpm-software-management/mock/issues/160
Feel free to complete/correct/support the request so it has a change to be taken into account and we can start working on integrated Go BuildRequires
Regards