Hello all, I've noticed that the Go (golang) Packaging Guidelines Draft[0] document has been stagnant for a while now and I'm curious what the next steps should be? Does this need to go through FESCo?
Also, since Go is statically compiled by default is this something we need to get an exception from FESCo similar to OCaml[1]?
Another topic of note is bundled libraries. The upstream Go community seems pretty content with just bundling in their dependencies since it's all statically linked anyways (yes, you can dynamically link with gcc-go but I've yet to find a single project out in community space doing that). For some popular Go projects the dependency list is over 100 deps[2] and are managed with something similar to Godep[3], I'm not sure how realistic it is for packagers to be expected to maintain that many dependencies. This also begs the question that if we do require a packager to maintain them, what happens if another project requires a different version of that dep? (This is similar in nature to what I like to call "ruby bundler hell").
If there were to be some sort of approval for these bundled libraries, should there be a defined specification of which Go dependency managers are supported for sake of security response so that we can check for packages that need rebuilding when a vulnerability is found? What kind of changes would be necessary for build tooling there? (Maybe something in this area I'm not thinking of?)
I wanted to at least get this conversation going because it appears there's already a number of Go packages in Fedora at this time without any approved standard and as the language continues to gain popularity I can only assume that number will increase.
At the time of this writing, on my laptop running Rawhide: $ dnf search golang | wc -l 279
Thank you, -AdamM
[0] - https://fedoraproject.org/wiki/PackagingDrafts/Go [1] - https://fedoraproject.org/wiki/Packaging:Guidelines#Programs_which_don.27t_n... [2] - https://github.com/openshift/origin/blob/master/Godeps/Godeps.json [3] - https://github.com/tools/godep
On Wed, Apr 29, 2015 at 11:38:59AM -0500, Adam Miller wrote:
Hello all, I've noticed that the Go (golang) Packaging Guidelines Draft[0] document has been stagnant for a while now and I'm curious what the next steps should be? Does this need to go through FESCo?
It shouldn't need to go through FESCo. See https://fedorahosted.org/fpc/ticket/382 for current state.
Also, since Go is statically compiled by default is this somethingwe need to get an exception from FESCo similar to OCaml[1]?
That's covered in the draft.
If there were to be some sort of approval for these bundledlibraries, should there be a defined specification of which Go dependency managers are supported for sake of security response so that we can check for packages that need rebuilding when a vulnerability is found? What kind of changes would be necessary for build tooling there? (Maybe something in this area I'm not thinking of?)
Now, the bundling issue is an exciting kettle of worms — although the problem of tons of unpackaged deps is not really that different from Ruby or even Python or Perl. I think it's fair to say that the _idea_ of the current approach -- first package to require it generally needs to do the work of getting the dependencies in too -- is geared towards an eventual benefit to the _next_ packages, which will then find there deps already nicely available. (Pain now, but globally reduced pain later.)
On Wed, Apr 29, 2015 at 11:50 AM, Matthew Miller mattdm@fedoraproject.org wrote:
On Wed, Apr 29, 2015 at 11:38:59AM -0500, Adam Miller wrote:
Hello all, I've noticed that the Go (golang) Packaging Guidelines Draft[0] document has been stagnant for a while now and I'm curious what the next steps should be? Does this need to go through FESCo?
It shouldn't need to go through FESCo. See https://fedorahosted.org/fpc/ticket/382 for current state.
Also, since Go is statically compiled by default is this somethingwe need to get an exception from FESCo similar to OCaml[1]?
That's covered in the draft.
Yup, I totally missed that. Apologies.
If there were to be some sort of approval for these bundledlibraries, should there be a defined specification of which Go dependency managers are supported for sake of security response so that we can check for packages that need rebuilding when a vulnerability is found? What kind of changes would be necessary for build tooling there? (Maybe something in this area I'm not thinking of?)
Now, the bundling issue is an exciting kettle of worms — although the problem of tons of unpackaged deps is not really that different from Ruby or even Python or Perl. I think it's fair to say that the _idea_ of the current approach -- first package to require it generally needs to do the work of getting the dependencies in too -- is geared towards an eventual benefit to the _next_ packages, which will then find there deps already nicely available. (Pain now, but globally reduced pain later.)
That's fair I suppose, I just think that the scenario is slightly different because it's build time vs runtime deps for Go vs Python/Ruby/Perl. At runtime that giant dep list disappears. Maybe I'm over thinking this but it does seem different to me. However, I agree that if we can deal with some pain upfront and have less later then all the better. Just from a ground zero standpoint it seems like a lot of churn.
Thanks for the quick reply, I'll follow along in the fpc trac ticket from now on.
-AdamM
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader -- packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging
On Wed, Apr 29, 2015 at 11:59:43AM -0500, Adam Miller wrote:
That's fair I suppose, I just think that the scenario is slightly different because it's build time vs runtime deps for Go vs Python/Ruby/Perl. At runtime that giant dep list disappears. Maybe I'm over thinking this but it does seem different to me. However, I agree that if we can deal with some pain upfront and have less later then all the better. Just from a ground zero standpoint it seems like a lot of churn.
So, I'm not necessarily _proposing_ this, but something to think about: I have a suspicion that a long tail of dependencies actually aren't used by many multiple projects that get into Fedora, meaning that the upfront pain does not actually translate to this benefit in a large number of cases. (Of course, there are other benefits, including easier security tracking, but since this is a strawman let me just set that aside for a bit.)
What if, instead of requiring separate, unbundled packaging of dependencies the first time they're required, they instead get documented somewhere. The _second_ time something needs the same thing, the packagers for both the first and second package work together to split out that dep into its own package. This a) defers the "payment" until closer to the point of "delivery" and b) means that the two packagers share the work. It would also mean fewer unloved packages which were created solely to fill a dep need — and maybe even orphaned if that need changes.
"MM" == Matthew Miller mattdm@fedoraproject.org writes:
MM> What if, instead of requiring separate, unbundled packaging of MM> dependencies the first time they're required, they instead get MM> documented somewhere. The _second_ time something needs the same MM> thing, the packagers for both the first and second package work MM> together to split out that dep into its own package.
FPC has, on a couple of instances, approved this kind of thing, but. really it doesn't work well. Why? Because you can't depend on the maintainers of the other package to do much of anything. Now you have either the second package held up because someone else doesn't want to expend the effort, or you have the bits properly unbundled but the first package never updates (or gets pulled from the distro) because nobody wants to put in the work to unbundle.
MM> It would also mean fewer unloved packages which were created solely MM> to fill a dep need — and maybe even orphaned if that need changes.
I don't see how that's a benefit, really. A separate package with a separate upstream that can be tracked and updated almost automatically with all of our fancy upstream release monitoring tools, versus some unloved bundled of bits that is completely ignored because someone just imported some random version once and then never touched it? I know which one I'd pick.
In fact, I believe the existence of these new upstream release monitoring tools makes the argument for unbundling even stronger.
- J<
On 04/29/2015 08:10 PM, Matthew Miller wrote:
On Wed, Apr 29, 2015 at 11:59:43AM -0500, Adam Miller wrote:
That's fair I suppose, I just think that the scenario is slightly different because it's build time vs runtime deps for Go vs Python/Ruby/Perl. At runtime that giant dep list disappears. Maybe I'm over thinking this but it does seem different to me. However, I agree that if we can deal with some pain upfront and have less later then all the better. Just from a ground zero standpoint it seems like a lot of churn.
So, I'm not necessarily _proposing_ this, but something to think about: I have a suspicion that a long tail of dependencies actually aren't used by many multiple projects that get into Fedora, meaning that the upfront pain does not actually translate to this benefit in a large number of cases. (Of course, there are other benefits, including easier security tracking, but since this is a strawman let me just set that aside for a bit.)
I am hoping for better future of golang projects :) All packaged source codes are used only as build-time dependencies. Not for development (go get is used instead). Once the projects stabilize and stop breaking API and tools start using stable releases than the whole golang ecosystem will come to mind. For this sake all dependencies should be unbundled, packages updated and spec files polished. This way we can encounter with problems and solve them as time goes. At the same time tools for analysis and health care can be created. And so on.
What if, instead of requiring separate, unbundled packaging of dependencies the first time they're required, they instead get documented somewhere. The _second_ time something needs the same thing, the packagers for both the first and second package work together to split out that dep into its own package. This a) defers the "payment" until closer to the point of "delivery" and b) means that the two packagers share the work. It would also mean fewer unloved packages which were created solely to fill a dep need — and maybe even orphaned if that need changes.
Hi,
On 04/29/2015 06:38 PM, Adam Miller wrote:
Hello all, I've noticed that the Go (golang) Packaging Guidelines Draft[0] document has been stagnant for a while now and I'm curious what the next steps should be? Does this need to go through FESCo?
Yes. There is still a lot of problems to be solved. Look at comment #38. As long as there are no releases which does not break back compatibility, it is hard to make rules on packaging dependencies. We can, of course, describe a perfect situation and then write down a list of exceptions and how to deal with it.
Also, since Go is statically compiled by default is this somethingwe need to get an exception from FESCo similar to OCaml[1]?
Another topic of note is bundled libraries. The upstream Gocommunity seems pretty content with just bundling in their dependencies since it's all statically linked anyways (yes, you can dynamically link with gcc-go but I've yet to find a single project out in community space doing that).
IIRC it was etcd or kubernetes who started this because of rapidly changing imported packages and their API breaking.
For some popular Go projects the dependency list is over 100deps[2] and are managed with something similar to Godep[3], I'm not sure how realistic it is for packagers to be expected to maintain that many dependencies.
At the moment I am maintaining almost all of them. As almost all of the builds are source codes only their maintenance is generic. I have already written a tool for it [1]. It allows to update spec file, build package, update it, override it and so on for each branch simultaneously. Some analysis of projects (API diff, check of Godeps.json file, etc.) is available as well.
This also begs the question that if we do require a packager to maintain them, what happens if another project requires a different version of that dep? (This is similar in nature to what I like to call "ruby bundler hell").
Yeap. Already happening. Until now I have not had to package different versions/commits of the same project/source code. I have always managed to find a commit of source codes that is working for all projects. But it is a different commit than upstream/Godeps.json uses. Another way is to use bundled deps (ugly). Or package more versions of the source code (hope this will not happen).
If there were to be some sort of approval for these bundledlibraries, should there be a defined specification of which Go dependency managers are supported for sake of security response so that we can check for packages that need rebuilding when a vulnerability is found? What kind of changes would be necessary for build tooling there? (Maybe something in this area I'm not thinking of?)
At the moment you can generate a graph of dependencies between golang packages using 'gofed scandeps' (for all package or specific package). Then by look find which package depends on which and choose ones for rebuilding. As gofed's documentation is poor there is still a lot of undescribed features and commands. But it should give you basic notion what the tool does.
I wanted to at least get this conversation going because itappears there's already a number of Go packages in Fedora at this time without any approved standard and as the language continues to gain popularity I can only assume that number will increase.
Almost all spec files are generated using gofed. From time to time kubernetes introduces new packages or existing packages need to be updated. In this case I am polishing their spec files and keeping them in shape. But as I am dealing with new problems (like moving project from one repository to another, change of import path prefix, etc.) final form of a spec file is changing.
At the time of this writing, on my laptop running Rawhide: $ dnf search golang | wc -l 279
Some builds belongs to the same package so the number is not so high. But still :)
Thank you, -AdamM
[0] - https://fedoraproject.org/wiki/PackagingDrafts/Go [1] - https://fedoraproject.org/wiki/Packaging:Guidelines#Programs_which_don.27t_n... [2] - https://github.com/openshift/origin/blob/master/Godeps/Godeps.json [3] - https://github.com/tools/godep -- packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging
packaging@lists.fedoraproject.org
-
Adam Miller -
Jan Chaloupka -
Jason L Tibbitts III -
Matthew Miller