Hey guys,
So I know we have http://fedoraproject.org/wiki/Packaging/Guidelines#Duplication_of_system_lib... which I think is pretty good, easy to understand and fairly simple.
The problem I think is that some upstream's still want to ship internal, modified libraries.
Prime examples are mono packages, I can think of Banshee, Cowbell and f-spot from personal experience.
It's got to point where another upstream is giving me a little bit of hard time over it all, I even had a Debian developer agree with our guidelines (they have the same).
Upstream says "it's a guideline not a rule".
_MY_ question is, what can we (Fedora) do to make it clear that we have clear cut rules for why we don't want packages providing internal libraries?
- NJ
Hi,
_MY_ question is, what can we (Fedora) do to make it clear that we have clear cut rules for why we don't want packages providing internal libraries?
As long as it's explained to upstream why we have this guide, then we've really done all we can.
Take Mono.Cecil. It's not in GAC as the API is not stable, so MonoDevelop and a few other packages all come with their own sources of Mono.Cecil which may or may not be the same as the ones we built Mono.Cecil with.
As MD accesses it's own Mono.Cecil which it built (which is not in GAC), it is not likely to interfere with anything, however upstream will use that as a reason for including their own sources.
Realistically, what can we do about it - nothing as if upstream want to do as they do then all we can do is remove the package or start submitting piles of bugs when things go wrong to try and force them around to our way of doing things. Some may, some may continue regardless.
Bit of a bummer that...
TTFN
Paul
Nigel Jones wrote:
Hey guys,
So I know we have http://fedoraproject.org/wiki/Packaging/Guidelines#Duplication_of_system_lib... which I think is pretty good, easy to understand and fairly simple.
The problem I think is that some upstream's still want to ship internal, modified libraries.
Prime examples are mono packages, I can think of Banshee, Cowbell and f-spot from personal experience.
It's got to point where another upstream is giving me a little bit of hard time over it all, I even had a Debian developer agree with our guidelines (they have the same).
Upstream says "it's a guideline not a rule".
_MY_ question is, what can we (Fedora) do to make it clear that we have clear cut rules for why we don't want packages providing internal libraries?
1) We could rename the Guidelines to: Fedora Packaging Rules.
I know that many people like to say that the Guidelines really are Guidelines and that they can be broken for good reason but I'd much rather have them be strictly followed -- but make the process of granting provisional exceptions easier.
Having "MUST" rules that people can choose to disregard is silly. Remaining flexible about changing the rules when faced with areas that they don't live up to is the aspect that we want to retain, promote, and improve.
2) Better organization. Each rule should have a short form and link to a long form that explains the reasoning behind it. This is something we need to take up with the docs team so we can figure out a better way to organize the rules. CC'ing Karsten for this part.
3) I suspect that if your upstream is resorting to telling you, a Fedora Contributor, that the Fedora Guidelines are non-binding, then no matter what we do, he won't see reason. We can do a better job of explaining it and we can get other distributions (which generally understand the reasoning here) to help put pressure on but at the end of the day the upstream author will either see the light or they won't.
One thing we could try in the distro-pressure regard is taking this up with the distributions group. ##distro on irc.freenode.net, http://distribution.freedesktop.org, and http://lists.freedesktop.org/mailman/listinfo/distributions
Do people think that's a good way to go? I'll send an email off to them if so.
-Toshio
On Thu, 2008-09-18 at 09:05 -0700, Toshio Kuratomi wrote:
Nigel Jones wrote:
Hey guys,
So I know we have http://fedoraproject.org/wiki/Packaging/Guidelines#Duplication_of_system_lib... which I think is pretty good, easy to understand and fairly simple.
The problem I think is that some upstream's still want to ship internal, modified libraries.
Prime examples are mono packages, I can think of Banshee, Cowbell and f-spot from personal experience.
It's got to point where another upstream is giving me a little bit of hard time over it all, I even had a Debian developer agree with our guidelines (they have the same).
Upstream says "it's a guideline not a rule".
_MY_ question is, what can we (Fedora) do to make it clear that we have clear cut rules for why we don't want packages providing internal libraries?
- We could rename the Guidelines to: Fedora Packaging Rules.
I know that many people like to say that the Guidelines really are Guidelines and that they can be broken for good reason but I'd much rather have them be strictly followed -- but make the process of granting provisional exceptions easier.
Maybe have two sets? Fedora Packaging Principals - things that MUST NOT be broken (think - do no harm, ensure patches go upstream, don't replace system libraries in a package etc), and then the Fedora Packaging Guidelines (where there can be a little bit of change due to special circumstances (I can't remove rpath because the package just won't work without it).
Having "MUST" rules that people can choose to disregard is silly. Remaining flexible about changing the rules when faced with areas that they don't live up to is the aspect that we want to retain, promote, and improve.
- Better organization. Each rule should have a short form and link to
a long form that explains the reasoning behind it. This is something we need to take up with the docs team so we can figure out a better way to organize the rules. CC'ing Karsten for this part.
Agreed
- I suspect that if your upstream is resorting to telling you, a Fedora
Contributor, that the Fedora Guidelines are non-binding, then no matter what we do, he won't see reason. We can do a better job of explaining it and we can get other distributions (which generally understand the reasoning here) to help put pressure on but at the end of the day the upstream author will either see the light or they won't.
One thing we could try in the distro-pressure regard is taking this up with the distributions group. ##distro on irc.freenode.net, http://distribution.freedesktop.org, and http://lists.freedesktop.org/mailman/listinfo/distributions
Do people think that's a good way to go? I'll send an email off to them if so.
I didn't know about the list, I think it'd be a good way to go, I know Debian seem to agree with our policy, it'd be nice to create a united front (in a way) to say to projects that it's unacceptable.
Really it all comes down to this:
If upstreams have a really good reason to change another upstreams code, why haven't they submit it to their upstream?
It gets even more silly when upstreams act as upstream for other minor libraries so that you get 5 or so copies floating about on your system because everyone has had to fork it, because they've explicitly said to maintainers to not package separately/as a subpackage.
- NJ
Nigel Jones wrote:
On Thu, 2008-09-18 at 09:05 -0700, Toshio Kuratomi wrote:
Nigel Jones wrote:
Hey guys,
So I know we have http://fedoraproject.org/wiki/Packaging/Guidelines#Duplication_of_system_lib... which I think is pretty good, easy to understand and fairly simple.
The problem I think is that some upstream's still want to ship internal, modified libraries.
Prime examples are mono packages, I can think of Banshee, Cowbell and f-spot from personal experience.
It's got to point where another upstream is giving me a little bit of hard time over it all, I even had a Debian developer agree with our guidelines (they have the same).
Upstream says "it's a guideline not a rule".
_MY_ question is, what can we (Fedora) do to make it clear that we have clear cut rules for why we don't want packages providing internal libraries?
- We could rename the Guidelines to: Fedora Packaging Rules.
I know that many people like to say that the Guidelines really are Guidelines and that they can be broken for good reason but I'd much rather have them be strictly followed -- but make the process of granting provisional exceptions easier.
Maybe have two sets? Fedora Packaging Principals - things that MUST NOT be broken (think - do no harm, ensure patches go upstream, don't replace system libraries in a package etc), and then the Fedora Packaging Guidelines (where there can be a little bit of change due to special circumstances (I can't remove rpath because the package just won't work without it).
Having "MUST" rules that people can choose to disregard is silly. Remaining flexible about changing the rules when faced with areas that they don't live up to is the aspect that we want to retain, promote, and improve.
- Better organization. Each rule should have a short form and link to
a long form that explains the reasoning behind it. This is something we need to take up with the docs team so we can figure out a better way to organize the rules. CC'ing Karsten for this part.
Agreed
- I suspect that if your upstream is resorting to telling you, a Fedora
Contributor, that the Fedora Guidelines are non-binding, then no matter what we do, he won't see reason. We can do a better job of explaining it and we can get other distributions (which generally understand the reasoning here) to help put pressure on but at the end of the day the upstream author will either see the light or they won't.
One thing we could try in the distro-pressure regard is taking this up with the distributions group. ##distro on irc.freenode.net, http://distribution.freedesktop.org, and http://lists.freedesktop.org/mailman/listinfo/distributions
Do people think that's a good way to go? I'll send an email off to them if so.
I didn't know about the list, I think it'd be a good way to go, I know Debian seem to agree with our policy, it'd be nice to create a united front (in a way) to say to projects that it's unacceptable.
Message sent: http://lists.freedesktop.org/archives/distributions/2008-September/000226.ht...
Anyone interested, feel free to contribute to the thread that will likely result. if this is seen as a good thing, we'll probably want to start documenting our justifications for doing this on the distributions.freedesktop.org wiki.
Really it all comes down to this:
If upstreams have a really good reason to change another upstreams code, why haven't they submit it to their upstream?
It gets even more silly when upstreams act as upstream for other minor libraries so that you get 5 or so copies floating about on your system because everyone has had to fork it, because they've explicitly said to maintainers to not package separately/as a subpackage.
-Toshio
On Fri, Sep 19, 2008 at 01:41:50AM +1200, Nigel Jones wrote:
So I know we have http://fedoraproject.org/wiki/Packaging/Guidelines#Duplication_of_system_lib... which I think is pretty good, easy to understand and fairly simple.
The problem I think is that some upstream's still want to ship internal, modified libraries.
Upstream says "it's a guideline not a rule".
_MY_ question is, what can we (Fedora) do to make it clear that we have clear cut rules for why we don't want packages providing internal libraries?
I'd ask the question differently: If upstream saw itself forced to duplicate/fork a lib how can you help making the forked bits back into the original upstream.
I'm not targeting convenience bundling of libs as some projects do/did (these should really be replaced by system libs!), but rather patched up copies of libs that either use specific patches suitable only for that project or the lib upstream generaly is accepting patches at a slow or non-existant rate.
Just to quote one such example: ffmpeg is a fast moving target, and any project depending on the lib API is cutting a checkout, patching it a up and using it for its own purposes. Replacing these internal ffmpegs with a system ffmpeg is a nightmare or even impossible w/o rewriting the app interface to it. Given that ffmpeg and friends fall under the patent forbidden class we don't see that directly in Fedora, but this issue is still out there.
Other examples that do live in Fedora are minilzo that was never designed to be a shared lib and the consumer apps never designed their build system for a shared minilzo resulting in packages being broken/stalled for months and years before entering Fedora (like libvcnserver and x11vnc).
https://bugzilla.redhat.com/show_bug.cgi?id=439772#c21 https://bugzilla.redhat.com/show_bug.cgi?id=439979#c24
I'd recommend to soften this guideline to something as "the packager should try hard to use system libs, and try to communicate the benefits to upstream, but if there are reasons not to use system libs, then he should document this in the specfile".
On Fri, 2008-09-19 at 10:34 +0300, Axel Thimm wrote:
On Fri, Sep 19, 2008 at 01:41:50AM +1200, Nigel Jones wrote:
So I know we have http://fedoraproject.org/wiki/Packaging/Guidelines#Duplication_of_system_lib... which I think is pretty good, easy to understand and fairly simple.
The problem I think is that some upstream's still want to ship internal, modified libraries.
Upstream says "it's a guideline not a rule".
_MY_ question is, what can we (Fedora) do to make it clear that we have clear cut rules for why we don't want packages providing internal libraries?
I'd ask the question differently: If upstream saw itself forced to duplicate/fork a lib how can you help making the forked bits back into the original upstream.
Then let me answer with my "developer's hat" on:
The reason why such "forks" exist, often is disagreement between "upstream" and "fork" devs, because they disagree on APIs/usage and the like.
Upstreams often accuse "fork devs" to be abusing their libraries/APIs (e.g. using undocumented internals). Conversely, "fork devs" often accuse "upstreams" not to listen to "users' demands".
I.e. in many cases such conflicts will hardly be resolvable.
Just to quote one such example: ffmpeg is a fast moving target, and any project depending on the lib API is cutting a checkout, patching it a up and using it for its own purposes. Replacing these internal ffmpegs with a system ffmpeg is a nightmare or even impossible w/o rewriting the app interface to it. Given that ffmpeg and friends fall under the patent forbidden class we don't see that directly in Fedora, but this issue is still out there.
Well, ffmpeg is a special case wrt. many issues. If they were doing a proper job, they would release properly versioned packages with properly versioned APIs, which could be installed in parallel.
I'd recommend to soften this guideline to something as "the packager should try hard to use system libs, and try to communicate the benefits to upstream, but if there are reasons not to use system libs, then he should document this in the specfile".
I am not sure this is a good idea. I'd rather be stricter on this, because this would force devs to think about what they are doing and packagers think about the quality of what they are packaging.
The unpleasant truth is: If a package bundles "modified libs/apps" from elsewhere, which can't be installed in parallel to the original libs/apps, this package's dev's are doing something wrong.
Ralf
On Friday, 19 September 2008 at 09:59, Ralf Corsepius wrote:
On Fri, 2008-09-19 at 10:34 +0300, Axel Thimm wrote:
[...]
Just to quote one such example: ffmpeg is a fast moving target, and any project depending on the lib API is cutting a checkout, patching it a up and using it for its own purposes. Replacing these internal ffmpegs with a system ffmpeg is a nightmare or even impossible w/o rewriting the app interface to it. Given that ffmpeg and friends fall under the patent forbidden class we don't see that directly in Fedora, but this issue is still out there.
Well, ffmpeg is a special case wrt. many issues. If they were doing a proper job, they would release properly versioned packages with properly versioned APIs, which could be installed in parallel.
They(we) don't have the manpower to do proper releases, but we do maintain properly versioned API/ABI. The position of a release manager for FFmpeg has been open for months. Many people complained, but nobody is willing to do the legwork.
Regards, R.
On Fri, Sep 19, 2008 at 02:59:22PM +0200, Dominik 'Rathann' Mierzejewski wrote:
On Friday, 19 September 2008 at 09:59, Ralf Corsepius wrote:
On Fri, 2008-09-19 at 10:34 +0300, Axel Thimm wrote:
[...]
Just to quote one such example: ffmpeg is a fast moving target, and any project depending on the lib API is cutting a checkout, patching it a up and using it for its own purposes. Replacing these internal ffmpegs with a system ffmpeg is a nightmare or even impossible w/o rewriting the app interface to it. Given that ffmpeg and friends fall under the patent forbidden class we don't see that directly in Fedora, but this issue is still out there.
Well, ffmpeg is a special case wrt. many issues. If they were doing a proper job, they would release properly versioned packages with properly versioned APIs, which could be installed in parallel.
They(we) don't have the manpower to do proper releases, but we do maintain properly versioned API/ABI. The position of a release manager for FFmpeg has been open for months. Many people complained, but nobody is willing to do the legwork.
Actually I did, you can read it up at the archives, but the ffmpeg devs didn't think this would be a good idea. The only positive comments were from other distribution packagers that would have even joined the release team taskforce, but w/o the developers' blessing (and at the very least a "do it now" assignment) there wasn't much we could do.
On Friday, 19 September 2008 at 19:01, Axel Thimm wrote:
On Fri, Sep 19, 2008 at 02:59:22PM +0200, Dominik 'Rathann' Mierzejewski wrote:
On Friday, 19 September 2008 at 09:59, Ralf Corsepius wrote:
On Fri, 2008-09-19 at 10:34 +0300, Axel Thimm wrote:
[...]
Just to quote one such example: ffmpeg is a fast moving target, and any project depending on the lib API is cutting a checkout, patching it a up and using it for its own purposes. Replacing these internal ffmpegs with a system ffmpeg is a nightmare or even impossible w/o rewriting the app interface to it. Given that ffmpeg and friends fall under the patent forbidden class we don't see that directly in Fedora, but this issue is still out there.
Well, ffmpeg is a special case wrt. many issues. If they were doing a proper job, they would release properly versioned packages with properly versioned APIs, which could be installed in parallel.
They(we) don't have the manpower to do proper releases, but we do maintain properly versioned API/ABI. The position of a release manager for FFmpeg has been open for months. Many people complained, but nobody is willing to do the legwork.
Actually I did, you can read it up at the archives, but the ffmpeg devs didn't think this would be a good idea. The only positive comments were from other distribution packagers that would have even joined the release team taskforce, but w/o the developers' blessing (and at the very least a "do it now" assignment) there wasn't much we could do.
I've just re-read that thread and AFAICT nobody said it wasn't a good idea. At least one developer offered to help. You even offered to prepare a release candidate but never did.
Regards, R.
On Fri, Sep 19, 2008 at 08:09:15PM +0200, Dominik 'Rathann' Mierzejewski wrote:
On Friday, 19 September 2008 at 19:01, Axel Thimm wrote:
On Fri, Sep 19, 2008 at 02:59:22PM +0200, Dominik 'Rathann' Mierzejewski wrote:
On Friday, 19 September 2008 at 09:59, Ralf Corsepius wrote:
On Fri, 2008-09-19 at 10:34 +0300, Axel Thimm wrote:
[...]
Just to quote one such example: ffmpeg is a fast moving target, and any project depending on the lib API is cutting a checkout, patching it a up and using it for its own purposes. Replacing these internal ffmpegs with a system ffmpeg is a nightmare or even impossible w/o rewriting the app interface to it. Given that ffmpeg and friends fall under the patent forbidden class we don't see that directly in Fedora, but this issue is still out there.
Well, ffmpeg is a special case wrt. many issues. If they were doing a proper job, they would release properly versioned packages with properly versioned APIs, which could be installed in parallel.
They(we) don't have the manpower to do proper releases, but we do maintain properly versioned API/ABI. The position of a release manager for FFmpeg has been open for months. Many people complained, but nobody is willing to do the legwork.
Actually I did, you can read it up at the archives, but the ffmpeg devs didn't think this would be a good idea. The only positive comments were from other distribution packagers that would have even joined the release team taskforce, but w/o the developers' blessing (and at the very least a "do it now" assignment) there wasn't much we could do.
I've just re-read that thread and AFAICT nobody said it wasn't a good idea. At least one developer offered to help. You even offered to prepare a release candidate but never did.
Please check the "ranks" of the developers, some of them were even banned from ffmpeg, and how on earth could I prepare a release if the developers decide not to allow for scm access?
No, the idea was silently dropped even though there were many downstreams ready to pick it up (not only me).
On Saturday, 20 September 2008 at 11:34, Axel Thimm wrote:
On Fri, Sep 19, 2008 at 08:09:15PM +0200, Dominik 'Rathann' Mierzejewski wrote:
On Friday, 19 September 2008 at 19:01, Axel Thimm wrote:
On Fri, Sep 19, 2008 at 02:59:22PM +0200, Dominik 'Rathann' Mierzejewski wrote:
On Friday, 19 September 2008 at 09:59, Ralf Corsepius wrote:
On Fri, 2008-09-19 at 10:34 +0300, Axel Thimm wrote:
[...]
Just to quote one such example: ffmpeg is a fast moving target, and any project depending on the lib API is cutting a checkout, patching it a up and using it for its own purposes. Replacing these internal ffmpegs with a system ffmpeg is a nightmare or even impossible w/o rewriting the app interface to it. Given that ffmpeg and friends fall under the patent forbidden class we don't see that directly in Fedora, but this issue is still out there.
Well, ffmpeg is a special case wrt. many issues. If they were doing a proper job, they would release properly versioned packages with properly versioned APIs, which could be installed in parallel.
They(we) don't have the manpower to do proper releases, but we do maintain properly versioned API/ABI. The position of a release manager for FFmpeg has been open for months. Many people complained, but nobody is willing to do the legwork.
Actually I did, you can read it up at the archives, but the ffmpeg devs didn't think this would be a good idea. The only positive comments were from other distribution packagers that would have even joined the release team taskforce, but w/o the developers' blessing (and at the very least a "do it now" assignment) there wasn't much we could do.
I've just re-read that thread and AFAICT nobody said it wasn't a good idea. At least one developer offered to help. You even offered to prepare a release candidate but never did.
Please check the "ranks" of the developers, some of them were even banned from ffmpeg,
I have no idea what you're talking about. Who was banned, exactly?
and how on earth could I prepare a release if the developers decide not to allow for scm access?
I never saw you asking for that.
Regards, R.
PS. Feel free to continue off-list, this is getting off-topic.
On Fri, Sep 19, 2008 at 09:59:42AM +0200, Ralf Corsepius wrote:
I'd ask the question differently: If upstream saw itself forced to duplicate/fork a lib how can you help making the forked bits back into the original upstream.
Then let me answer with my "developer's hat" on:
The reason why such "forks" exist, often is disagreement between "upstream" and "fork" devs, because they disagree on APIs/usage and the like.
Upstreams often accuse "fork devs" to be abusing their libraries/APIs (e.g. using undocumented internals). Conversely, "fork devs" often accuse "upstreams" not to listen to "users' demands".
I.e. in many cases such conflicts will hardly be resolvable.
Yes, and sometimes as in the case of ffmpeg and minilzo upstream even wants you to take a snapshot, e.g. the "forking" is in their consent.
Just to quote one such example: ffmpeg is a fast moving target, and any project depending on the lib API is cutting a checkout, patching it a up and using it for its own purposes. Replacing these internal ffmpegs with a system ffmpeg is a nightmare or even impossible w/o rewriting the app interface to it. Given that ffmpeg and friends fall under the patent forbidden class we don't see that directly in Fedora, but this issue is still out there.
Well, ffmpeg is a special case wrt. many issues. If they were doing a proper job, they would release properly versioned packages with properly versioned APIs, which could be installed in parallel.
Even if so, would Fedora package 20 different versions of ffmpeg to satisfy the 20 different consumers? There wouldn't be any benefits to an internal lib either: If there is a security flaw fixed in ffmpeg 1004.0.1 the versions 900.x.y to 1003.x.y would be just as insecure as external packages.
I'd recommend to soften this guideline to something as "the packager should try hard to use system libs, and try to communicate the benefits to upstream, but if there are reasons not to use system libs, then he should document this in the specfile".
I am not sure this is a good idea. I'd rather be stricter on this, because this would force devs to think about what they are doing and packagers think about the quality of what they are packaging.
The unpleasant truth is: If a package bundles "modified libs/apps" from elsewhere, which can't be installed in parallel to the original libs/apps, this package's dev's are doing something wrong.
The truth is that sometimes upstream makes some decisions, we can try to forward our position, but upstream may ignore us or maybe will have a good reason to counter our position (for ffmpeg and minilzo I believe upstream is correct and we should be the ones revising the guidelines).
That's why I suggest that the packager tries hard to do it in the typical shared way, but if there are sane reasons to use internal libs to not let the package dry out forever in some review queue like for x11vnc.
Axel Thimm wrote:
On Fri, Sep 19, 2008 at 09:59:42AM +0200, Ralf Corsepius wrote:
I'd ask the question differently: If upstream saw itself forced to duplicate/fork a lib how can you help making the forked bits back into the original upstream.
Then let me answer with my "developer's hat" on:
The reason why such "forks" exist, often is disagreement between "upstream" and "fork" devs, because they disagree on APIs/usage and the like.
Upstreams often accuse "fork devs" to be abusing their libraries/APIs (e.g. using undocumented internals). Conversely, "fork devs" often accuse "upstreams" not to listen to "users' demands".
I.e. in many cases such conflicts will hardly be resolvable.
Yes, and sometimes as in the case of ffmpeg and minilzo upstream even wants you to take a snapshot, e.g. the "forking" is in their consent.
Upstream is wrong. The consequences of their actions don't come back to haunt them, though, they come back to haunt us. If there's a security hole in their library snapshot, their answer can be "we fixed that two months ago. Why didn't you update?" Our answer would have to be: "We have an unfixed vulnerability in gnome-foo-player, toms-foo-player, and foo-plus-player. We are not able to fix these until we perform a port to an incompatible ffmpeg snapshot for the maintainers of the affected software."
Just to quote one such example: ffmpeg is a fast moving target, and any project depending on the lib API is cutting a checkout, patching it a up and using it for its own purposes. Replacing these internal ffmpegs with a system ffmpeg is a nightmare or even impossible w/o rewriting the app interface to it. Given that ffmpeg and friends fall under the patent forbidden class we don't see that directly in Fedora, but this issue is still out there.
Well, ffmpeg is a special case wrt. many issues. If they were doing a proper job, they would release properly versioned packages with properly versioned APIs, which could be installed in parallel.
Even if so, would Fedora package 20 different versions of ffmpeg to satisfy the 20 different consumers? There wouldn't be any benefits to an internal lib either: If there is a security flaw fixed in ffmpeg 1004.0.1 the versions 900.x.y to 1003.x.y would be just as insecure as external packages.
I think there is benefit: 1) If the library is statically linked in the application it's harder to find in a quick audit.
2) If the flaw affects a range of library versions (for instance, 899.x.y is unaffected), having them be packaged separately makes finding the affected versions and fixing them easier than finding out which versions of the private library each app has.
Also, one of the goals of a distribution is to make programs and libraries work together. So the approach we'd likely want to take is choosing a few versions of the library that we could support (probably in conjunction with other distros) and then porting the apps to one of those library versions and getting upstreams to update their private libs to those versions since we were willing to do the port for them.
I'd recommend to soften this guideline to something as "the packager should try hard to use system libs, and try to communicate the benefits to upstream, but if there are reasons not to use system libs, then he should document this in the specfile".
I am not sure this is a good idea. I'd rather be stricter on this, because this would force devs to think about what they are doing and packagers think about the quality of what they are packaging.
The unpleasant truth is: If a package bundles "modified libs/apps" from elsewhere, which can't be installed in parallel to the original libs/apps, this package's dev's are doing something wrong.
The truth is that sometimes upstream makes some decisions, we can try to forward our position, but upstream may ignore us or maybe will have a good reason to counter our position (for ffmpeg and minilzo I believe upstream is correct and we should be the ones revising the guidelines).
That's why I suggest that the packager tries hard to do it in the typical shared way, but if there are sane reasons to use internal libs to not let the package dry out forever in some review queue like for x11vnc.
I am against this approach. But perhaps you'll have a useful counter to the points I raised.
-Toshio
On Fri, Sep 19, 2008 at 11:49:15AM -0700, Toshio Kuratomi wrote:
Axel Thimm wrote:
On Fri, Sep 19, 2008 at 09:59:42AM +0200, Ralf Corsepius wrote:
I'd ask the question differently: If upstream saw itself forced to duplicate/fork a lib how can you help making the forked bits back into the original upstream.
Then let me answer with my "developer's hat" on:
The reason why such "forks" exist, often is disagreement between "upstream" and "fork" devs, because they disagree on APIs/usage and the like.
Upstreams often accuse "fork devs" to be abusing their libraries/APIs (e.g. using undocumented internals). Conversely, "fork devs" often accuse "upstreams" not to listen to "users' demands".
I.e. in many cases such conflicts will hardly be resolvable.
Yes, and sometimes as in the case of ffmpeg and minilzo upstream even wants you to take a snapshot, e.g. the "forking" is in their consent.
Upstream is wrong. The consequences of their actions don't come back to haunt them, though, they come back to haunt us. If there's a security hole in their library snapshot, their answer can be "we fixed that two months ago. Why didn't you update?" Our answer would have to be: "We have an unfixed vulnerability in gnome-foo-player, toms-foo-player, and foo-plus-player. We are not able to fix these until we perform a port to an incompatible ffmpeg snapshot for the maintainers of the affected software."
Or backport to the versions needed. That's what RHEL does. And you now have the choice to
o either not package it o package it as a shared external lib in its own package and have the same problem.
Using "system libraries" just means blessing one version of the upstream project. Which if you want to follow security issues would need to be the latest and greatest. Which means that projects that have cut an older version of it will not run/build anymore.
So at the end we just don't ship them, as we do for x11vnc. Is that the preferred outcome?
Just to quote one such example: ffmpeg is a fast moving target, and any project depending on the lib API is cutting a checkout, patching it a up and using it for its own purposes. Replacing these internal ffmpegs with a system ffmpeg is a nightmare or even impossible w/o rewriting the app interface to it. Given that ffmpeg and friends fall under the patent forbidden class we don't see that directly in Fedora, but this issue is still out there.
Well, ffmpeg is a special case wrt. many issues. If they were doing a proper job, they would release properly versioned packages with properly versioned APIs, which could be installed in parallel.
Even if so, would Fedora package 20 different versions of ffmpeg to satisfy the 20 different consumers? There wouldn't be any benefits to an internal lib either: If there is a security flaw fixed in ffmpeg 1004.0.1 the versions 900.x.y to 1003.x.y would be just as insecure as external packages.
I think there is benefit:
- If the library is statically linked in the application it's harder to
find in a quick audit.
- If the flaw affects a range of library versions (for instance,
899.x.y is unaffected), having them be packaged separately makes finding the affected versions and fixing them easier than finding out which versions of the private library each app has.
Also, one of the goals of a distribution is to make programs and libraries work together. So the approach we'd likely want to take is choosing a few versions of the library that we could support (probably in conjunction with other distros) and then porting the apps to one of those library versions and getting upstreams to update their private libs to those versions since we were willing to do the port for them.
Well, be my guest. Many have attempted to do so in some cases, but if upstream is *fast*, you will never catch up, and different distributions will chose different snapshots depending on the consumer apps they target, so coordinating will be more than difficult.
But in principle I agree: If we could afford the manpower to keep all comsumers compatible to the latest stable release of a library that would be the ideal world. But we don't have that manpower, which is not just packaging, but true upstream work.
A reality check shows that we even lack the definition of "latest stable upstream release" in some cases like ffmpeg.
I'd recommend to soften this guideline to something as "the packager should try hard to use system libs, and try to communicate the benefits to upstream, but if there are reasons not to use system libs, then he should document this in the specfile".
I am not sure this is a good idea. I'd rather be stricter on this, because this would force devs to think about what they are doing and packagers think about the quality of what they are packaging.
The unpleasant truth is: If a package bundles "modified libs/apps" from elsewhere, which can't be installed in parallel to the original libs/apps, this package's dev's are doing something wrong.
The truth is that sometimes upstream makes some decisions, we can try to forward our position, but upstream may ignore us or maybe will have a good reason to counter our position (for ffmpeg and minilzo I believe upstream is correct and we should be the ones revising the guidelines).
That's why I suggest that the packager tries hard to do it in the typical shared way, but if there are sane reasons to use internal libs to not let the package dry out forever in some review queue like for x11vnc.
I am against this approach. But perhaps you'll have a useful counter to the points I raised.
The points you raised are nice in an ideal over manpowered world, but in reality we see packages stalling in review queues forever instead.
Anyway, these were my 0.02. I'm not really seeing anyone from the FPC favouring this, even less willing to pick this up and driving this through, so I'll rather stick to agreeing to disagree. :)
Axel Thimm wrote:
On Fri, Sep 19, 2008 at 11:49:15AM -0700, Toshio Kuratomi wrote:
Axel Thimm wrote:
On Fri, Sep 19, 2008 at 09:59:42AM +0200, Ralf Corsepius wrote:
I'd ask the question differently: If upstream saw itself forced to duplicate/fork a lib how can you help making the forked bits back into the original upstream.
Then let me answer with my "developer's hat" on:
The reason why such "forks" exist, often is disagreement between "upstream" and "fork" devs, because they disagree on APIs/usage and the like.
Upstreams often accuse "fork devs" to be abusing their libraries/APIs (e.g. using undocumented internals). Conversely, "fork devs" often accuse "upstreams" not to listen to "users' demands".
I.e. in many cases such conflicts will hardly be resolvable.
Yes, and sometimes as in the case of ffmpeg and minilzo upstream even wants you to take a snapshot, e.g. the "forking" is in their consent.
Upstream is wrong. The consequences of their actions don't come back to haunt them, though, they come back to haunt us. If there's a security hole in their library snapshot, their answer can be "we fixed that two months ago. Why didn't you update?" Our answer would have to be: "We have an unfixed vulnerability in gnome-foo-player, toms-foo-player, and foo-plus-player. We are not able to fix these until we perform a port to an incompatible ffmpeg snapshot for the maintainers of the affected software."
Or backport to the versions needed. That's what RHEL does. And you now have the choice to
o either not package it
This one's not true. We've already released it. If we remove gnome-foo-player, toms-foo-player, and foo-plus-player from the repository, all we're doing is keeping future people from getting the software. We aren't helping the people who have already received the software.
o package it as a shared external lib in its own package and have the same problem.
Using "system libraries" just means blessing one version of the upstream project. Which if you want to follow security issues would need to be the latest and greatest. Which means that projects that have cut an older version of it will not run/build anymore.
You didn't address the benefits that I outlined below:
- If the library is statically linked in the application it's harder to
find in a quick audit.
- If the flaw affects a range of library versions (for instance,
899.x.y is unaffected), having them be packaged separately makes finding the affected versions and fixing them easier than finding out which versions of the private library each app has.
So at the end we just don't ship them, as we do for x11vnc. Is that the preferred outcome?
It might be. If we don't have the manpower to keep things secure then we have no business inflicting them upon our users.
Just to quote one such example: ffmpeg is a fast moving target, and any project depending on the lib API is cutting a checkout, patching it a up and using it for its own purposes. Replacing these internal ffmpegs with a system ffmpeg is a nightmare or even impossible w/o rewriting the app interface to it. Given that ffmpeg and friends fall under the patent forbidden class we don't see that directly in Fedora, but this issue is still out there.
Well, ffmpeg is a special case wrt. many issues. If they were doing a proper job, they would release properly versioned packages with properly versioned APIs, which could be installed in parallel.
Even if so, would Fedora package 20 different versions of ffmpeg to satisfy the 20 different consumers? There wouldn't be any benefits to an internal lib either: If there is a security flaw fixed in ffmpeg 1004.0.1 the versions 900.x.y to 1003.x.y would be just as insecure as external packages.
I think there is benefit:
- If the library is statically linked in the application it's harder to
find in a quick audit.
- If the flaw affects a range of library versions (for instance,
899.x.y is unaffected), having them be packaged separately makes finding the affected versions and fixing them easier than finding out which versions of the private library each app has.
Also, one of the goals of a distribution is to make programs and libraries work together. So the approach we'd likely want to take is choosing a few versions of the library that we could support (probably in conjunction with other distros) and then porting the apps to one of those library versions and getting upstreams to update their private libs to those versions since we were willing to do the port for them.
Well, be my guest. Many have attempted to do so in some cases, but if upstream is *fast*, you will never catch up, and different distributions will chose different snapshots depending on the consumer apps they target, so coordinating will be more than difficult.
But in principle I agree: If we could afford the manpower to keep all comsumers compatible to the latest stable release of a library that would be the ideal world. But we don't have that manpower, which is not just packaging, but true upstream work.
A reality check shows that we even lack the definition of "latest stable upstream release" in some cases like ffmpeg.
Re-read what I wrote. I'm saying "choosing a few versions of the library that we could support (probably in conjunction with other distros) and then porting the apps to one of those library versions". If there's a set of packages that we care enough about that use a library which can't commit to an API for more than a month then we and other distros have to care about the situation and fix it between ourselves. And yes, this is programming that I'm talking about, not just packaging.
I'd recommend to soften this guideline to something as "the packager should try hard to use system libs, and try to communicate the benefits to upstream, but if there are reasons not to use system libs, then he should document this in the specfile".
I am not sure this is a good idea. I'd rather be stricter on this, because this would force devs to think about what they are doing and packagers think about the quality of what they are packaging.
The unpleasant truth is: If a package bundles "modified libs/apps" from elsewhere, which can't be installed in parallel to the original libs/apps, this package's dev's are doing something wrong.
The truth is that sometimes upstream makes some decisions, we can try to forward our position, but upstream may ignore us or maybe will have a good reason to counter our position (for ffmpeg and minilzo I believe upstream is correct and we should be the ones revising the guidelines).
That's why I suggest that the packager tries hard to do it in the typical shared way, but if there are sane reasons to use internal libs to not let the package dry out forever in some review queue like for x11vnc.
I am against this approach. But perhaps you'll have a useful counter to the points I raised.
The points you raised are nice in an ideal over manpowered world, but in reality we see packages stalling in review queues forever instead.
Anyway, these were my 0.02. I'm not really seeing anyone from the FPC favouring this, even less willing to pick this up and driving this through, so I'll rather stick to agreeing to disagree. :)
heh. Okay. :-)
-Toshio
On Friday, 19 September 2008 at 09:34, Axel Thimm wrote:
On Fri, Sep 19, 2008 at 01:41:50AM +1200, Nigel Jones wrote:
So I know we have http://fedoraproject.org/wiki/Packaging/Guidelines#Duplication_of_system_lib... which I think is pretty good, easy to understand and fairly simple.
The problem I think is that some upstream's still want to ship internal, modified libraries.
Upstream says "it's a guideline not a rule".
_MY_ question is, what can we (Fedora) do to make it clear that we have clear cut rules for why we don't want packages providing internal libraries?
[...]
Just to quote one such example: ffmpeg is a fast moving target, and any project depending on the lib API is cutting a checkout, patching it a up and using it for its own purposes. Replacing these internal ffmpegs with a system ffmpeg is a nightmare or even impossible w/o rewriting the app interface to it. Given that ffmpeg and friends fall under the patent forbidden class we don't see that directly in Fedora, but this issue is still out there.
I don't know if you've been following FFmpeg development lately, but they have improved over the last year or so to the point that no ABI breakage occurs without bumping the major version of the affected library. The pkg-config support is put properly in place, too, so if you haven't done that already, it's high time to begin convincing depdendent projects to start supporting shared FFmpeg. I've already begun working on fixing the main consumer of FFmpeg, MPlayer, to do that.
Regards, R.
On Fri, Sep 19, 2008 at 02:56:25PM +0200, Dominik 'Rathann' Mierzejewski wrote:
On Friday, 19 September 2008 at 09:34, Axel Thimm wrote:
On Fri, Sep 19, 2008 at 01:41:50AM +1200, Nigel Jones wrote:
So I know we have http://fedoraproject.org/wiki/Packaging/Guidelines#Duplication_of_system_lib... which I think is pretty good, easy to understand and fairly simple.
The problem I think is that some upstream's still want to ship internal, modified libraries.
Upstream says "it's a guideline not a rule".
_MY_ question is, what can we (Fedora) do to make it clear that we have clear cut rules for why we don't want packages providing internal libraries?
[...]
Just to quote one such example: ffmpeg is a fast moving target, and any project depending on the lib API is cutting a checkout, patching it a up and using it for its own purposes. Replacing these internal ffmpegs with a system ffmpeg is a nightmare or even impossible w/o rewriting the app interface to it. Given that ffmpeg and friends fall under the patent forbidden class we don't see that directly in Fedora, but this issue is still out there.
I don't know if you've been following FFmpeg development lately, but they have improved over the last year or so to the point that no ABI breakage occurs
Note I mentioned the API, which is still changing on a regular basis. For ffmpeg it doesn't actually help that there are no releases ever either.
without bumping the major version of the affected library. The pkg-config support is put properly in place, too, so if you haven't done that already, it's high time to begin convincing depdendent projects to start supporting shared FFmpeg. I've already begun working on fixing the main consumer of FFmpeg, MPlayer, to do that.
Unless ffmpeg actually releases anything again I doubt that too many projects will try to depend on an external shared lib whose API stability window is a few weeks.
This is not a criticism against the very nice work of the ffmpeg people. They just need the freedom of a non stable API for being able to develop as fast and take into account that projects need to cut a snapshot (actually they enforce this explicitely).
On Friday, 19 September 2008 at 18:58, Axel Thimm wrote:
On Fri, Sep 19, 2008 at 02:56:25PM +0200, Dominik 'Rathann' Mierzejewski wrote:
On Friday, 19 September 2008 at 09:34, Axel Thimm wrote:
On Fri, Sep 19, 2008 at 01:41:50AM +1200, Nigel Jones wrote:
So I know we have http://fedoraproject.org/wiki/Packaging/Guidelines#Duplication_of_system_lib... which I think is pretty good, easy to understand and fairly simple.
The problem I think is that some upstream's still want to ship internal, modified libraries.
Upstream says "it's a guideline not a rule".
_MY_ question is, what can we (Fedora) do to make it clear that we have clear cut rules for why we don't want packages providing internal libraries?
[...]
Just to quote one such example: ffmpeg is a fast moving target, and any project depending on the lib API is cutting a checkout, patching it a up and using it for its own purposes. Replacing these internal ffmpegs with a system ffmpeg is a nightmare or even impossible w/o rewriting the app interface to it. Given that ffmpeg and friends fall under the patent forbidden class we don't see that directly in Fedora, but this issue is still out there.
I don't know if you've been following FFmpeg development lately, but they have improved over the last year or so to the point that no ABI breakage occurs
Note I mentioned the API, which is still changing on a regular basis. For ffmpeg it doesn't actually help that there are no releases ever either.
API is not changing on a regular basis either. If there are incompatible changes, they are accompanied by a major version bump.
without bumping the major version of the affected library. The pkg-config support is put properly in place, too, so if you haven't done that already, it's high time to begin convincing depdendent projects to start supporting shared FFmpeg. I've already begun working on fixing the main consumer of FFmpeg, MPlayer, to do that.
Unless ffmpeg actually releases anything again I doubt that too many projects will try to depend on an external shared lib whose API stability window is a few weeks.
Actually most of what we have in livna/rpmfusion does work with external shared libs. And the API stability window is rather closer to 3 years.
Regards, R.
On Fri, Sep 19, 2008 at 07:55:38PM +0200, Dominik 'Rathann' Mierzejewski wrote:
Note I mentioned the API, which is still changing on a regular basis. For ffmpeg it doesn't actually help that there are no releases ever either.
API is not changing on a regular basis either. If there are incompatible changes, they are accompanied by a major version bump.
The soname changes are for the ABI changes, the API does change more than often.
without bumping the major version of the affected library. The pkg-config support is put properly in place, too, so if you haven't done that already, it's high time to begin convincing depdendent projects to start supporting shared FFmpeg. I've already begun working on fixing the main consumer of FFmpeg, MPlayer, to do that.
Unless ffmpeg actually releases anything again I doubt that too many projects will try to depend on an external shared lib whose API stability window is a few weeks.
Actually most of what we have in livna/rpmfusion does work with external shared libs. And the API stability window is rather closer to 3 years.
So the recent moving of the header files which is part of the API was three years ago and not that summer?
Everything that requires an upstream's intervention to make a library link again is API breakage. If the consumer app FTBFS it's an API breakage.
3 years ago ffmpeg was a different project. Try building anything from livna against a one year old ffmpeg, you'll be surprised. :/
On Saturday, 20 September 2008 at 11:31, Axel Thimm wrote:
On Fri, Sep 19, 2008 at 07:55:38PM +0200, Dominik 'Rathann' Mierzejewski wrote:
Note I mentioned the API, which is still changing on a regular basis. For ffmpeg it doesn't actually help that there are no releases ever either.
API is not changing on a regular basis either. If there are incompatible changes, they are accompanied by a major version bump.
The soname changes are for the ABI changes, the API does change more than often.
without bumping the major version of the affected library. The pkg-config support is put properly in place, too, so if you haven't done that already, it's high time to begin convincing depdendent projects to start supporting shared FFmpeg. I've already begun working on fixing the main consumer of FFmpeg, MPlayer, to do that.
Unless ffmpeg actually releases anything again I doubt that too many projects will try to depend on an external shared lib whose API stability window is a few weeks.
Actually most of what we have in livna/rpmfusion does work with external shared libs. And the API stability window is rather closer to 3 years.
So the recent moving of the header files which is part of the API was three years ago and not that summer?
While I agree that moving headers around is an API change, it is far less problematic than changing function or structure names. It did bring a long needed consistency to header locations, after all.
Everything that requires an upstream's intervention to make a library link again is API breakage.
Linking was unaffected by that change.
If the consumer app FTBFS it's an API breakage.
Well, it was trivial to fix.
Regards, R.
packaging@lists.fedoraproject.org
-
Axel Thimm -
Dominik 'Rathann' Mierzejewski -
Nigel Jones -
Paul F. Johnson -
Ralf Corsepius -
Toshio Kuratomi