The folks over at Coverity have offered to allow Fedora Extras to use their services, and would like a yes or no. Rather than make this decision in a vacuum, I believe that the Fedora Extras Steering Committee has earned the right to make this decision for themselves.
If we can get a decision by the end of the week, that would be great.
What follows is my own analysis, for whatever it's worth.
PROS
+ It's good technology, and has been used in Linux projects previously with success. Google "coverity linux" or something similar.
+ If we act on the results, it could be a great boon for the FE code quality in general.
+ It doesn't cost us anything.
+ It forms a relationship between Coverity and Red Hat, and sets the table for more work partnership later, if things go well.
+ Bugs are bugs, and flaws are flaws. We should be happy to know about them, however they are found, and we should fix them.
CONS
+ It's not open source, but there is no free alternative that can do the same thing.
+ We need to make sure it doesn't disrupt or break our build system too much. So that will require some technical work and time from certain folks.
My gut is that we should say that we're interested, and start hashing out the technical details of how it will all work with them.
If we go ahead, I think that in addition to the Board, someone in FESCO needs to "own" this and be our point person for technical questions, etc.
Thanks, Max
On Wed, Aug 30, 2006 at 01:29:36PM -0400, Max Spevack wrote:
The folks over at Coverity have offered to allow Fedora Extras to use their services, and would like a yes or no.
+1. More eyes (in this case, even electronic eyes) is good.
CONS
- We need to make sure it doesn't disrupt or break our build system too
much. So that will require some technical work and time from certain folks.
I thought their checking work was being done not on the build environment but on their own servers using SCM access. No?
scan.coverity.com also requires personal registration (sends an email with your name, email address, and "reason for needing access") for each project the list (which looks like it's one project per program/app, I don't see a category like "Fedora Extras" to sign up for blanketwise) in order to see the list of bugs. Any chance we could relax that somewhat, tie it into the FE accounts system or something? Else everyone wanting to look at that list needs to register at coverity's site, yes?
The current registration looks like:
To get access to the FOOBAR report please supply your: Full Name: Email: Your association with the project and purpose of access:
There's no privacy policy listed (though their main page's "subscribe to our mailing list" does say "We will NEVER share your e-mail with anyone and you can unsubscribe at any time." there's no similar statement or opt-out for sales calls made using the info you provide.
Otherwise, I do like the idea.
Thanks, Matt
Matt Domsch wrote:
I thought their checking work was being done not on the build environment but on their own servers using SCM access. No?
This setup would be separate and independent from the scan.coverity.com setup and access to the bug database would also be under FC/FE committee control.
On Wed, Aug 30, 2006 at 01:29:36PM -0400, Max Spevack wrote:
- It's not open source, but there is no free alternative that can do the
same thing.
I don't want to spoil anything and I'm not the activest FLOSS agitators, but I see a conflict of goals and tools.
The kernel-uses-bitkeeper-technology created more noise than it served good and bitkeeper was closer to open source than coverity while Linus was less pondering on FLOSS principles than the Fedora goals do, so projecting that to the future I see endless threads about the pure-FLOSS Linux using non-FLOSS tools.
There is an argument often brought up in these situations which goes like "since no FLOSS alternative exists, we need to use that". But the same is true about ipw* firmwares/closed source daemons, closed source 3D graphics and so on. There is even discussion of not allowing external kernel modules, even fully FLOSSed ones, in Fedora to demonstrate Fedora's embracement and loyality to FLOSS.
If we want to open the backdoor to non-FLOSS bits we will be blamed on being selective and non-open ourselves serving only our needs at hand.
Don't get me wrong, as I said I'm no FLOSS die-hard agitator, and I would personally welcome a code checker. But it does conflict with Fedora's manifesto and will create a flood of noise and bad marketing.
On Wed, 2006-08-30 at 21:57 +0200, Axel Thimm wrote:
On Wed, Aug 30, 2006 at 01:29:36PM -0400, Max Spevack wrote:
- It's not open source, but there is no free alternative that can do the
same thing.
I don't want to spoil anything and I'm not the activest FLOSS agitators, but I see a conflict of goals and tools.
The kernel-uses-bitkeeper-technology created more noise than it served good and bitkeeper was closer to open source than coverity while Linus was less pondering on FLOSS principles than the Fedora goals do, so projecting that to the future I see endless threads about the pure-FLOSS Linux using non-FLOSS tools.
There is an argument often brought up in these situations which goes like "since no FLOSS alternative exists, we need to use that". But the same is true about ipw* firmwares/closed source daemons, closed source 3D graphics and so on. There is even discussion of not allowing external kernel modules, even fully FLOSSed ones, in Fedora to demonstrate Fedora's embracement and loyality to FLOSS.
Not quite. Yes, Fedora's mantra is upstream in regards to kernel modules simply because it benefits everyone in the long run. But the current issue stems from maintenance in the long run and resource issues for the Core kernel devs. Please don't confuse that with FLOSS zealotry.
If we want to open the backdoor to non-FLOSS bits we will be blamed on being selective and non-open ourselves serving only our needs at hand.
Don't get me wrong, as I said I'm no FLOSS die-hard agitator, and I would personally welcome a code checker. But it does conflict with Fedora's manifesto and will create a flood of noise and bad marketing.
There are two major items you're ignoring.
1) Coverity is not required by any means. Your bitkeeper analogy is flawed because it was the _only_ SCM tool being used for the kernel. It was essentially a gate.
2) Fedora isn't shipping coverity. It doesn't taint the distribution in any way.
josh
We have been trying to keep Fedora's Infrastructure completely FOSS for the purpose of making it reproducible and easy to contribute improvements. This is a noble goal.
Comparing Coverity to Bitkeeper is not a fair comparison because Fedora and any projects that reproduce it would not depend on it. Coverity would in part protect Fedora, but this really is a tool for improving upstream projects, and Fedora would just make it easier to funnel analysis and reports.
We have long wanted to implement post-build check reports in order to improve package quality in an automated fashion. Coverity could just be another post-build check in that list.
On the other hand, we may want to implement Coverity in a different way than post-check. The output needs to be kept private to the individual package owners and possibly security group people so security embargoes can be handled in a responsible way in cooperation with upstream projects. We also want to avoid slowing down the build, sign and push process any further.
My Proposal ========== A good compromise would be for Coverity to be run outside of the scope of the Fedora Project as just a Red Hat thing. It would run asynchronously on the binary RPMS in pushed repositories. If Fedora contributors are interested in helping to better automate this they are free to do so.
This way Fedora and upstream benefits from Coverity analysis, and Fedora remains ideologically pure.
Thoughts?
Warren Togami wtogami@redhat.com
On Wed, 30 Aug 2006 13:29:36 -0400 (EDT), Max Spevack wrote:
My gut is that we should say that we're interested, and start hashing out the technical details of how it will all work with them.
If we go ahead, I think that in addition to the Board, someone in FESCO needs to "own" this and be our point person for technical questions, etc.
I agree on both points.
Christian
On Wed, 2006-08-30 at 18:10 -0400, Warren Togami wrote:
We have been trying to keep Fedora's Infrastructure completely FOSS for the purpose of making it reproducible and easy to contribute improvements. This is a noble goal.
Which infrastructure? Extras or Core. Because if you mean Fedora in general, then I'm sorry but that's a bit off. The Core buildsys is not open sourced.
Comparing Coverity to Bitkeeper is not a fair comparison because Fedora and any projects that reproduce it would not depend on it. Coverity would in part protect Fedora, but this really is a tool for improving upstream projects, and Fedora would just make it easier to funnel analysis and reports.
Yes.
We have long wanted to implement post-build check reports in order to improve package quality in an automated fashion. Coverity could just be another post-build check in that list.
Yes.
On the other hand, we may want to implement Coverity in a different way than post-check. The output needs to be kept private to the individual package owners and possibly security group people so security embargoes can be handled in a responsible way in cooperation with upstream projects. We also want to avoid slowing down the build, sign and push process any further.
My Proposal
A good compromise would be for Coverity to be run outside of the scope of the Fedora Project as just a Red Hat thing. It would run asynchronously on the binary RPMS in pushed repositories. If Fedora contributors are interested in helping to better automate this they are free to do so.
Erm... doesn't coverity need _source_?
This way Fedora and upstream benefits from Coverity analysis, and Fedora remains ideologically pure.
*cough* Core buildsys *cough*
josh
On Wed, 2006-08-30 at 18:10 -0400, Warren Togami wrote:
My Proposal
A good compromise would be for Coverity to be run outside of the scope of the Fedora Project as just a Red Hat thing. It would run asynchronously on the binary RPMS in pushed repositories. If Fedora contributors are interested in helping to better automate this they are free to do so.
This way Fedora and upstream benefits from Coverity analysis, and Fedora remains ideologically pure.
Thoughts?
Warren, its a _source_ analysis tool. Running it on binary packages won't do much good...
On Wed, 2006-08-30 at 21:41 -0500, Josh Boyer wrote:
On Wed, 2006-08-30 at 18:10 -0400, Warren Togami wrote:
We have been trying to keep Fedora's Infrastructure completely FOSS for the purpose of making it reproducible and easy to contribute improvements. This is a noble goal.
Which infrastructure? Extras or Core. Because if you mean Fedora in general, then I'm sorry but that's a bit off. The Core buildsys is not open sourced.
Yes, but not b/c we chose it. That's just a function of managerial stupidity.
There will be a reckoning.
oh yes, there will be a reckoning.
-sv
On Wed, 2006-08-30 at 22:47 -0400, seth vidal wrote:
On Wed, 2006-08-30 at 21:41 -0500, Josh Boyer wrote:
On Wed, 2006-08-30 at 18:10 -0400, Warren Togami wrote:
We have been trying to keep Fedora's Infrastructure completely FOSS for the purpose of making it reproducible and easy to contribute improvements. This is a noble goal.
Which infrastructure? Extras or Core. Because if you mean Fedora in general, then I'm sorry but that's a bit off. The Core buildsys is not open sourced.
Yes, but not b/c we chose it. That's just a function of managerial stupidity.
I never intended to imply otherwise :)
There will be a reckoning.
oh yes, there will be a reckoning.
I look forward to the day. :)
josh
Axel Thimm wrote:
The kernel-uses-bitkeeper-technology created more noise than it served good and bitkeeper was closer to open source than coverity while Linus was less pondering on FLOSS principles than the Fedora goals do, so projecting that to the future I see endless threads about the pure-FLOSS Linux using non-FLOSS tools.
There is an argument often brought up in these situations which goes like "since no FLOSS alternative exists, we need to use that". But the same is true about ipw* firmwares/closed source daemons, closed source 3D graphics and so on. There is even discussion of not allowing external kernel modules, even fully FLOSSed ones, in Fedora to demonstrate Fedora's embracement and loyality to FLOSS.
Well, maybe Fedora should ditch apache, wireshark, perl, python, samba, subversion, vim, xmms and a lot of other packages then, as they are already checked by Coverity. Check the banner on scan.coverity.com, even the Linux kernel is.
Nils Breunese.
On Wed, Aug 30, 2006 at 03:05:47PM -0500, Josh Boyer wrote:
On Wed, 2006-08-30 at 21:57 +0200, Axel Thimm wrote:
On Wed, Aug 30, 2006 at 01:29:36PM -0400, Max Spevack wrote:
- It's not open source, but there is no free alternative that can do the
same thing.
I don't want to spoil anything and I'm not the activest FLOSS agitators, but I see a conflict of goals and tools.
The kernel-uses-bitkeeper-technology created more noise than it served good and bitkeeper was closer to open source than coverity while Linus was less pondering on FLOSS principles than the Fedora goals do, so projecting that to the future I see endless threads about the pure-FLOSS Linux using non-FLOSS tools.
There is an argument often brought up in these situations which goes like "since no FLOSS alternative exists, we need to use that". But the same is true about ipw* firmwares/closed source daemons, closed source 3D graphics and so on. There is even discussion of not allowing external kernel modules, even fully FLOSSed ones, in Fedora to demonstrate Fedora's embracement and loyality to FLOSS.
Not quite. Yes, Fedora's mantra is upstream in regards to kernel modules simply because it benefits everyone in the long run. But the current issue stems from maintenance in the long run and resource issues for the Core kernel devs. Please don't confuse that with FLOSS zealotry.
If we want to open the backdoor to non-FLOSS bits we will be blamed on being selective and non-open ourselves serving only our needs at hand.
Don't get me wrong, as I said I'm no FLOSS die-hard agitator, and I would personally welcome a code checker. But it does conflict with Fedora's manifesto and will create a flood of noise and bad marketing.
There are two major items you're ignoring.
- Coverity is not required by any means. Your bitkeeper analogy is
flawed because it was the _only_ SCM tool being used for the kernel. It was essentially a gate.
That's debatable. From a user's POV the SCM used is not seen. And precicely because of the reasons you outline and the FLOSS ideology of several kernel developers there were gateways to other SCMs as well as simple diff&patch submissions.
- Fedora isn't shipping coverity. It doesn't taint the distribution in
any way.
That doesn't matter, the kernel tarballs didn't contain bitkeeper, too. The cause does not justify the means.
Furthermore the fact that you mentioned in other posts to this thread, that FC uses beehive which is closed source, has its truth and shame, but whenever this was (seldomly) addressed in the past people kept saying that they would iron it out and open-source it. Meanwhile there emerged better open source technologies on the market, so the demand and motivation to do so is low. But I'm sure that in principle RH would go through the effort to open-source it if it was worth while, just as they did with much larger formerly closed source projects like gfs and ds.
Anyway to cut a long story short it boilds down to a matter of principles. Ask yourself *why* Fedora has a certain FLOSS ideology that disallows for example packaging and shipping of non-FLOSS software and the answer will be applying to why Fedora should not be using/supporting non-FLOSS in any other way. See also the discussion of using non-open media formats on redhat.com. It all spins around a chosen core ideology.
Again: Don't mistake me for a FLOSS zealot, I'm just seeing these issues coming up. As I see it it will either lead to confirming the strict endorsement of FLOSS ideology and officially refuse any non-FLOSS contributions or it will lead to a softening of Fedora's definition of FLOSS goals to allow for using non-FLOSS technology.
Matthias Clasen wrote:
On Wed, 2006-08-30 at 18:10 -0400, Warren Togami wrote:
My Proposal
A good compromise would be for Coverity to be run outside of the scope of the Fedora Project as just a Red Hat thing. It would run asynchronously on the binary RPMS in pushed repositories. If Fedora contributors are interested in helping to better automate this they are free to do so.
This way Fedora and upstream benefits from Coverity analysis, and Fedora remains ideologically pure.
Thoughts?
Warren, its a _source_ analysis tool. Running it on binary packages won't do much good...
Same thing in principle. Why does Coverity need to be in our build system? It could very well work asynchronously outside of Fedora from our SRPMS.
Warren Togami wtogami@redhat.com
Warren Togami wrote:
Same thing in principle. Why does Coverity need to be in our build system? It could very well work asynchronously outside of Fedora from our SRPMS.
If you don't do it in the build system there is no guarantee that the tests actually test the same thing as the code which is shipped.
Ulrich Drepper wrote:
Warren Togami wrote:
Same thing in principle. Why does Coverity need to be in our build system? It could very well work asynchronously outside of Fedora from our SRPMS.
If you don't do it in the build system there is no guarantee that the tests actually test the same thing as the code which is shipped.
How so? mock putting together its own buildroots and doing rpmbuild -bp is not the same thing?
Warren Togami wtogami@redhat.com
Warren Togami wrote:
How so? mock putting together its own buildroots and doing rpmbuild -bp is not the same thing?
The nature of the two tasks running independently automatically makes it impossible to guarantee that the same code is analyzed and shipped. The nice thing about the way the tools work is that they can work transparently while building the actual packages.
This not only guarantees the same sources are analyzed and shipped, it also significantly reduces the resources needed since the build root and untarred packages don't have to be created again.
Also note that the actual Coverity tool isn't run at the time of the data collection. This is only a tiny little piece which collects the data. The actual analysis is done in a separate step.
On Wed, 2006-08-30 at 18:10 -0400, Warren Togami wrote:
We have been trying to keep Fedora's Infrastructure completely FOSS for the purpose of making it reproducible and easy to contribute improvements. This is a noble goal.
Comparing Coverity to Bitkeeper is not a fair comparison because Fedora and any projects that reproduce it would not depend on it. Coverity would in part protect Fedora, but this really is a tool for improving upstream projects, and Fedora would just make it easier to funnel analysis and reports.
We have long wanted to implement post-build check reports in order to improve package quality in an automated fashion. Coverity could just be another post-build check in that list.
On the other hand, we may want to implement Coverity in a different way than post-check. The output needs to be kept private to the individual package owners and possibly security group people so security embargoes can be handled in a responsible way in cooperation with upstream projects. We also want to avoid slowing down the build, sign and push process any further.
My Proposal
A good compromise would be for Coverity to be run outside of the scope of the Fedora Project as just a Red Hat thing. It would run asynchronously on the binary RPMS in pushed repositories. If Fedora contributors are interested in helping to better automate this they are free to do so.
Note that we may not be able to deploy the Coverity bits on the same build machines that Extras packages are built on right now; mainly because the people who have access to those machines are not employed by Red Hat. There's an open question as to whether Coverity will permit non-Red Hat contributors access to machines that run the proprietary Coverity binaries (which contain a fair amount of their IP and trade secrets and such) without signing some legal document. The sensitive bits are precisely those that run during the package build.
I think the easiest solution at the current time is to run the Coverity scans on one or two parallel machines that harvest successful build results from the actual Extras buildsystem, and which non-Red Hat people don't have shell access to. Furthermore, this ensures that released Extras packages are fully externally reproducible, since the Coverity scanner sits between the build scripts and GCC. The web-based reports portal would be still be accessible to package maintainers of course.
Like Warren says, then there's no slowdown for the build system, we stay clear of any difficult contractual or legal issues related to access to Coverity binaries, and the packages are completely externally reproducible.
Is there any extra hardware available? The Coverity bits don't run on PPC yet either, they are i386 & x86-64 only right now, so we don't need any more OpenPOWER boxes, only a few more Dells.
Dan
This way Fedora and upstream benefits from Coverity analysis, and Fedora remains ideologically pure.
Thoughts?
Warren Togami wtogami@redhat.com
On Thu, 2006-08-31 at 16:20 -0400, Dan Williams wrote:
Note that we may not be able to deploy the Coverity bits on the same build machines that Extras packages are built on right now;
I would hope that the infrastructure team would balk on putting any proprietary software on the build systems anyway. That's a sure fire way to cause undiagnosable issues and to prevent moving forward with the OS installs. Also there is the security concern. We don't want un audited bits of software being part of our build process. Sure coverity could say we audited our own software, but still.... Call it the paranoia in me, but I certainly wouldn't want it on my build systems, and I don't think internally at Red Hat we'll (release engineering) be to keen on putting it in OUR build system either.
infrastructure@lists.fedoraproject.org