Hey, folks.
So in the recent proven tester discussion, and in various other threads, I've oft stated that the limits of the current Bodhi karma system are a significant problem, and the planned Bodhi 2.0 karma system has to potentially to significantly improve our update testing process. But it occurred to me that I haven't really laid out why in much detail, and those who aren't as involved with the process as we in QA are might not have a really clear vision of why this is so important.
So, I thought I'd lay it out in the form of a glorious vision of the future! Note: I have zero UI design skills. This UI as described would suck. But the idea is that there would be a decent UI which *represents all the described choices*.
In the Great Bodhi Of The Future, for any given update, a tester will not simply have a comment box and a drop-down for -1, 0 or +1. They will see:
* The list of test cases associated with the package, with a PASS / FAIL choice for each
* Checkboxes for 'This update does / does not break critical path functionality' (with a link out to the critpath definition)
* A checkbox for 'I installed this update and continued to use my system as normal and did not note any regressions'
* Any custom choices the package maintainer opts to provide, via some kind of interface to Bodhi
This is the kind of flexibility that would make karma massively more useful. The tighter definition of what feedback actually *means* provides far more information to the maintainer, and enables us to automate certain outcomes much more aggressively.
For me, one the principal benefits of such a system would be that we could make the 'This update breaks critical path functionality' checkbox an absolutely red flashing light, wailing siren emergency button. I mean, you hit that thing and trucks roll from Fedora HQ, metaphorically speaking. It would have a confirmation page which clearly described the impact of asserting that an update broke critpath, so we could be confident that it didn't get falsely triggered very often. I'd suggest that:
1. Any update that is marked as 'critpath breaking' by a FAS-registered tester would be blocked from going any further in the update process without manual intervention (no autopushes at all)
2. Any update marked as 'critpath breaking' by a proven tester would be blocked from being pushed stable at all - automatically or manually - until the PT modified the feedback or it was overridden by someone with appropriately godlike powers (TBD, but probably not just the maintainer)
3. Any update marked as 'critpath breaking' should probably get announced on at least test-announce and/or devel-announce
4. Any update marked as 'critpath breaking' *after it has already been pushed* would similarly trigger a major response: notify maintainer very hard, notify lists, generally do stuff to make sure it gets immediate attention
We would also obviously be able to offer maintainers more nuanced options for autopushing updates - require a certain number of passes on a certain set of test cases, for instance.
To go a bit into the theory, the really nasty limitation of the current system is we simply have very little easily consumable indication of what the karma provided on an update *means*. A +1 can mean anything from 'I booted and nothing exploded' to 'I tested every line of this code, then did it backwards standing on my head'. A -1 can mean 'I booted this update and then my cat got sick, I think there's a connection!', 'this update breaks $REALLY OBSCURE FEATURE Z', or 'this update exploded my monitor'. We just _don't know_. You can get the information from comments, usually, but that's obviously a complete non-starter for any kind of programmatized or automated action based on the feedback. In particular, we currently can't do anything dramatic based on negative feedback because of the uncertainty around exactly what it means. We do have a few policies about when to file negative feedback, but even this only very slightly mitigates the problem - we can't say 'only file negative feedback if the update breaks critpath', that's just not feasible. So we can't identify the 'really really negative' feedback and respond appropriate drastically to it without causing lots of pain through false positives (or rather, false negatives...)
With a more advanced feedback system we can identify the really big problems and be much more aggressive about handling them, without over-reacting to smaller problems. We could, correspondingly, be a bit less strict about how much *positive* feedback you need to push an update, if we can be a bit more confident that we'll definitely identify the really bad problems through negative feedback.
On Tue, Nov 22, 2011 at 22:03, Adam Williamson awilliam@redhat.com wrote:
- Any update marked as 'critpath breaking' by a proven tester would be
blocked from being pushed stable at all - automatically or manually - until the PT modified the feedback or it was overridden by someone with appropriately godlike powers (TBD, but probably not just the maintainer)
Here I'd add something like "three proventesters can turn over one proventester" and only if that doesn't work activate someone with godlike powers. Just because godlike powers should never have to be used in an ideal community/world and I think this overturn rule can help to get us a little closer to ideal ;)
-- red
On Tue, 2011-11-22 at 22:17 +0100, Sandro "red" Mathys wrote:
On Tue, Nov 22, 2011 at 22:03, Adam Williamson awilliam@redhat.com wrote:
- Any update marked as 'critpath breaking' by a proven tester would be
blocked from being pushed stable at all - automatically or manually - until the PT modified the feedback or it was overridden by someone with appropriately godlike powers (TBD, but probably not just the maintainer)
Here I'd add something like "three proventesters can turn over one proventester" and only if that doesn't work activate someone with godlike powers. Just because godlike powers should never have to be used in an ideal community/world and I think this overturn rule can help to get us a little closer to ideal ;)
Possibly, but in that case, I'd want the mechanism to require the 'overturning' PTs to explicitly consider the 'omg raptors' PT. i.e., if 3 PTs check 'doesn't break critpath' and then one PT checks 'breaks critpath!', those three earlier tests should not be considered to override the one later one. Other PTs would have to explicitly say that they tested whatever the 'omg raptors!' PT said was broken and could not reproduce, in order to overturn their feedback.
(And then we'd go after the incorrect PT with the 2x4 with a rusty nail in it...)
On Tue, Nov 22, 2011 at 22:59, Adam Williamson awilliam@redhat.com wrote:
(And then we'd go after the incorrect PT with the 2x4 with a rusty nail in it...)
Sometimes as a PT I feel like I would like to +1 an update, but I don't feel like waving this critpath update through. Mostly because I don't know the software good enough or because another tester had a negative experience. In that situations I'd like to upkarma the update without giving it the critpath-OK.
Take this example: https://admin.fedoraproject.org/updates/FEDORA-2011-16237/kernel-3.1.2-1.fc1...
Right now, a PT has voiced that he's seen a regression. He also states that the issue he's seeing might have other reasons. So I'd like to have him eventually +1/-1 this update, but still I have tested this and would like to upkarma it, to contribute to the total number of necessary karma.
Maybe there should simply be another checkbox "treat me as a non-PT this single time".
-- red
On 11/22/2011 04:03 PM, Adam Williamson wrote:
Hey, folks.
So in the recent proven tester discussion, and in various other threads, I've oft stated that the limits of the current Bodhi karma system are a significant problem, and the planned Bodhi 2.0 karma system has to potentially to significantly improve our update testing process. But it occurred to me that I haven't really laid out why in much detail, and those who aren't as involved with the process as we in QA are might not have a really clear vision of why this is so important.
So, I thought I'd lay it out in the form of a glorious vision of the future! Note: I have zero UI design skills. This UI as described would suck. But the idea is that there would be a decent UI which *represents all the described choices*.
IMHO, this seems like the sort of thing that would benefit from a discussion session at the upcoming FUDCon where there will be members of the Fedora Design team and the main coders of Bodhi. Not that I want to do it in secret, but coming up with a solid battle plan that also has the decent UI (in a mockup) as a first step seems like a better plan to success.
~tom
== Fedora Project
On Fri, 2011-12-02 at 17:24 -0500, Tom Callaway wrote:
IMHO, this seems like the sort of thing that would benefit from a discussion session at the upcoming FUDCon where there will be members of the Fedora Design team and the main coders of Bodhi. Not that I want to do it in secret, but coming up with a solid battle plan that also has the decent UI (in a mockup) as a first step seems like a better plan to success.
That certainly seems like a good idea.