On Fri, 2010-12-03 at 23:33 +0100, Karol Cieśla wrote:
Hi, i agree with both of you ,so that rather than just sending e-mails i would like initiate something which could improve the current situation.
So the steps I see now look as follows: I stage
- someone could sketch out/outline/describe the functionality of
Wiki/TCMS we would like to have for the test cases (if possible also provide graphical form) -i.e table, page, buttons
Hurry and I discussed this topic yesterday on IRC. I suspect she'll take a similar approach, but the general idea will be to first identify what our QA community needs are when it comes to test documentation/results. For example, what works well, what isn't working well, where do we want to improve when it comes to our current test management.
- discuss with the whole community such form -pros,cons
As always, I expect the process will be transparent and there will be plenty of opportunity for anyone interested to get involved and help move things forward.
- reach out to people maintaining the Wiki and ask them if they can
create such form wit help of Wiki/TCMS and can sustain bigger number of test cases (i.e up to 300)
If the analysis shows that investing in wiki-based solution is the most sustainable option forward, there are several solutions available (http://www.mediawiki.org/wiki/Extension:Semantic_Forms and possibly http://www.mediawiki.org/wiki/Extension:StructuredInput). My impression with Semantic was that while it may scratch the wiki form-input itch for us, it's certainly not without a startup cost and I'm still unclear whether that's the right tool for the job. It's a fun experiment, but without knowing what where our current gaps are, and where we want to improve, it's hard to make a decision to invest in semantic-mediawiki over another actively maintained upstream project.
II stage
To ponder as the whole community the process of adding/creating/deleting/marking/prioritizing test cases.
While it would be an interesting discussion, I don't know if we'd be able to move forward with actionable work after a general community ponder session. I'd fear that would revolve too much around ponies [1]. I'm inclined to put emphasis on reviewing what we're doing now, understand why we're doing it, document the pro's con's and prioritize the MUSTHAVE features.
P.S -i can try to prepare for the middle of the next week some draft to point 1 -it will be something simple but reflecting the idea.
Always welcome additional hands to help move the discussion forward!
Thanks, James
W dniu 2010-12-03 17:36, James Laska pisze:
Just a few extra thoughts on the subject ...
On Fri, 2010-12-03 at 07:30 -0800, Adam Williamson wrote:
On Fri, 2010-12-03 at 16:06 +0100, xcieja wrote:
Hi, yes, you are right there are tests, but in my opinion they are in few different places under different categories.
That wasn't what I meant: I meant we already use the Wiki for the purposes you identified as an advantage of a TCMS (listing the tests that need to be performed in relation to some specific process, and whether they have already been performed, by whom, and with which result).
I think we could organise them better -i.e create test category and put all of them instead of many places.
We sure could, but we don't necessarily need a TCMS to do this. :) Note that we do try to keep them all within one Wiki namespace and we do use Wiki categories to organize some test cases.
Moreover, i just have taken a look briefly and i see there are round 100 test cases in total (please correct me if i am wrong).I think that for such project/system it is not enough at all.
We have big community, let`s assume everyone from QA create one test, we will have quite huge number of tests and obviously more faults detected before main release, less corrections after=better stability,usability-> better overall opinion.
Sure, we can always do with more test cases.
More test cases/plans would certainly change the conversation a bit. I think we all want to increase the value that the Fedora QA team can offer to the project. One way to increase our value is by improving our test coverage by way of test documentation (procedures, plans and cases). There are plenty of other ways ... but we can save those for other threads.
I've always been hesitant to add tests for the sake of adding tests. Test plans/cases are just like software. If the tests aren't addressing a priority issue, they won't be used as much, and like unused software, will suffer from bit rot. The best test cases/plans are the ones frequently used, referenced and have maintainer buy-in. Meaning, if the tests fail, the maintainer cares. I want to grow the library of tests we maintain and run, but hopefully grow in a manner and pace that we, as a community, can sustain.
With the test plans that Adam points to, I'm pretty confident in our ability to develop, discuss/debate and execute desktop and installation tests as a community. We've ironed out the kinks in the workflow, increased community engagement and developed good test plans as a result. My impression is we are ready for additional test areas.
That's what's exciting to me about the proventesters effort. As you can tell from recent (and old) devel@ list threads, testing proposed updates is important work that's needed, requested by package maintainers and well under-documentated. I don't worry as much that tests written for frequent proventester use will go stale given it's been a long-standing exposure in the project. Also, given the huge number of components in Fedora, there is room for just about every contributor to participate and carve out a niche. But which tests do we prioritize first, where do we write the tests, where to review+discuss them, how to run them etc... (more on this later).
For me, these are two separate (but related) efforts. TCMS is tool designed to address specific workflow/tracking needs. We also need to determine how to best to sustainably expand the test coverage we can offer to the project. We have a wiki-based "TCMS" now. It has met our needs for the current set of organized test efforts. It's not perfect, but the return on the investment has been huge. The questions I'd like to see answered in ticket#152, is (1) whether the wiki can continue to scale as our test management needs grow, and (2) what aspects of our wiki-based TCMS are good/bad?
Thanks, James