#152: Test Cases Management ---------------------------+------------------------------------------------ Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Keywords: retrospective | ---------------------------+------------------------------------------------ = problem =
We've been using mediawiki to manage tests and record results, though it's easy to approach, it has limitations such as managing, tracking and querying cases+results. So is it time for us to consider another solution such as TCMS?
= analysis =
Here I quoted the suggestion from Victer Chen:
some advantages of TCMS: * Better to record, track and query test results, * Allow user focus on test contents instead of document maintenance. * Share test cases
However, it couldn't be flexible as wiki. I think the most important things need balance are below:
- Barriers Fedora should provide very low barriers. TCMS could be configured to allow anonymous user login and limit it's permission. Also, we can consider using fedora account to login TCMS.
- Safety Wiki provides the ability to rollback content easily, while TCMS couldn't. What TCMS can do is recording all the history and allow user recovery it manually. But if we configure the permission well, it should not be a big problem.
= enhancement recommendation =
Work with TCMS team and package it to Fedora. Set up the system and use it in a small scale first. If it turns out good, enlarger the scale.
#152: Test Cases Management --------------------------+------------------------------------------------- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------------+------------------------------------------------- Comment (by adamwill):
I know James has been talking about this with some RH people. We discussed it on the phone and agreed that we like the idea in principle but the ease and flexibility of the current wiki system is something we don't want to lose, so we thought we should set some quite high barriers: any tcms system must be as easy to use (both for those creating the test cases and those running them) as the wiki is.
Hi, i can say i am new to Fedora in general as project but i have participated in many testing campaigns professionaly, so honestly i see this TCMS(?) or any other way to organise test cases as big add value.
It would be really great if we had a list of test cases for whole system (of course everyone can contibute) and mark each of them as successful/failed, track the status etc...
The one of the advantages is that i am looking at such list of test cases and i am able to easily figure out what has been already tested and wht remaining is.
If have missed something and such 'thing" exists please correct me an forget the idea, otherwise what do you think ??
Regards, Karol
W dniu 20:59, Fedora QA pisze:
ם�z�BjǬ1�ځ鞞ߑz�+���㰝����秅��zg��+Z��'{늊�)���̊W�����h��y ���w��h��W��*'E�(��b�r�� +v��,��-�� ��z{[ɧZ����'�Z��!j��z{Z�H�����a��"�(��G��)��v+ �� v+h��^�'y��j �y�aj� �) ��u梞��ȩ����^y� jw_��bn)b�* ����ޞ�"�+2�צ��(��a�x0y�'����%�Ǭ������� �.�� ��&z����b�ڮ���Ƨ��&�̬�馺�[y� j̭�� n�a~�톋 r���)��z�\jǬjwm�� ���x-�隲�^�)"���rG�Q��l���v�چ�-y�+���v�ک��� ��y��&����W����m���y�+j,��h��y�+j��y�+i ��+r�.���e===
On Thu, 2010-12-02 at 23:12 +0100, xcieja wrote:
Hi, i can say i am new to Fedora in general as project but i have participated in many testing campaigns professionaly, so honestly i see this TCMS(?) or any other way to organise test cases as big add value.
It would be really great if we had a list of test cases for whole system (of course everyone can contibute) and mark each of them as successful/failed, track the status etc...
The one of the advantages is that i am looking at such list of test cases and i am able to easily figure out what has been already tested and wht remaining is.
If have missed something and such 'thing" exists please correct me an forget the idea, otherwise what do you think ??
We already do this, for most of our programmed testing, via the Wiki, using matrices. See:
https://fedoraproject.org/wiki/Test_Results:Current_Desktop_Test https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test
for instance.
Hi, yes, you are right there are tests, but in my opinion they are in few different places under different categories. I think we could organise them better -i.e create test category and put all of them instead of many places.
Moreover, i just have taken a look briefly and i see there are round 100 test cases in total (please correct me if i am wrong).I think that for such project/system it is not enough at all.
We have big community, let`s assume everyone from QA create one test, we will have quite huge number of tests and obviously more faults detected before main release, less corrections after=better stability,usability-> better overall opinion.
Please don`t get me wrong -if it works fine like now i don`t want to create something new.Please treat as as comments of newcomer in the project.
Regards, Karol W dniu 20:59, Adam Williamson pisze:
On Thu, 2010-12-02 at 23:12 +0100, xcieja wrote:
Hi, i can say i am new to Fedora in general as project but i have participated in many testing campaigns professionaly, so honestly i see this TCMS(?) or any other way to organise test cases as big add value.
It would be really great if we had a list of test cases for whole system (of course everyone can contibute) and mark each of them as successful/failed, track the status etc...
The one of the advantages is that i am looking at such list of test cases and i am able to easily figure out what has been already tested and wht remaining is.
If have missed something and such 'thing" exists please correct me an forget the idea, otherwise what do you think ??
We already do this, for most of our programmed testing, via the Wiki, using matrices. See:
https://fedoraproject.org/wiki/Test_Results:Current_Desktop_Test https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test
for instance.
On Fri, 2010-12-03 at 16:06 +0100, xcieja wrote:
Hi, yes, you are right there are tests, but in my opinion they are in few different places under different categories.
That wasn't what I meant: I meant we already use the Wiki for the purposes you identified as an advantage of a TCMS (listing the tests that need to be performed in relation to some specific process, and whether they have already been performed, by whom, and with which result).
I think we could organise them better -i.e create test category and put all of them instead of many places.
We sure could, but we don't necessarily need a TCMS to do this. :) Note that we do try to keep them all within one Wiki namespace and we do use Wiki categories to organize some test cases.
Moreover, i just have taken a look briefly and i see there are round 100 test cases in total (please correct me if i am wrong).I think that for such project/system it is not enough at all.
We have big community, let`s assume everyone from QA create one test, we will have quite huge number of tests and obviously more faults detected before main release, less corrections after=better stability,usability-> better overall opinion.
Sure, we can always do with more test cases.
Please don`t get me wrong -if it works fine like now i don`t want to create something new.Please treat as as comments of newcomer in the project.
Not at all :)
Just a few extra thoughts on the subject ...
On Fri, 2010-12-03 at 07:30 -0800, Adam Williamson wrote:
On Fri, 2010-12-03 at 16:06 +0100, xcieja wrote:
Hi, yes, you are right there are tests, but in my opinion they are in few different places under different categories.
That wasn't what I meant: I meant we already use the Wiki for the purposes you identified as an advantage of a TCMS (listing the tests that need to be performed in relation to some specific process, and whether they have already been performed, by whom, and with which result).
I think we could organise them better -i.e create test category and put all of them instead of many places.
We sure could, but we don't necessarily need a TCMS to do this. :) Note that we do try to keep them all within one Wiki namespace and we do use Wiki categories to organize some test cases.
Moreover, i just have taken a look briefly and i see there are round 100 test cases in total (please correct me if i am wrong).I think that for such project/system it is not enough at all.
We have big community, let`s assume everyone from QA create one test, we will have quite huge number of tests and obviously more faults detected before main release, less corrections after=better stability,usability-> better overall opinion.
Sure, we can always do with more test cases.
More test cases/plans would certainly change the conversation a bit. I think we all want to increase the value that the Fedora QA team can offer to the project. One way to increase our value is by improving our test coverage by way of test documentation (procedures, plans and cases). There are plenty of other ways ... but we can save those for other threads.
I've always been hesitant to add tests for the sake of adding tests. Test plans/cases are just like software. If the tests aren't addressing a priority issue, they won't be used as much, and like unused software, will suffer from bit rot. The best test cases/plans are the ones frequently used, referenced and have maintainer buy-in. Meaning, if the tests fail, the maintainer cares. I want to grow the library of tests we maintain and run, but hopefully grow in a manner and pace that we, as a community, can sustain.
With the test plans that Adam points to, I'm pretty confident in our ability to develop, discuss/debate and execute desktop and installation tests as a community. We've ironed out the kinks in the workflow, increased community engagement and developed good test plans as a result. My impression is we are ready for additional test areas.
That's what's exciting to me about the proventesters effort. As you can tell from recent (and old) devel@ list threads, testing proposed updates is important work that's needed, requested by package maintainers and well under-documentated. I don't worry as much that tests written for frequent proventester use will go stale given it's been a long-standing exposure in the project. Also, given the huge number of components in Fedora, there is room for just about every contributor to participate and carve out a niche. But which tests do we prioritize first, where do we write the tests, where to review+discuss them, how to run them etc... (more on this later).
For me, these are two separate (but related) efforts. TCMS is tool designed to address specific workflow/tracking needs. We also need to determine how to best to sustainably expand the test coverage we can offer to the project. We have a wiki-based "TCMS" now. It has met our needs for the current set of organized test efforts. It's not perfect, but the return on the investment has been huge. The questions I'd like to see answered in ticket#152, is (1) whether the wiki can continue to scale as our test management needs grow, and (2) what aspects of our wiki-based TCMS are good/bad?
Thanks, James
Hi, i agree with both of you ,so that rather than just sending e-mails i would like initiate something which could improve the current situation.
So the steps I see now look as follows: *I stage* _1)_ someone could sketch out/outline/describe the functionality of Wiki/TCMS we would like to have for the test cases (if possible also provide graphical form) -i.e table, page, buttons _2)_ discuss with the whole community such form -pros,cons _3)_ reach out to people maintaining the Wiki and ask them if they can create such form wit help of Wiki/TCMS and can sustain bigger number of test cases (i.e up to 300)
*II stage*
To ponder as the whole community the process of adding/creating/deleting/marking/prioritizing test cases.
P.S -i can try to prepare for the middle of the next week some draft to point 1 -it will be something simple but reflecting the idea.
Regards, Karol
W dniu 2010-12-03 17:36, James Laska pisze:
Just a few extra thoughts on the subject ...
On Fri, 2010-12-03 at 07:30 -0800, Adam Williamson wrote:
On Fri, 2010-12-03 at 16:06 +0100, xcieja wrote:
Hi, yes, you are right there are tests, but in my opinion they are in few different places under different categories.
That wasn't what I meant: I meant we already use the Wiki for the purposes you identified as an advantage of a TCMS (listing the tests that need to be performed in relation to some specific process, and whether they have already been performed, by whom, and with which result).
I think we could organise them better -i.e create test category and put all of them instead of many places.
We sure could, but we don't necessarily need a TCMS to do this. :) Note that we do try to keep them all within one Wiki namespace and we do use Wiki categories to organize some test cases.
Moreover, i just have taken a look briefly and i see there are round 100 test cases in total (please correct me if i am wrong).I think that for such project/system it is not enough at all.
We have big community, let`s assume everyone from QA create one test, we will have quite huge number of tests and obviously more faults detected before main release, less corrections after=better stability,usability-> better overall opinion.
Sure, we can always do with more test cases.
More test cases/plans would certainly change the conversation a bit. I think we all want to increase the value that the Fedora QA team can offer to the project. One way to increase our value is by improving our test coverage by way of test documentation (procedures, plans and cases). There are plenty of other ways ... but we can save those for other threads.
I've always been hesitant to add tests for the sake of adding tests. Test plans/cases are just like software. If the tests aren't addressing a priority issue, they won't be used as much, and like unused software, will suffer from bit rot. The best test cases/plans are the ones frequently used, referenced and have maintainer buy-in. Meaning, if the tests fail, the maintainer cares. I want to grow the library of tests we maintain and run, but hopefully grow in a manner and pace that we, as a community, can sustain.
With the test plans that Adam points to, I'm pretty confident in our ability to develop, discuss/debate and execute desktop and installation tests as a community. We've ironed out the kinks in the workflow, increased community engagement and developed good test plans as a result. My impression is we are ready for additional test areas.
That's what's exciting to me about the proventesters effort. As you can tell from recent (and old) devel@ list threads, testing proposed updates is important work that's needed, requested by package maintainers and well under-documentated. I don't worry as much that tests written for frequent proventester use will go stale given it's been a long-standing exposure in the project. Also, given the huge number of components in Fedora, there is room for just about every contributor to participate and carve out a niche. But which tests do we prioritize first, where do we write the tests, where to review+discuss them, how to run them etc... (more on this later).
For me, these are two separate (but related) efforts. TCMS is tool designed to address specific workflow/tracking needs. We also need to determine how to best to sustainably expand the test coverage we can offer to the project. We have a wiki-based "TCMS" now. It has met our needs for the current set of organized test efforts. It's not perfect, but the return on the investment has been huge. The questions I'd like to see answered in ticket#152, is (1) whether the wiki can continue to scale as our test management needs grow, and (2) what aspects of our wiki-based TCMS are good/bad?
Thanks, James
On Fri, 2010-12-03 at 23:33 +0100, Karol Cieśla wrote:
Hi, i agree with both of you ,so that rather than just sending e-mails i would like initiate something which could improve the current situation.
So the steps I see now look as follows: I stage
- someone could sketch out/outline/describe the functionality of
Wiki/TCMS we would like to have for the test cases (if possible also provide graphical form) -i.e table, page, buttons
Hurry and I discussed this topic yesterday on IRC. I suspect she'll take a similar approach, but the general idea will be to first identify what our QA community needs are when it comes to test documentation/results. For example, what works well, what isn't working well, where do we want to improve when it comes to our current test management.
- discuss with the whole community such form -pros,cons
As always, I expect the process will be transparent and there will be plenty of opportunity for anyone interested to get involved and help move things forward.
- reach out to people maintaining the Wiki and ask them if they can
create such form wit help of Wiki/TCMS and can sustain bigger number of test cases (i.e up to 300)
If the analysis shows that investing in wiki-based solution is the most sustainable option forward, there are several solutions available (http://www.mediawiki.org/wiki/Extension:Semantic_Forms and possibly http://www.mediawiki.org/wiki/Extension:StructuredInput). My impression with Semantic was that while it may scratch the wiki form-input itch for us, it's certainly not without a startup cost and I'm still unclear whether that's the right tool for the job. It's a fun experiment, but without knowing what where our current gaps are, and where we want to improve, it's hard to make a decision to invest in semantic-mediawiki over another actively maintained upstream project.
II stage
To ponder as the whole community the process of adding/creating/deleting/marking/prioritizing test cases.
While it would be an interesting discussion, I don't know if we'd be able to move forward with actionable work after a general community ponder session. I'd fear that would revolve too much around ponies [1]. I'm inclined to put emphasis on reviewing what we're doing now, understand why we're doing it, document the pro's con's and prioritize the MUSTHAVE features.
P.S -i can try to prepare for the middle of the next week some draft to point 1 -it will be something simple but reflecting the idea.
Always welcome additional hands to help move the discussion forward!
Thanks, James
W dniu 2010-12-03 17:36, James Laska pisze:
Just a few extra thoughts on the subject ...
On Fri, 2010-12-03 at 07:30 -0800, Adam Williamson wrote:
On Fri, 2010-12-03 at 16:06 +0100, xcieja wrote:
Hi, yes, you are right there are tests, but in my opinion they are in few different places under different categories.
That wasn't what I meant: I meant we already use the Wiki for the purposes you identified as an advantage of a TCMS (listing the tests that need to be performed in relation to some specific process, and whether they have already been performed, by whom, and with which result).
I think we could organise them better -i.e create test category and put all of them instead of many places.
We sure could, but we don't necessarily need a TCMS to do this. :) Note that we do try to keep them all within one Wiki namespace and we do use Wiki categories to organize some test cases.
Moreover, i just have taken a look briefly and i see there are round 100 test cases in total (please correct me if i am wrong).I think that for such project/system it is not enough at all.
We have big community, let`s assume everyone from QA create one test, we will have quite huge number of tests and obviously more faults detected before main release, less corrections after=better stability,usability-> better overall opinion.
Sure, we can always do with more test cases.
More test cases/plans would certainly change the conversation a bit. I think we all want to increase the value that the Fedora QA team can offer to the project. One way to increase our value is by improving our test coverage by way of test documentation (procedures, plans and cases). There are plenty of other ways ... but we can save those for other threads.
I've always been hesitant to add tests for the sake of adding tests. Test plans/cases are just like software. If the tests aren't addressing a priority issue, they won't be used as much, and like unused software, will suffer from bit rot. The best test cases/plans are the ones frequently used, referenced and have maintainer buy-in. Meaning, if the tests fail, the maintainer cares. I want to grow the library of tests we maintain and run, but hopefully grow in a manner and pace that we, as a community, can sustain.
With the test plans that Adam points to, I'm pretty confident in our ability to develop, discuss/debate and execute desktop and installation tests as a community. We've ironed out the kinks in the workflow, increased community engagement and developed good test plans as a result. My impression is we are ready for additional test areas.
That's what's exciting to me about the proventesters effort. As you can tell from recent (and old) devel@ list threads, testing proposed updates is important work that's needed, requested by package maintainers and well under-documentated. I don't worry as much that tests written for frequent proventester use will go stale given it's been a long-standing exposure in the project. Also, given the huge number of components in Fedora, there is room for just about every contributor to participate and carve out a niche. But which tests do we prioritize first, where do we write the tests, where to review+discuss them, how to run them etc... (more on this later).
For me, these are two separate (but related) efforts. TCMS is tool designed to address specific workflow/tracking needs. We also need to determine how to best to sustainably expand the test coverage we can offer to the project. We have a wiki-based "TCMS" now. It has met our needs for the current set of organized test efforts. It's not perfect, but the return on the investment has been huge. The questions I'd like to see answered in ticket#152, is (1) whether the wiki can continue to scale as our test management needs grow, and (2) what aspects of our wiki-based TCMS are good/bad?
Thanks, James
#152: Test Cases Management --------------------------+------------------------------------------------- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------------+------------------------------------------------- Comment (by jlaska):
I love the idea of doing a test event with a pilot instance of nitrate/TCMS. It was fun doing that with the semantic-mediawiki, and I think we learn a lot with that approach. We've all talked about TCMS and how best to integrate it with our workflow. There are definite shortcomings to our current wiki-based approach that limit our ability to growth, but there are plenty of positive lessons learned that I think we should recognize and prioritize.
Hurry: Would you be willing to start drafting some wiki content so we can capture this? Either a requirements or technology comparison document? I don't want this to be a free-for-all 'I want a pony' list of features we want of a TCMS solution. But perhaps starting with documenting the pro's and con's with our current wiki-based workflow? The goal for me would be to identify what our MUST-HAVE and NICE-TO-HAVE's are from any solution. Ideally, we can use that to determine the best route forward (whether it's filing bugs/tickets/rfe's against TCMS, or something else).
#152: Test Cases Management --------------------------+------------------------------------------------- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------------+------------------------------------------------- Comment (by rhe):
I've drafted [https://fedoraproject.org/wiki/Rhe/tcms_requirements_proposal a page] for your reference. So far I listed the use cases, pros and cons in wiki-based tcms and nitrate tcms, separately. Feel free to comment/discuss them. Thanks!
#152: Test Cases Management --------------------------+------------------------------------------------- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------------+------------------------------------------------- Comment (by aalam):
TCMS is really good idea for Test Creation and Manage Test plan than wiki.
Is it necessary to have permission other than view for anonymous user, like Translation team allow only FAS account user to translation on http://translate.fedoraproject.org?
#152: Test Cases Management --------------------------+------------------------------------------------- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------------+------------------------------------------------- Comment (by rhe):
Replying to [comment:4 aalam]:
TCMS is really good idea for Test Creation and Manage Test plan than
wiki.
Is it necessary to have permission other than view for anonymous user,
like Translation team allow only FAS account user to translation on http://translate.fedoraproject.org?
I personally consider it as a NICE-TO-HAVE, since it's one big advantage for wiki-based system to allow anonymous user editing pages with certain name space, like 'test day:' and 'Test_Results:'. So far the nitrate system only has read permission for anonymous user, but this can be customized if we review the requirement and decide to use it.
#152: Test Cases Management --------------------------+------------------------------------------------- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------------+------------------------------------------------- Comment (by xcieja):
Hi, I have found following table:[https://fedoraproject.org/wiki/User:Johannbg/Draft/QA/Test_Days/Features/EXT...]. In my opinion it can serve as draft of functionality we would like to have or we should have. Obviously i would add to the table additional column and introduced a rule that each test case has to have corresponding name like : Netoworking 1, Networking 2 and so on...
Now, i think the question is if we can do it via Wiki or TCMS.
#152: Test Cases Management --------------------------+------------------------------------------------- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------------+------------------------------------------------- Comment (by rhe):
Replying to [comment:6 xcieja]:
Hi, I have found following
table:[https://fedoraproject.org/wiki/User:Johannbg/Draft/QA/Test_Days/Features/EXT...].
In my opinion it can serve as draft of functionality we would like to
have or we should have. Obviously i would add to the table additional column and introduced a rule that each test case has to have corresponding name like : Netoworking 1, Networking 2 and so on...
Now, i think the question is if we can do it via Wiki or TCMS.
Do you mean [https://fedoraproject.org/wiki/User_talk:Johannbg/Draft/QA/Test_Days/Feature... this example]? AFAIK, for wiki, one have to manually rename the page link when needed. For Nitrate TCMS, you can use 'alias' and 'tag' attitude of each test case to correspond them.
#152: Test Cases Management --------------------------+------------------------------------------------- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------------+------------------------------------------------- Comment (by rhe):
The [https://fedoraproject.org/wiki/Rhe/tcms_requirements_proposal proposal page] has been updated and written in the table format.
To further identify this issue, the following items will be analysed in details one by one:
- Identify our current test workflows (or use cases) - Identify problems with our current system and workflows - e.g. what would we like to do but are difficult/impossible with wiki - Objectively compare feature sets of both tools in relation to the important workflows - Identify+prioritize any feature gaps we'd need from nitrate in order to make a smooth transition - Document a plan/roadmap for the transition
Where each item will be documented in a separate page, and the use cases page for the 1st one has been drafted:
- https://fedoraproject.org/wiki/Rhe/tcms_use_cases
Feel free to add comments or tell some uncovered use cases. Thanks.
#152: Test Cases Management --------------------------+------------------------------------------------- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------------+------------------------------------------------- Comment (by adamwill):
Replying to [comment:6 xcieja]:
Hi, I have found following
table:[https://fedoraproject.org/wiki/User:Johannbg/Draft/QA/Test_Days/Features/EXT...].
In my opinion it can serve as draft of functionality we would like to
have or we should have. Obviously i would add to the table additional column and introduced a rule that each test case has to have corresponding name like : Netoworking 1, Networking 2 and so on...
Now, i think the question is if we can do it via Wiki or TCMS.
Of course we can do it via the Wiki, as that example is done using the Wiki =) we already use a similar table format for test case results for most test days, and the validation testing. The drawbacks of using the wiki to track results are mostly in cleanliness (it's too free-form so users can mess up the tables), consistency (we don't really store all the results for any given test case in one place, instead we store results for *some particular event* which referenced those test cases), flexibility (see what rhe wrote) and comprehensiveness (a more sophisticated system could probably include more detail for each result, perhaps automatically retrieved from the user).
#152: Test Cases Management --------------------------+------------------------------------------------- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------------+------------------------------------------------- Comment (by rhe):
I further updated the use cases page:
- https://fedoraproject.org/wiki/Rhe/tcms_use_cases
And based on these, the use cases between the two tcms are compared in the link below:
- https://fedoraproject.org/wiki/Rhe/tcms_Comparison
Feel free to add comments. Thanks!
#152: Test Cases Management --------------------------+------------------------------------------------- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------------+------------------------------------------------- Comment (by xcieja):
Hi, I have few questions regarding to red fields by Wiki side: 1) '''Review status - manually add reviewer''' -does it mean that if create test case and perform it, thereafter i want to allow others to see the result i need to add particulars users to "list"" in order to give them rigths to see the status ???
2) '''Result summary/report - "-"''' -is it not possible at all to generate the report ???
#152: Test Cases Management --------------------------+------------------------------------------------- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------------+------------------------------------------------- Comment (by rhe):
Replying to [comment:11 xcieja]:
Hi, I have few questions regarding to red fields by Wiki side:
- '''Review status - manually add reviewer''' -does it mean that if
create test case and perform it, thereafter i want to allow others to see the result i need to add particulars users to "list"" in order to give them rigths to see the status ???
No, even the anonymous user can view the test results on wiki. Here I mean that when create a new test plan(or test case) and wait for someone's review, there's no status as proposed or reviewed, one has to add a 'draft' sign in the page, and after review, the 'draft' sign is removed and the reviewer's name is added manually. But in Nitrate, the status of a new test plan is set as 'proposed', and after reviewed, its status will be changed to 'confirmed' AFAIK. It's easier for tracking and managing test plans/cases.
- '''Result summary/report - "-"''' -is it not possible at all to
generate the report ??? So far, we use the [http://fedoraproject.org/wiki/QA/SOP_Release_Validation_Test_Event#Report_an... curl command] to extract bug report. I don't think wiki can generate report unless plugins are added.
#152: Test Cases Management --------------------------+------------------------------------------------- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------------+------------------------------------------------- Comment (by xcieja):
After familiarize myself with above comparison, better fits to my own preferences Wiki. It seems there are more useful features, sometimes not availible directly but via plugins, even though it looks more user friendly.
My score: 1:0 for Wiki
#152: Test Cases Management --------------------------+------------------------------------------------- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------------+------------------------------------------------- Comment (by rhe):
Replying to [comment:13 xcieja]:
After familiarize myself with above comparison, better fits to my own
preferences Wiki.
It seems there are more useful features, sometimes not availible
directly but via plugins, even though it looks more user friendly.
My score: 1:0 for Wiki
Hehe, thanks for voting for wiki. Note that the comparison is based on the use cases on wiki, so most of the compared features so far are already contained in the wiki and that's why the green fields in the wiki side are more than the nitrate side. But with deeper research on Nitrate TCMS, I'll find more features/advantages in it and add more features for comparison. Also, the red fields in Nitrate side are not permanent, they can be resolved by the tcms team if we decide those features as MUST-HAVE.
#152: Test Cases Management --------------------------+------------------------------------------------- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------------+------------------------------------------------- Comment (by rhe):
[https://fedoraproject.org/wiki/Rhe/tcms_Comparison wiki vs nitrate page] is updated again. Last time there were some unknown fields in nitrate side. By doing some tests on it, I verified several of them and added some new features. Please refer to it. So far, I think the general use cases have been covered and converted to related features for comparison.
#152: Test Cases Management --------------------------+------------------------------------------------- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------------+------------------------------------------------- Comment (by rhe):
Replying to [comment:8 rhe]:
The [https://fedoraproject.org/wiki/Rhe/tcms_requirements_proposal
proposal page] has been updated and written in the table format.
To further identify this issue, the following items will be analysed in
details one by one:
- Identify our current test workflows (or use cases)
- Objectively compare feature sets of both tools in relation to the important workflows
Since I called for the final review of these two item last week, from this week, I will move to next step and the pages are moved to formal links: * https://fedoraproject.org/wiki/Tcms_use_cases * https://fedoraproject.org/wiki/Tcms_Comparison
But feedback is still welcomed to improve them.
- Identify problems with our current system and workflows - e.g. what would we like to do but are difficult/impossible with wiki
- Identify+prioritize any feature gaps we'd need from nitrate in order to make a smooth transition
These two issues can be considered together. This week I will focus on identifying Must-Have and Nice-to-Have features in the [https://fedoraproject.org/wiki/Tcms_Comparison comparison table], and then summarize in another page.
Meanwhile, I will start the transition preparation work to write scripts for conversing xml files exported from wiki to files whose format can be imported to nitrate. Feel free to add comments/ideas/suggestions. Thanks.
#152: Consider alternative test case management systems --------------------+------------------------------------------------------- Reporter: rhe | Owner: rhe Type: task | Status: new Priority: major | Milestone: Component: Wiki | Version: Resolution: | Keywords: retrospective --------------------+------------------------------------------------------- Changes (by adamwill):
* summary: Test Cases Management => Consider alternative test case management systems * type: enhancement => task * milestone: Fedora 16 =>
Comment:
rhe continued to work on this - the use cases and comparison pages linked above are stuffed with info and very useful - but we did not come to a final decision.
I'm going to leave this open as it's still a topic worth considering, but it doesn't need to be tied to a specific release.
#152: Consider alternative test case management systems --------------------+--------------------------- Reporter: rhe | Owner: rhe Type: task | Status: new Priority: major | Milestone: Component: Wiki | Version: Resolution: | Keywords: retrospective Blocked By: | Blocking: --------------------+--------------------------- Changes (by adamwill):
* cc: test@… (added)
Comment:
...and we're still considering it. :)
Nitrate/TCMS is still a contender, the other major contender we have ATM is Moztrap: https://github.com/mozilla/moztrap . Another contender is Ubuntu's QATracker, which is more similar in design to the wiki system (I suspect they actually started off by more or less cloning our wiki design...), but has some significant limitations (it doesn't do environments, so they make each arch a separate 'build', the API is limited, and tflink isn't a big fan of the architecture in general).
In the meantime, I made some significant enhancements to the wiki system, which I call Wikitcms:
https://fedoraproject.org/wiki/Wikitcms
and built python-wikitcms and relval on top of it: https://www.happyassassin.net/wikitcms/
which address some of the deficiencies of the wiki system. We do really need to look at making Moztrap or Nitrate/TCMS compatible with our requirements, though.
-
Adam Williamson -
fedora-badges -
James Laska -
Karol Cieśla