Today, I have pushed (to master branch) a patch that adds an easy way for testing exception handling (and bug reporting) in Anaconda. It utilizes unix signals and can be fired by sending SIGUSR1 signal to the main anaconda process (there are usually two processes, main is the one with the lower PID). Probably the easiest way to do that is running this command from tty2 (or window #2 in tmux):
$ kill -USR1 `cat /var/run/anaconda.pid`
Please note that it may need some additional interaction with the GUI to get the "An unhandled exception occured..." window, as unix signals are processed only when Gtk.main loop runs some (re)action.
The underlaying code raises an exception in a separate thread and with a stack containing non-ascii characters (worst case scenario designed from the bugs in exception handling seen so far). It also produces a unique stack trace so that new bug is filed every time exception handling and bug reporting in anaconda is tested.
These changes apply to the next build of anaconda (IIRC anaconda-18.9-1).
On Mon, 17 Sep 2012 18:36:21 +0200 Vratislav Podzimek vpodzime@redhat.com wrote:
<snip>
The underlaying code raises an exception in a separate thread and with a stack containing non-ascii characters (worst case scenario designed from the bugs in exception handling seen so far). It also produces a unique stack trace so that new bug is filed every time exception handling and bug reporting in anaconda is tested.
Isn't that going to greatly increase the number of bugs filed against anaconda and work against the duplicate detection in ABRT/libreport? I didn't think that there were that many problems with improperly detected duplicates.
These changes apply to the next build of anaconda (IIRC anaconda-18.9-1).
Any guesses on when that build will be ready?
Tim
On Mon, 17 Sep 2012 18:36:21 +0200 Vratislav Podzimek vpodzime@redhat.com wrote:
<snip>
The underlaying code raises an exception in a separate thread and with a stack containing non-ascii characters (worst case scenario designed from the bugs in exception handling seen so far). It also produces a unique stack trace so that new bug is filed every time exception handling and bug reporting in anaconda is tested.
Isn't that going to greatly increase the number of bugs filed against anaconda and work against the duplicate detection in ABRT/libreport? I didn't think that there were that many problems with improperly detected duplicates.
Hey, this is just for us manually triggering exceptions, for QA purposes. It replaces 'traceback' boot option that was available in F17. Of course it is expected that we close such fake reports afterwards.
Thanks, Vratislav, for your efforts. I'll adjust our test cases.
On Mon, 2012-09-17 at 10:47 -0600, Tim Flink wrote:
On Mon, 17 Sep 2012 18:36:21 +0200 Vratislav Podzimek vpodzime@redhat.com wrote:
<snip>
The underlaying code raises an exception in a separate thread and with a stack containing non-ascii characters (worst case scenario designed from the bugs in exception handling seen so far). It also produces a unique stack trace so that new bug is filed every time exception handling and bug reporting in anaconda is tested.
Isn't that going to greatly increase the number of bugs filed against anaconda and work against the duplicate detection in ABRT/libreport? I didn't think that there were that many problems with improperly detected duplicates.
This is up to you, guys. Summary of such bugs will state: "NOTABUG: testing exception handling" and if not you, we will definitely close such bugs instantly. Problem is that there were bugs, that could have been hit only when new bugreport have been created. Marking as duplicate doesn't test adding attachments and so on.
These changes apply to the next build of anaconda (IIRC anaconda-18.9-1).
Any guesses on when that build will be ready?
I think it will be ready soon, probably this week.
Today, I have pushed (to master branch) a patch that adds an easy way for testing exception handling (and bug reporting) in Anaconda. It utilizes unix signals and can be fired by sending SIGUSR1 signal to the main anaconda process (there are usually two processes, main is the one with the lower PID). Probably the easiest way to do that is running this command from tty2 (or window #2 in tmux):
$ kill -USR1 `cat /var/run/anaconda.pid`
I have updated our test case with the new instructions: https://fedoraproject.org/wiki/QA:Testcase_Anaconda_save_traceback_to_bugzil...
We have more test cases regarding bug reporting from Anaconda:
https://fedoraproject.org/wiki/QA:Testcase_Anaconda_save_traceback_to_disk According to developers in #anaconda, every time there is an exception it is automatically written to anaconda-tb-* file. Jiri Moskovcak says there are no plans to add 'save to disk' option to libreport.
https://fedoraproject.org/wiki/QA:Testcase_Anaconda_save_traceback_to_remote... Jiri Moskovcak says there are no plans to add 'save to remote system' option to libreport.
These test cases were created because in Fedora 17 Anaconda/libreport supported these options. Our release criteria also say: " The installer must be able to report failures to Bugzilla and local disk, with appropriate information included " Of course we have to change them, otherwise Alpha couldn't have been released (unless we consider creating anaconda-tb-* file as "report to local disk", which we very well might do).
My opinion is that we should merge "save to bugzilla" and "save to disk" test cases into one - we will check the presence of anaconda-tb-* file and try to report to bugzilla. We should remove "save to remote system" test case, because it is no longer supported and we have no criteria related to it. Our existing criterion about "reporting to local disk" doesn't need adjustments, if we agree that the automatic creation of anaconda-tb-* file satisfies it.
Last but not least, we should consider removing https://fedoraproject.org/wiki/QA:Testcase_Anaconda_traceback_debug_mode because I don't see how that helps our users. That is mainly useful for developers, but no user will ever use that, they will use updates.img at best. There is no reason for us to test it.
Comments welcome.
On Wed, 2012-09-19 at 13:31 -0400, Kamil Paral wrote:
Today, I have pushed (to master branch) a patch that adds an easy way for testing exception handling (and bug reporting) in Anaconda. It utilizes unix signals and can be fired by sending SIGUSR1 signal to the main anaconda process (there are usually two processes, main is the one with the lower PID). Probably the easiest way to do that is running this command from tty2 (or window #2 in tmux):
$ kill -USR1 `cat /var/run/anaconda.pid`
I have updated our test case with the new instructions: https://fedoraproject.org/wiki/QA:Testcase_Anaconda_save_traceback_to_bugzil...
We have more test cases regarding bug reporting from Anaconda:
https://fedoraproject.org/wiki/QA:Testcase_Anaconda_save_traceback_to_disk According to developers in #anaconda, every time there is an exception it is automatically written to anaconda-tb-* file. Jiri Moskovcak says there are no plans to add 'save to disk' option to libreport.
https://fedoraproject.org/wiki/QA:Testcase_Anaconda_save_traceback_to_remote... Jiri Moskovcak says there are no plans to add 'save to remote system' option to libreport.
If this isn't supposed to come back into libreport, we should just drop the test case, or replace it with one to ensure the tb file is accessible in /tmp.
These test cases were created because in Fedora 17 Anaconda/libreport supported these options. Our release criteria also say: " The installer must be able to report failures to Bugzilla and local disk, with appropriate information included " Of course we have to change them, otherwise Alpha couldn't have been released (unless we consider creating anaconda-tb-* file as "report to local disk", which we very well might do).
My opinion is that we should merge "save to bugzilla" and "save to disk" test cases into one - we will check the presence of anaconda-tb-* file and try to report to bugzilla. We should remove "save to remote system" test case, because it is no longer supported and we have no criteria related to it. Our existing criterion about "reporting to local disk" doesn't need adjustments, if we agree that the automatic creation of anaconda-tb-* file satisfies it.
I'd say that's a fine plan except we should change the 'report to local disk' wording in the criterion. That's certainly not how we'd write it if we were starting with a libreport that didn't have a 'save crash report to local disk' option, so we should change it.
Last but not least, we should consider removing https://fedoraproject.org/wiki/QA:Testcase_Anaconda_traceback_debug_mode because I don't see how that helps our users. That is mainly useful for developers, but no user will ever use that, they will use updates.img at best. There is no reason for us to test it.
I usually want to at least recall why we had something in the criteria in the *first* place before we take it out, but aside from that I have no concrete objections. Anyone remember why we put that in there? anaconda folks, is it something you consider critical?
On Wed, 2012-09-19 at 21:32 -0700, Adam Williamson wrote:
Last but not least, we should consider removing https://fedoraproject.org/wiki/QA:Testcase_Anaconda_traceback_debug_mode because I don't see how that helps our users. That is mainly useful for developers, but no user will ever use that, they will use updates.img at best. There is no reason for us to test it.
I usually want to at least recall why we had something in the criteria in the *first* place before we take it out, but aside from that I have no concrete objections. Anyone remember why we put that in there? anaconda folks, is it something you consider critical?
We were discussing if we even want the Debug button in the window (because, you know, it can lead users to hell) and it appeared, there were some cases, when an anaconda developer guided some user (online) through the debugging session to get some additional info about the crash. That's why we complicated our lives by leaving the button there and trying to hide it from the not-so-curious users. It would be great if that functionality was tested, but I believe it is not so critical.
Last but not least, we should consider removing https://fedoraproject.org/wiki/QA:Testcase_Anaconda_traceback_debug_mode because I don't see how that helps our users. That is mainly useful for developers, but no user will ever use that, they will use updates.img at best. There is no reason for us to test it.
I usually want to at least recall why we had something in the criteria in the *first* place before we take it out, but aside from that I have no concrete objections. Anyone remember why we put that in there? anaconda folks, is it something you consider critical?
We were discussing if we even want the Debug button in the window (because, you know, it can lead users to hell) and it appeared, there were some cases, when an anaconda developer guided some user (online) through the debugging session to get some additional info about the crash. That's why we complicated our lives by leaving the button there and trying to hide it from the not-so-curious users. It would be great if that functionality was tested, but I believe it is not so critical.
Ah, Vratislav replied before I even managed to talk to him :-)
It seems that it is useful to keep this test case in the matrix then, with no milestone assigned, as it already is. I'll update the test case with the new way how to trigger the crash.
I have updated our test case with the new instructions: https://fedoraproject.org/wiki/QA:Testcase_Anaconda_save_traceback_to_bugzil...
We have more test cases regarding bug reporting from Anaconda:
https://fedoraproject.org/wiki/QA:Testcase_Anaconda_save_traceback_to_disk According to developers in #anaconda, every time there is an exception it is automatically written to anaconda-tb-* file. Jiri Moskovcak says there are no plans to add 'save to disk' option to libreport.
https://fedoraproject.org/wiki/QA:Testcase_Anaconda_save_traceback_to_remote... Jiri Moskovcak says there are no plans to add 'save to remote system' option to libreport.
If this isn't supposed to come back into libreport, we should just drop the test case, or replace it with one to ensure the tb file is accessible in /tmp.
Done. "save to disk" and "save to remote system" test cases dropped, traceback file checking added to "save to bugzilla" test case.
I'd say that's a fine plan except we should change the 'report to local disk' wording in the criterion. That's certainly not how we'd write it if we were starting with a libreport that didn't have a 'save crash report to local disk' option, so we should change it.
Old criterion: " The installer must be able to report failures to Bugzilla and local disk, with appropriate information included "
Proposed new criterion: " The installer must be able to save failure information to local disk and report it to Bugzilla "
Last but not least, we should consider removing https://fedoraproject.org/wiki/QA:Testcase_Anaconda_traceback_debug_mode because I don't see how that helps our users. That is mainly useful for developers, but no user will ever use that, they will use updates.img at best. There is no reason for us to test it.
I usually want to at least recall why we had something in the criteria in the *first* place before we take it out, but aside from that I have no concrete objections. Anyone remember why we put that in there? anaconda folks, is it something you consider critical?
I think it was created somewhat organically [1] - there was a button in the installer, so we wrote a test case for it.
I'll talk to anaconda developers and ask them for more details about this feature.
On Thu, 2012-09-20 at 05:05 -0400, Kamil Paral wrote:
Last but not least, we should consider removing https://fedoraproject.org/wiki/QA:Testcase_Anaconda_traceback_debug_mode because I don't see how that helps our users. That is mainly useful for developers, but no user will ever use that, they will use updates.img at best. There is no reason for us to test it.
I usually want to at least recall why we had something in the criteria in the *first* place before we take it out, but aside from that I have no concrete objections. Anyone remember why we put that in there? anaconda folks, is it something you consider critical?
I think it was created somewhat organically [1] - there was a button in the installer, so we wrote a test case for it.
I'll talk to anaconda developers and ask them for more details about this feature.
Oh right, sorry, brain dead: I was thinking of the criteria, not test cases, for some reason.
For test cases, I reckon that as long as a test case is still valid and testing something that may be useful in some scenario, there's no profit in dropping it. But if it no longer relates to a criterion, it should be made an 'optional' test - i.e. it should have no release phase (Alpha, Beta, Final) in the column on the left of the table. Test cases with no associated release are optional, we don't have to do them, but if we have extra time, we can do them.
If we ever get lots and lots of optional test cases it may be worth making sure they're all at the bottom of the tables or separating them out some other way, but for now there's only a few.
If we ever get lots and lots of optional test cases it may be worth making sure they're all at the bottom of the tables or separating them out some other way, but for now there's only a few.
When talking about it, I wondered several times if there is any particular order in these test cases (except for Test Area column): https://fedoraproject.org/wiki/QA:Fedora_18_Install_Results_Template#General...
If nobody minds, I'd like to re-shuffle them: 1. first ordering key is Test Area 2. second ordering key is Release Level 3. if there are no Alpha test cases for a specific Test Area, then we put the whole Test Area below Test Areas that contain some Alpha tests 4. same as above for Beta 5. empty Release Level test cases go last
So, the majority of test cases will stay in the same sequence as they are now, just e.g. "Memory Test" goes almost to the bottom, "Install Repository" goes below "Partitioning" (because there is no Alpha test case in Install Repository), and so on.
I know the test cases can be sorted manually. Still I think the default ordering should be better.
If we ever get lots and lots of optional test cases it may be worth making sure they're all at the bottom of the tables or separating them out some other way, but for now there's only a few.
When talking about it, I wondered several times if there is any particular order in these test cases (except for Test Area column): https://fedoraproject.org/wiki/QA:Fedora_18_Install_Results_Template#General...
If nobody minds, I'd like to re-shuffle them:
- first ordering key is Test Area
- second ordering key is Release Level
- if there are no Alpha test cases for a specific Test Area, then we
put the whole Test Area below Test Areas that contain some Alpha tests 4. same as above for Beta 5. empty Release Level test cases go last
So, the majority of test cases will stay in the same sequence as they are now, just e.g. "Memory Test" goes almost to the bottom, "Install Repository" goes below "Partitioning" (because there is no Alpha test case in Install Repository), and so on.
I know the test cases can be sorted manually. Still I think the default ordering should be better.
You know what happens if nobody replies, right? Stuff gets done!
I have changed the default test case ordering. Hopefully it's a bit better. It looks better to me.
On 09/20/2012 04:32 AM, Adam Williamson wrote:
Last but not least, we should consider removing https://fedoraproject.org/wiki/QA:Testcase_Anaconda_traceback_debug_mode because I don't see how that helps our users. That is mainly useful for developers, but no user will ever use that, they will use updates.img at best. There is no reason for us to test it.
I usually want to at least recall why we had something in the criteria in the*first* place before we take it out, but aside from that I have no concrete objections. Anyone remember why we put that in there?
I think this is the test case that was supposed to test anaconda failure and handling mode along with basically is the path from failure to reporting to bugzilla working for the end user.
Btw do we archive ( move ) those wiki pages somewhere under the QA namespace when they are deprecated?
JBG
On Thu, 2012-09-20 at 05:46 -0400, Kamil Paral wrote:
Btw do we archive ( move ) those wiki pages somewhere under the QA namespace when they are deprecated?
Good question. I don't know. I added a warning box to the page and removed it from the test matrix skeleton, that should work. It's still reachable through categories.
There's a category for them:
https://fedoraproject.org/wiki/Category:Obsolete_Test_Cases
When I'm obsoleting a test case I usually stick a {{admon=/note at the top explaining why and since when it's obsolete, e.g. https://fedoraproject.org/wiki/QA:Testcase_install_repository_NFS_default .
There's a category for them:
Thanks, modified.