Hi!
Currently the Standard Test Interface does not define how to distinguish test failure from an infrastructure error. When ansible playbook is run it usually does some environment preparation and only then executes tests. Upon a problem encountered during provisioning or configuration of the test environment ansible returns non-zero exit code but there is not an easy way how to detect such error from a successful test execution of test which failed (returns non-zero exit code as well).
Let's make this part of the specification more clear so that tools can detect infrastructure errors and report them as such to the end user. This will help to direct problems (errors versus failures) to the right audience (infrastructure staff versus developer/tester).
See the following ticket for proposed solution and share your thoughts there:
https://pagure.io/fedora-ci/general/issue/18
Thanks.
psss...