Hi!
Currently the Standard Test Interface does not define how to
distinguish test failure from an infrastructure error. When
ansible playbook is run it usually does some environment
preparation and only then executes tests. Upon a problem
encountered during provisioning or configuration of the test
environment ansible returns non-zero exit code but there is not an
easy way how to detect such error from a successful test execution
of test which failed (returns non-zero exit code as well).
Let's make this part of the specification more clear so that tools
can detect infrastructure errors and report them as such to the
end user. This will help to direct problems (errors versus
failures) to the right audience (infrastructure staff versus
developer/tester).
See the following ticket for proposed solution and share your
thoughts there:
https://pagure.io/fedora-ci/general/issue/18
Thanks.
psss...
