Hi!
Currently the Standard Test Interface does not define how to distinguish test failure from an infrastructure error. When ansible playbook is run it usually does some environment preparation and only then executes tests. Upon a problem encountered during provisioning or configuration of the test environment ansible returns non-zero exit code but there is not an easy way how to detect such error from a successful test execution of test which failed (returns non-zero exit code as well).
Let's make this part of the specification more clear so that tools can detect infrastructure errors and report them as such to the end user. This will help to direct problems (errors versus failures) to the right audience (infrastructure staff versus developer/tester).
See the following ticket for proposed solution and share your thoughts there:
https://pagure.io/fedora-ci/general/issue/18
Thanks.
psss...
Hi,
here's an update for those who were not involved in the issue: After a bit longer discussion we've come to the following agreement:
* Introduce a new "results.yml" file which should contain results of individual tests in a defined format
* Ansible playbook should return non-zero status on infrastructure error and not on test failures
This should allow the pipeline to clearly detect infrastructure errors and report them accordingly. See the ticket for more details and the list of proposed changes:
* https://pagure.io/fedora-ci/general/issue/18
The solution is introducing new requirement for tests which are not using Standard Test Interface, so please review this and let us know if you have any concerns. Thanks.
psss...
On Tue, 11 Dec 2018 at 14:05, Petr Šplíchal psplicha@redhat.com wrote:
Hi!
Currently the Standard Test Interface does not define how to distinguish test failure from an infrastructure error. When ansible playbook is run it usually does some environment preparation and only then executes tests. Upon a problem encountered during provisioning or configuration of the test environment ansible returns non-zero exit code but there is not an easy way how to detect such error from a successful test execution of test which failed (returns non-zero exit code as well).
Let's make this part of the specification more clear so that tools can detect infrastructure errors and report them as such to the end user. This will help to direct problems (errors versus failures) to the right audience (infrastructure staff versus developer/tester).
See the following ticket for proposed solution and share your thoughts there:
https://pagure.io/fedora-ci/general/issue/18Thanks.
psss...
On Thu, 7 Feb 2019 at 12:24, Petr Šplíchal psplicha@redhat.com wrote:
Hi,
here's an update for those who were not involved in the issue: After a bit longer discussion we've come to the following agreement:
Introduce a new "results.yml" file which should contain results of individual tests in a defined format
Ansible playbook should return non-zero status on infrastructure error and not on test failures
This should allow the pipeline to clearly detect infrastructure errors and report them accordingly. See the ticket for more details and the list of proposed changes:
The solution is introducing new requirement for tests which are not using Standard Test Interface, so please review this and let us know if you have any concerns. Thanks.
This should be "introducing a new requirement for tests which are not using Standard Test Roles". Sorry for the confusion.
psss...
On Tue, 11 Dec 2018 at 14:05, Petr Šplíchal psplicha@redhat.com wrote:
Hi!
Currently the Standard Test Interface does not define how to distinguish test failure from an infrastructure error. When ansible playbook is run it usually does some environment preparation and only then executes tests. Upon a problem encountered during provisioning or configuration of the test environment ansible returns non-zero exit code but there is not an easy way how to detect such error from a successful test execution of test which failed (returns non-zero exit code as well).
Let's make this part of the specification more clear so that tools can detect infrastructure errors and report them as such to the end user. This will help to direct problems (errors versus failures) to the right audience (infrastructure staff versus developer/tester).
See the following ticket for proposed solution and share your thoughts there:
https://pagure.io/fedora-ci/general/issue/18Thanks.
psss...
Hi,
an update of the Standard Test Interface based on the changes agreed in [1] is now available for review:
https://pagure.io/fedora-ci/docs/pull-request/5
Review is appreciated because this change is introducing a backward incompatible behaviour. Please, share your thoughts and concerns directly in the pull request. Thanks.
psss...
[1] https://pagure.io/fedora-ci/general/issue/18
On Thu, 7 Feb 2019 at 12:24, Petr Šplíchal psplicha@redhat.com wrote:
Hi,
here's an update for those who were not involved in the issue: After a bit longer discussion we've come to the following agreement:
Introduce a new "results.yml" file which should contain results of individual tests in a defined format
Ansible playbook should return non-zero status on infrastructure error and not on test failures
This should allow the pipeline to clearly detect infrastructure errors and report them accordingly. See the ticket for more details and the list of proposed changes:
The solution is introducing new requirement for tests which are not using Standard Test Roles, so please review this and let us know if you have any concerns. Thanks.
psss...
On Tue, 11 Dec 2018 at 14:05, Petr Šplíchal psplicha@redhat.com wrote:
Hi!
Currently the Standard Test Interface does not define how to distinguish test failure from an infrastructure error. When ansible playbook is run it usually does some environment preparation and only then executes tests. Upon a problem encountered during provisioning or configuration of the test environment ansible returns non-zero exit code but there is not an easy way how to detect such error from a successful test execution of test which failed (returns non-zero exit code as well).
Let's make this part of the specification more clear so that tools can detect infrastructure errors and report them as such to the end user. This will help to direct problems (errors versus failures) to the right audience (infrastructure staff versus developer/tester).
See the following ticket for proposed solution and share your thoughts there:
https://pagure.io/fedora-ci/general/issue/18Thanks.
psss...
Hi!
The proposed changes have been implemented in the standard-test-roles and will soon land in updates:
https://bodhi.fedoraproject.org/updates/FEDORA-MODULAR-2019-ae9c582836 https://bodhi.fedoraproject.org/updates/FEDORA-MODULAR-2019-c28ebce16b
Let us know if you encounter any problems. Thanks.
psss...
On Thu, 28 Feb 2019 at 15:52, Petr Šplíchal psplicha@redhat.com wrote:
Hi,
an update of the Standard Test Interface based on the changes agreed in [1] is now available for review:
https://pagure.io/fedora-ci/docs/pull-request/5Review is appreciated because this change is introducing a backward incompatible behaviour. Please, share your thoughts and concerns directly in the pull request. Thanks.
psss...
[1] https://pagure.io/fedora-ci/general/issue/18
On Thu, 7 Feb 2019 at 12:24, Petr Šplíchal psplicha@redhat.com wrote:
Hi,
here's an update for those who were not involved in the issue: After a bit longer discussion we've come to the following agreement:
Introduce a new "results.yml" file which should contain results of individual tests in a defined format
Ansible playbook should return non-zero status on infrastructure error and not on test failures
This should allow the pipeline to clearly detect infrastructure errors and report them accordingly. See the ticket for more details and the list of proposed changes:
The solution is introducing new requirement for tests which are not using Standard Test Roles, so please review this and let us know if you have any concerns. Thanks.
psss...
On Tue, 11 Dec 2018 at 14:05, Petr Šplíchal psplicha@redhat.com wrote:
Hi!
Currently the Standard Test Interface does not define how to distinguish test failure from an infrastructure error. When ansible playbook is run it usually does some environment preparation and only then executes tests. Upon a problem encountered during provisioning or configuration of the test environment ansible returns non-zero exit code but there is not an easy way how to detect such error from a successful test execution of test which failed (returns non-zero exit code as well).
Let's make this part of the specification more clear so that tools can detect infrastructure errors and report them as such to the end user. This will help to direct problems (errors versus failures) to the right audience (infrastructure staff versus developer/tester).
See the following ticket for proposed solution and share your thoughts there:
https://pagure.io/fedora-ci/general/issue/18Thanks.
psss...
-
Petr Šplíchal