Lars from the standard-test-roles [0] project is using our test specification for invoking tests. He brought up a good point (I paraphrase):
Many tests are destructive in that they change the state of the VM or container that they are running in. However when using the Ansible Dynamic Inventory means the entire tests.yml playbook is run against the same VM or Docker image started from the test subject.
I would suggest our spec is amended in section "1.7.2 Invocation" to have the first sentence replaced with:
"The testing system MUST run each playbook matching the glob tests/tests*.yml in the dist-git repo. Each of these files constitutes a test suite. Each test suite is invoked independently by the testing system as follows."
What do you think?
Cheers,
Stef
On 19.09.2017 10:33, Stef Walter wrote:
Lars from the standard-test-roles [0] project is using our test specification for invoking tests. He brought up a good point (I paraphrase):
Many tests are destructive in that they change the state of the VM or container that they are running in. However when using the Ansible Dynamic Inventory means the entire tests.yml playbook is run against the same VM or Docker image started from the test subject.
I would suggest our spec is amended in section "1.7.2 Invocation" to have the first sentence replaced with:
"The testing system MUST run each playbook matching the glob tests/tests*.yml in the dist-git repo. Each of these files constitutes a test suite. Each test suite is invoked independently by the testing system as follows."
I have made this change to the invocation spec. This change is backwards compatible with the previous spec, and current known instances of tests that meet the spec.
Cheers,
Stef
ci@lists.stg.fedoraproject.org
-
Stef Walter