Hi,

The standard test interface defines `tests/tests.yml` as the sole playbook to be run to test a repository. This makes it impossible to run multiple independent tests on fresh instances of the same test subjects.

I noticed this while converting the tests for linux-system-roles to this standard. We're running different ansible playbooks against virtual machine images as test subjects and would like each playbook to start from a fresh machine. We've already had a bug where one of the tests didn't clean up correctly and influenced the result of the next test.

Thus, I suggest changing the "Invocation" section to say that the testing system must independently run all playbooks `tests/tests*.yml` in the dist-git repo. This is compatible with existing repositories.

If we decide to go this way, we'll have to decide how to handle artifacts in this case. I see two reasonable choices:

1. Keep one artifacts directory and let the authors of the playbooks take care that they don't overwrite each other's result. We couldn't run the tests in parallel in this case, because tests might want to concatenate a main`test.log`.

2. Have a separate artifacts directory for each test run.

What do you think?

Lars