On Wed, May 22, 2019 at 13:25:17 +0200, Petr Šplíchal psplicha@redhat.com wrote:
Hi Bruno,
for your use case (I see you have a single shell test script in your pull request [1]) I would recommend to use the basic role:
https://docs.fedoraproject.org/en-US/ci/standard-test-roles/#_basic
It would take care of correctly copying artifacts and also handle test script return code to report appropriate result. Also adding a list of required packages is easy there. You can also have a look at a few more examples here:
The script doesn't return a failure unless it has a problem with something outside what is being tested. You need to run grep to see if any of the tests, run in loops, fails. That messes with the output. I could probably string something together not much different than I did for the wrapper case though.
I'm close enough with the wrapper set up now, that I'll probably see if I can finish with it. However, the documentation being off could cause a lot of confusion for others trying to create tests for their packages and something should be done to fix it or warn people off.
I've also been seeing random failures with set up of the environment. Not always the same thing. I'm worried there might be issues with gating because of this. I'm not sure how to rerun a test that has been used for gating.