On Wed, 22 May 2019 at 14:55, Bruno Wolff III <bruno@wolff.to> wrote:
>
> On Wed, May 22, 2019 at 13:25:17 +0200,
>   Petr Šplíchal <psplicha@redhat.com> wrote:
> >Hi Bruno,
> >
> >for your use case (I see you have a single shell test script in
> >your pull request [1]) I would recommend to use the basic role:
> >
> >https://docs.fedoraproject.org/en-US/ci/standard-test-roles/#_basic
> >
> >It would take care of correctly copying artifacts and also
> >handle test script return code to report appropriate result.
> >Also adding a list of required packages is easy there. You can
> >also have a look at a few more examples here:
>
> The script doesn't return a failure unless it has a problem with
> something outside what is being tested. You need to run grep to
> see if any of the tests, run in loops, fails. That messes with
> the output. I could probably string something together not much
> different than I did for the wrapper case though.

You could do something like this:

- hosts: localhost
  roles:
  - role: standard-test-basic
    tags:
    - classic
    tests:
    - compression:
        dir: .
        run: "./Squashfs-compression-test.sh 2>&1 | tee /tmp/test.log && ! grep -iq failed /tmp/test.log"
    required_packages:
    - kernel-modules-extra
    - rpmdevtools

Which looks much more concise.

> I'm close enough with the wrapper set up now, that I'll probably
> see if I can finish with it. However, the documentation being
> off could cause a lot of confusion for others trying to create
> tests for their packages and something should be done to fix it
> or warn people off.

What exactly in the docs should cause the confusion? Should any of
these pages be updated?

https://docs.fedoraproject.org/en-US/ci/quick-start-guide/
https://docs.fedoraproject.org/en-US/ci/standard-test-roles/
https://docs.fedoraproject.org/en-US/ci/examples/

Or are you missing some essential content there?

> I've also been seeing random failures with set up of the
> environment.  Not always the same thing. I'm worried there might
> be issues with gating because of this. I'm not sure how to rerun
> a test that has been used for gating.

Including the "[citest]" string in a pull request comment does
rerun all enabled tests.

psss...