Hello,
the testing system is required to inspect the exit code of the test suite [1]:
To invoke the playbook, the *testing system*:
- MUST examine the exit code of the playbook. A zero exit code is
successful *test result*, non-zero is failure.
However, the test suite is not required to return the correct exit code. Under
The playbook and its *test suite* or *test framework*:
there is no mention of returning the exit code. I think that should be clarified, because it can be expected that people implementing SI will only read those parts relevant for them. I believe this should be added:
The playbook and its *test suite* or *test framework*:
- MUST return a zero exit code of the playbook if the test result is a
pass, or a non-zero exit code if the test result is a fail.
Thoughts?
On Fri, Sep 01, 2017 at 02:02:16PM +0200, Kamil Paral wrote:
Hello,
the testing system is required to inspect the exit code of the test suite [1]:
To invoke the playbook, the testing system: 6. MUST examine the exit code of the playbook. A zero exit code is successful test result, non-zero is failure.However, the test suite is not required to return the correct exit code. Under
The playbook and its test suite or test framework:there is no mention of returning the exit code. I think that should be clarified, because it can be expected that people implementing SI will only read those parts relevant for them. I believe this should be added:
The playbook and its test suite or test framework: 6. MUST return a zero exit code of the playbook if the test result is a pass, or a non-zero exit code if the test result is a fail.Thoughts?
This sounds fair to me
+1
Pierre
On Mon, Sep 4, 2017 at 12:01 PM, Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Fri, Sep 01, 2017 at 02:02:16PM +0200, Kamil Paral wrote:
Hello,
the testing system is required to inspect the exit code of the test
suite
[1]:
To invoke the playbook, the testing system: 6. MUST examine the exit code of the playbook. A zero exit code is successful test result, non-zero is failure.However, the test suite is not required to return the correct exit
code.
Under
The playbook and its test suite or test framework:there is no mention of returning the exit code. I think that should be clarified, because it can be expected that people implementing SI will only read those parts relevant for them. I believe this should be
added:
The playbook and its test suite or test framework: 6. MUST return a zero exit code of the playbook if the test resultis a
pass, or a non-zero exit code if the test result is a fail.Thoughts?
This sounds fair to me
+1
Pierre
I've implemented the change here: https://fedoraproject.org/w/index.php?title=Changes%2FInvokingTests&diff...
On 07.09.2017 10:20, Kamil Paral wrote:
On Mon, Sep 4, 2017 at 12:01 PM, Pierre-Yves Chibon <pingou@pingoured.fr mailto:pingou@pingoured.fr> wrote:
On Fri, Sep 01, 2017 at 02:02:16PM +0200, Kamil Paral wrote: > Hello, > > the testing system is required to inspect the exit code of the test suite > [1]: > > To invoke the playbook, the testing system: > 6. MUST examine the exit code of the playbook. A zero exit code is > successful test result, non-zero is failure. > > However, the test suite is not required to return the correct exit code. > Under > > The playbook and its test suite or test framework: > > there is no mention of returning the exit code. I think that should be > clarified, because it can be expected that people implementing SI will > only read those parts relevant for them. I believe this should be added: > > The playbook and its test suite or test framework: > > 6. MUST return a zero exit code of the playbook if the test result is a > pass, or a non-zero exit code if the test result is a fail. > > Thoughts? > > [1] https://fedoraproject.org/wiki/Changes/InvokingTests <https://fedoraproject.org/wiki/Changes/InvokingTests> This sounds fair to me +1 PierreI've implemented the change here: https://fedoraproject.org/w/index.php?title=Changes%2FInvokingTests&diff...
Makes sense to me. To be clear this is a spec only change and does not affect any already ported/wrapped/written tests. Correct?
Stef
On Mon, Sep 11, 2017 at 2:09 PM, Stef Walter stefw@redhat.com wrote:
Makes sense to me. To be clear this is a spec only change and does not affect any already ported/wrapped/written tests. Correct?
Stef
Yes, I only adjust the wiki page, nothing else.
ci@lists.stg.fedoraproject.org
-
Kamil Paral -
Pierre-Yves Chibon -
Stef Walter