Hello,
I'd like to get a few clarifications regarding the default test playbook example here: https://fedoraproject.org/wiki/CI/Tests#Wrapping_a_script_test
First, the example seems to suggest that all binaries should be placed in /usr/local/bin:
- name: Create the folder where we will store the tests action: file state=directory path={{ item }} owner=root group=root with_items: - /usr/local/bin
- name: Install the test files copy: src={{ item.file }} dest=/usr/local/bin/{{ item.dest }} mode=0755 with_items: - {file: test-simple, dest: test-simple }
But I don't see such a requirement in the SI. Why shouldn't I run the test executable from its current location (i.e. the git checkout directory)? And if they can, why make the example unnecessarily complex?
Second, I'm a bit confused by the exec command in the test execution line:
- name: Execute the tests shell: exec > {{ artifacts }}/output && /usr/local/bin/test-simple 2>&1
I've never seen this usage so I had to study what this does exactly in the bash man page. I believe it's the same thing as this:
shell: /usr/local/bin/test-simple >{{ artifacts }}/output 2>&1
So why exactly don't we use this more readable and more often used syntax to redirect a process output? An example should be as simple as possible.
Third, SI says:
The playbook and its test suite or test framework: MUST place the main readable output of the test suite into a test.log file in the artifacts variable folder. This MUST happen even if some of the test suites fail.
However, the default example doesn't contain any reference to *test.log* and uses *output* file instead. Shouldn't those examples above redirect to *test.log* instead? Or is *test.log* not supposed to be an stdout+stderr output of the main test process? I don't exactly understand the definition from SI, what *main readable output* is supposed to be. When the test suite consists of X individual tests, are they supposed to concatenate their output to *test.log*?
It would be great to have these clarified.
Thanks, Kamil
On Wed, Sep 6, 2017 at 12:33 PM, Kamil Paral kparal@redhat.com wrote:
Third, SI says:
The playbook and its test suite or test framework: MUST place the main readable output of the test suite into a test.log file in the artifacts variable folder. This MUST happen even if some of the test suites fail.
However, the default example doesn't contain any reference to *test.log* and uses *output* file instead. Shouldn't those examples above redirect to *test.log* instead?
It turns out I've been looking at old cached wiki content. After refreshing, I see that *test.log* is referenced instead.
Or is *test.log* not supposed to be an stdout+stderr output of the main test process? I don't exactly understand the definition from SI, what *main readable output* is supposed to be. When the test suite consists of X individual tests, are they supposed to concatenate their output to *test.log*?
These questions still stand, though.
On Wed, Sep 6, 2017 at 7:12 AM, Kamil Paral kparal@redhat.com wrote:
On Wed, Sep 6, 2017 at 12:33 PM, Kamil Paral kparal@redhat.com wrote:
Third, SI says:
The playbook and its test suite or test framework: MUST place the main readable output of the test suite into a test.log file in the artifacts variable folder. This MUST happen even if some of the test suites fail.
However, the default example doesn't contain any reference to *test.log* and uses *output* file instead. Shouldn't those examples above redirect to *test.log* instead?
It turns out I've been looking at old cached wiki content. After refreshing, I see that *test.log* is referenced instead.
Or is *test.log* not supposed to be an stdout+stderr output of the main test process? I don't exactly understand the definition from SI, what *main readable output* is supposed to be. When the test suite consists of X individual tests, are they supposed to concatenate their output to *test.log*?
These questions still stand, though.
You should see the test.log file as the main file to debug the run so it should contain the information to debug issues. So it's up to the test implementer to decide if it's stdout or stderr or both according to what he needs to debug. Does it make sense?
Fred
On Wed, Sep 6, 2017 at 2:07 PM, Frederic Lepied flepied@redhat.com wrote:
On Wed, Sep 6, 2017 at 7:12 AM, Kamil Paral kparal@redhat.com wrote:
On Wed, Sep 6, 2017 at 12:33 PM, Kamil Paral kparal@redhat.com wrote:
Third, SI says:
The playbook and its test suite or test framework: MUST place the main readable output of the test suite into a test.log file in the artifacts variable folder. This MUST happen even if some of the test suites fail.
However, the default example doesn't contain any reference to *test.log* and uses *output* file instead. Shouldn't those examples above redirect to *test.log* instead?
It turns out I've been looking at old cached wiki content. After refreshing, I see that *test.log* is referenced instead.
Or is *test.log* not supposed to be an stdout+stderr output of the main test process? I don't exactly understand the definition from SI, what *main readable output* is supposed to be. When the test suite consists of X individual tests, are they supposed to concatenate their output to *test.log*?
These questions still stand, though.
You should see the test.log file as the main file to debug the run so it should contain the information to debug issues. So it's up to the test implementer to decide if it's stdout or stderr or both according to what he needs to debug. Does it make sense?
Fred
It does, thanks. Will you include this clarification in the spec? The main important information for me here is that the purpose of that file is to *debug* the execution, and most often it'll be stdout+stderr. Because "readable output", as it currently written, might be understood as human or machine readable, i.e. a json with detailed results, which is something completely different.
On Wed, Sep 6, 2017 at 8:30 AM, Kamil Paral kparal@redhat.com wrote:
On Wed, Sep 6, 2017 at 2:07 PM, Frederic Lepied flepied@redhat.com wrote:
On Wed, Sep 6, 2017 at 7:12 AM, Kamil Paral kparal@redhat.com wrote:
On Wed, Sep 6, 2017 at 12:33 PM, Kamil Paral kparal@redhat.com wrote:
Third, SI says:
The playbook and its test suite or test framework: MUST place the main readable output of the test suite into a test.log file in the artifacts variable folder. This MUST happen even if some of the test suites fail.
However, the default example doesn't contain any reference to *test.log* and uses *output* file instead. Shouldn't those examples above redirect to *test.log* instead?
It turns out I've been looking at old cached wiki content. After refreshing, I see that *test.log* is referenced instead.
Or is *test.log* not supposed to be an stdout+stderr output of the main test process? I don't exactly understand the definition from SI, what *main readable output* is supposed to be. When the test suite consists of X individual tests, are they supposed to concatenate their output to *test.log*?
These questions still stand, though.
You should see the test.log file as the main file to debug the run so it should contain the information to debug issues. So it's up to the test implementer to decide if it's stdout or stderr or both according to what he needs to debug. Does it make sense?
Fred
It does, thanks. Will you include this clarification in the spec? The main important information for me here is that the purpose of that file is to *debug* the execution, and most often it'll be stdout+stderr. Because "readable output", as it currently written, might be understood as human or machine readable, i.e. a json with detailed results, which is something completely different.
I'll update the specs if no one objects in the coming days.
Fred
On 09/06/2017 07:07 AM, Frederic Lepied wrote:
On Wed, Sep 6, 2017 at 7:12 AM, Kamil Paral <kparal@redhat.com mailto:kparal@redhat.com> wrote:
On Wed, Sep 6, 2017 at 12:33 PM, Kamil Paral <kparal@redhat.com <mailto:kparal@redhat.com>> wrote: Third, SI says: The playbook and its test suite or test framework: MUST place the main readable output of the test suite into a test.log file in the artifacts variable folder. This MUST happen even if some of the test suites fail. However, the default example doesn't contain any reference to /test.log/ and uses /output/ file instead. Shouldn't those examples above redirect to /test.log/ instead? It turns out I've been looking at old cached wiki content. After refreshing, I see that /test.log/ is referenced instead. Or is /test.log/ not supposed to be an stdout+stderr output of the main test process? I don't exactly understand the definition from SI, what /main readable output/ is supposed to be. When the test suite consists of X individual tests, are they supposed to concatenate their output to /test.log/? These questions still stand, though.You should see the test.log file as the main file to debug the run so it should contain the information to debug issues. So it's up to the test implementer to decide if it's stdout or stderr or both according to what he needs to debug. Does it make sense?
Fred
For what it's worth, the standard-test-beakerlib and standard-test-rhts included in the standard-test-roles package [1] currently write test.log as a brief summary of the results of individual tests. For example,
PASS test-1 PASS test-2 FAIL test-3 ...
Beyond that, standard-test-beakerlib captures a test./test-name/.log artifact containing the stdout+stderr for each individual test. (eg., test.test-1.log, test.test-2.log, etc.), whereas standard-test-rhts includes the entire job./XX/ tree (generated from restraint) as an artifact.
[1] https://pagure.io/standard-test-roles
Merlin
On Wed, Sep 6, 2017 at 7:18 PM, Merlin Mathesius mmathesi@redhat.com wrote:
For what it's worth, the standard-test-beakerlib and standard-test-rhts included in the standard-test-roles package [1] currently write test.log as a brief summary of the results of individual tests. For example,
PASS test-1 PASS test-2 FAIL test-3 ...
Beyond that, standard-test-beakerlib captures a test.*test-name*.log artifact containing the stdout+stderr for each individual test. (eg., test.test-1.log, test.test-2.log, etc.), whereas standard-test-rhts includes the entire job.*XX* tree (generated from restraint) as an artifact.
[1] https://pagure.io/standard-test-roles
Merlin
Thanks Merlin. That's exactly what I wanted to avoid, each author understanding the spec differently.
I think there are two major kinds of artifacts - an stdout+stderr output used for debugging problems, and (optionally, but frequently) a result summary. I wouldn't probably try to standardize the result summary file name, it's not strictly required and the file type can vary a lot - results.html, summary.txt, results.tap, results/index.html, etc. But it would be useful to have a standard name for the stdout+stderr output, so that you always know where to look when debugging problems (test.log is as good name for it as any).
So if we go this way, some of the roles in standard-test-roles should get updated.
On 06.09.2017 12:33, Kamil Paral wrote:
Hello,
I'd like to get a few clarifications regarding the default test playbook example here: https://fedoraproject.org/wiki/CI/Tests#Wrapping_a_script_test
First, the example seems to suggest that all binaries should be placed in /usr/local/bin:
name: Create the folder where we will store the tests action: file state=directory path={{ item }} owner=root group=root with_items:
- /usr/local/bin
name: Install the test files copy: src={{ item.file }} dest=/usr/local/bin/{{ item.dest }} mode=0755 with_items:
- {file: test-simple, dest: test-simple }
But I don't see such a requirement in the SI. Why shouldn't I run the test executable from its current location (i.e. the git checkout directory)? And if they can, why make the example unnecessarily complex?
I looked through the thread but didn't find an answer to this part of your question.
This is because when running against a docker container or atomic host test subject (qcow2 and docker respectively) such shell script tests (ditto for beakerlib) need to be placed inside the test subject in order to be run.
Cheers,
Stef
On Mon, Sep 11, 2017 at 2:11 PM, Stef Walter stefw@redhat.com wrote:
I looked through the thread but didn't find an answer to this part of your question.
This is because when running against a docker container or atomic host test subject (qcow2 and docker respectively) such shell script tests (ditto for beakerlib) need to be placed inside the test subject in order to be run.
Cheers,
Stef
I see, that makes sense. Thanks for explanation. I'm looking forward to replies to my other questions as well :) Thanks.
-
Frederic Lepied -
Kamil Paral -
Merlin Mathesius -
Stef Walter