CI September 2017

ci@lists.fedoraproject.org

6 participants
6 discussions

Multiple test suite playbooks
by Stef Walter 27 Sep '17

27 Sep '17

Lars from the standard-test-roles [0] project is using our test specification for invoking tests. He brought up a good point (I paraphrase): Many tests are destructive in that they change the state of the VM or container that they are running in. However when using the Ansible Dynamic Inventory means the entire tests.yml playbook is run against the same VM or Docker image started from the test subject. I would suggest our spec is amended in section "1.7.2 Invocation" to have the first sentence replaced with: "The testing system MUST run each playbook matching the glob tests/tests*.yml in the dist-git repo. Each of these files constitutes a test suite. Each test suite is invoked independently by the testing system as follows." What do you think? Cheers, Stef

1 1

Allow multiple independent tests for each project
by Lars Karlitski 19 Sep '17

19 Sep '17

Hi, The standard test interface defines `tests/tests.yml` as the sole playbook to be run to test a repository. This makes it impossible to run multiple independent tests on fresh instances of the same test subjects. I noticed this while converting the tests for linux-system-roles to this standard. We're running different ansible playbooks against virtual machine images as test subjects and would like each playbook to start from a fresh machine. We've already had a bug where one of the tests didn't clean up correctly and influenced the result of the next test. Thus, I suggest changing the "Invocation" section to say that the testing system must independently run all playbooks `tests/tests*.yml` in the dist-git repo. This is compatible with existing repositories. If we decide to go this way, we'll have to decide how to handle artifacts in this case. I see two reasonable choices: 1. Keep one artifacts directory and let the authors of the playbooks take care that they don't overwrite each other's result. We couldn't run the tests in parallel in this case, because tests might want to concatenate a main`test.log`. 2. Have a separate artifacts directory for each test run. What do you think? Lars

1 0

standard interface or standard test interface
by Kamil Paral 13 Sep '17

13 Sep '17

I wonder how set you are on the "standard interface" name. I find it too generic, and confusing for most people not fully involved with it. Interface for *what*? When we talk about it in our team, we mostly call it "standard test interface", because then it's clearer what it means, and it even has a nice STI abbreviation when writing (while SI is too short and ambiguous). Thoughts on calling it "standard test interface" instead? If you think this is too much of a bikeshedding, I apologize, ignore me. Kamil

2 1

SI - multiple subjects concatenation
by Kamil Paral 13 Sep '17

13 Sep '17

SI says [1]: Often only one *test subject* is passed in. However multiple subjects may > be concatenated together in a shell escaped string. The playbook and > inventory script split the string. > It does not say what the separator character is. It might be a space, it might be any whitespace, it might be a comma, it might be a semicolon, etc. This should be clarified. I'm also confused what "shell escaped" means here. It's not like we'll pipe this to bash, we'll process this with ansible. And even if we did, you still want the raw value, not the escaped one, as the variable, don't you? E.g. for "foo" and "bar" subjects, you want this: $ export TEST_SUBJECTS='foo bar' $ echo $TEST_SUBJECTS foo bar and not this: $ export TEST_SUBJECTS='foo\ bar' $ echo $TEST_SUBJECTS foo\ bar Correct? But that brings another question - what if I have the separator character inside the subject name? You might say that's not likely with current fedora artfifacts, but a standard interface definition should be forward-looking. Even now it is technically possible to have a space inside an .iso/.qcow2 image name. It might be even more common for some future test subject type. The spec should describe this as well. Personally, I think we should get away from string parsing, if we want to design a reliable system. String parsing and especially shell escaping is a never ending story of unexpected failures and corner case issues. It would be better to use a proper structure for this. As it turns out, ansible can do this already [2]. You can provide a true list to it via --extra-vars, by passing in a json, like this: $ ansible-playbook tests.yml --extra-vars '{"subjects": ["foo","bar","even a space"]}' This way you don't need to do string parsing in an ansible playbook, you can immediately use its native structures, like module: do something for {{item}} with_items: "{{subjects}}" You also don't need to deal with character escaping (especially shell one), all of that is handled by json. This makes writing tests.yml a lot easier, I think. There's of course a question what to do with TEST_SUBJECTS envvar if we make "subjects" a native ansible variable. I'd personally use json for it as well, so: $ export TEST_SUBJECTS='["foo", "bar", "even a space"]' There's a json parser for every possible language, and there are convenient tools for bash as well [3]. But maybe I'd like to first understand why we require the envvars to be present in the first place. I don't think they're needed. Is it just a convenience measure? Thanks, Kamil [1] https://fedoraproject.org/wiki/Changes/InvokingTests#Invocation [2] http://docs.ansible.com/ansible/latest/playbooks_variables.html#passing-var… [3] https://stedolan.github.io/jq/

2 1

a few questions regarding the example playbook for running tests
by Kamil Paral 12 Sep '17

12 Sep '17

Hello, I'd like to get a few clarifications regarding the default test playbook example here: https://fedoraproject.org/wiki/CI/Tests#Wrapping_a_script_test First, the example seems to suggest that all binaries should be placed in /usr/local/bin: - name: Create the folder where we will store the tests action: file state=directory path={{ item }} owner=root group=root with_items: - /usr/local/bin - name: Install the test files copy: src={{ item.file }} dest=/usr/local/bin/{{ item.dest }} mode=0755 with_items: - {file: test-simple, dest: test-simple } But I don't see such a requirement in the SI. Why shouldn't I run the test executable from its current location (i.e. the git checkout directory)? And if they can, why make the example unnecessarily complex? Second, I'm a bit confused by the exec command in the test execution line: - name: Execute the tests shell: exec > {{ artifacts }}/output && /usr/local/bin/test-simple 2>&1 I've never seen this usage so I had to study what this does exactly in the bash man page. I believe it's the same thing as this: shell: /usr/local/bin/test-simple >{{ artifacts }}/output 2>&1 So why exactly don't we use this more readable and more often used syntax to redirect a process output? An example should be as simple as possible. Third, SI says: > The playbook and its test suite or test framework: > MUST place the main readable output of the test suite into a test.log > file in the artifacts variable folder. This MUST happen even if some of the > test suites fail. > However, the default example doesn't contain any reference to *test.log* and uses *output* file instead. Shouldn't those examples above redirect to *test.log* instead? Or is *test.log* not supposed to be an stdout+stderr output of the main test process? I don't exactly understand the definition from SI, what *main readable output* is supposed to be. When the test suite consists of X individual tests, are they supposed to concatenate their output to *test.log*? It would be great to have these clarified. Thanks, Kamil

4 8

SI - clarification about exit codes
by Kamil Paral 12 Sep '17

12 Sep '17

Hello, the testing system is required to inspect the exit code of the test suite [1]: To invoke the playbook, the *testing system*: > 6. MUST examine the exit code of the playbook. A zero exit code is > successful *test result*, non-zero is failure. > However, the test suite is not required to return the correct exit code. Under > The playbook and its *test suite* or *test framework*: > there is no mention of returning the exit code. I think that should be clarified, because it can be expected that people implementing SI will only read those parts relevant for them. I believe this should be added: > The playbook and its *test suite* or *test framework*: > > 6. MUST return a zero exit code of the playbook if the test result is a > pass, or a non-zero exit code if the test result is a fail. > Thoughts? [1] https://fedoraproject.org/wiki/Changes/InvokingTests

3 4

2024

2023

2022

2021

2020

2019

2018

2017

CI September 2017