I've added documentation about launching test subjects using Ansible Dynamic inventory scripts. Test subjects are the thing to be tested. For example a QCow2, container image, or often a set of RPMs. These get launched into a form that Ansible (and thus the test playbooks) can run against them:
The tutorial style documentation is here:
https://fedoraproject.org/wiki/CI/Tests
More documentation will be landing here about writing tests, preparing a dist-git repo for tests, and so on.
The information about the spec is here:
https://fedoraproject.org/wiki/Changes/InvokingTests
Note that older version of the spec used to have a test_local.yml file, and test subject specific playbooks. These test subject specific playbooks are no longer necessary with the inventory.
In addition test_local.yml may be used in place of tests.yml in the documentation. Once more people have looked over this, I'll open some pull requests to rename test_local.yml to tests.yml.
For the documentation to work, we do need to package the inventory scripts that are already present in standard-test-roles. In addition, I've opened a pull request to standard-test-roles for adding support for launching docker containers as dynamic inventory:
https://pagure.io/standard-test-roles/pull-request/22
Cheers,
Stef
On 07/17/2017 05:34 AM, Stef Walter wrote:
I've added documentation about launching test subjects using Ansible Dynamic inventory scripts. Test subjects are the thing to be tested. For example a QCow2, container image, or often a set of RPMs. These get launched into a form that Ansible (and thus the test playbooks) can run against them:
The tutorial style documentation is here:
https://fedoraproject.org/wiki/CI/Tests
More documentation will be landing here about writing tests, preparing a dist-git repo for tests, and so on.
The information about the spec is here:
https://fedoraproject.org/wiki/Changes/InvokingTests
Note that older version of the spec used to have a test_local.yml file, and test subject specific playbooks. These test subject specific playbooks are no longer necessary with the inventory.
In addition test_local.yml may be used in place of tests.yml in the documentation. Once more people have looked over this, I'll open some pull requests to rename test_local.yml to tests.yml.
For the documentation to work, we do need to package the inventory scripts that are already present in standard-test-roles. In addition, I've opened a pull request to standard-test-roles for adding support for launching docker containers as dynamic inventory:
Hi, Stef,
Thank you for providing the documentation and examples at https://fedoraproject.org/wiki/CI/Tests.
PR#22 and several other fixe/improvement PRs have been merged.
I was able to experiment with the inventory scripts a bit, and there's a few improvements I'd like to see:
1. The docker script is not compatible with Fedora 25 due to a particular improvement in docker (combining "--rm" and "--detach" for "docker run ...") that's available on F26 that the script is taking advantage of. F25 could be accommodated (I think) by eliminating the "--rm" from "docker run ..." and doing a "docker rm -f ..." during cleanup.
2. The qcow2 script creates a log file in the current directory. That should really be in the artifacts directory--or other directory of choice. What would you think of having the script honor a log directory environment variable? (eg., "LOGDIR")
3. The docker script should probably create a log file too. ("docker logs ... > logfile")
I'd be happy to take a shot at implementing the above improvements. Then, I think we have something ready to tag and package as v2.0.
I would also be interested in seeing the example "test.yml" you refer to on the wiki page. I couldn't find it checked in anywhere.
Also, do you have any thoughts on how to have a single test.yml playbook cleanly pick and choose amongst which tests to run in which environments? Some tests simply won't run in a container and/or cloud image, and so should be skipped when they can't run properly. I also imagine there could be tests that should be run ONLY in a container and/or cloud image. I attempted to implement something like this with the "selector" role (https://pagure.io/standard-test-roles/blob/master/f/roles/standard-test-sele...), but it feels rather clumsy to use (eg., https://upstreamfirst.fedorainfracloud.org/sed/blob/master/f/test_local.yml).
Cheers,
Stef
On 18.07.2017 16:38, Merlin Mathesius wrote:
On 07/17/2017 05:34 AM, Stef Walter wrote:
I've added documentation about launching test subjects using Ansible Dynamic inventory scripts. Test subjects are the thing to be tested. For example a QCow2, container image, or often a set of RPMs. These get launched into a form that Ansible (and thus the test playbooks) can run against them:
The tutorial style documentation is here:
https://fedoraproject.org/wiki/CI/Tests
More documentation will be landing here about writing tests, preparing a dist-git repo for tests, and so on.
The information about the spec is here:
https://fedoraproject.org/wiki/Changes/InvokingTests
Note that older version of the spec used to have a test_local.yml file, and test subject specific playbooks. These test subject specific playbooks are no longer necessary with the inventory.
In addition test_local.yml may be used in place of tests.yml in the documentation. Once more people have looked over this, I'll open some pull requests to rename test_local.yml to tests.yml.
For the documentation to work, we do need to package the inventory scripts that are already present in standard-test-roles. In addition, I've opened a pull request to standard-test-roles for adding support for launching docker containers as dynamic inventory:
Hi, Stef,
Thank you for providing the documentation and examples at https://fedoraproject.org/wiki/CI/Tests.
PR#22 and several other fixe/improvement PRs have been merged.
I was able to experiment with the inventory scripts a bit, and there's a few improvements I'd like to see:
- The docker script is not compatible with Fedora 25 due to a
particular improvement in docker (combining "--rm" and "--detach" for "docker run ...") that's available on F26 that the script is taking advantage of. F25 could be accommodated (I think) by eliminating the "--rm" from "docker run ..." and doing a "docker rm -f ..." during cleanup.
Good catch thanks. Hope this fixes that:
https://pagure.io/standard-test-roles/pull-request/23
- The qcow2 script creates a log file in the current directory. That
should really be in the artifacts directory--or other directory of choice. What would you think of having the script honor a log directory environment variable? (eg., "LOGDIR")
Ah yes. It's such a shame that Ansible doesn't pass it's -e variables to inventory scripts. For this reason the spec mandates also passing the artifacts directory as an environment variable:
$TEST_ARTIFACTS
https://fedoraproject.org/wiki/Changes/InvokingTests#Invocation
I'll start to make use of it to accomplish this.
- The docker script should probably create a log file too. ("docker
logs ... > logfile")
Will do along with the above.
I'd be happy to take a shot at implementing the above improvements. Then, I think we have something ready to tag and package as v2.0.
Wonderful.
I would also be interested in seeing the example "test.yml" you refer to on the wiki page. I couldn't find it checked in anywhere.
See the gzip git repo. But you can typically substitute test_local.yml.
Also, do you have any thoughts on how to have a single test.yml playbook cleanly pick and choose amongst which tests to run in which environments? Some tests simply won't run in a container and/or cloud image, and so should be skipped when they can't run properly. I also imagine there could be tests that should be run ONLY in a container and/or cloud image. I attempted to implement something like this with the "selector" role (https://pagure.io/standard-test-roles/blob/master/f/roles/standard-test-sele...), but it feels rather clumsy to use (eg., https://upstreamfirst.fedorainfracloud.org/sed/blob/master/f/test_local.yml).
I believe we're brainstorming this today. We could factor this aspect in.
Stef
On 07/19/2017 07:32 AM, Stef Walter wrote:
On 18.07.2017 16:38, Merlin Mathesius wrote:
On 07/17/2017 05:34 AM, Stef Walter wrote:
I've added documentation about launching test subjects using Ansible Dynamic inventory scripts. Test subjects are the thing to be tested. For example a QCow2, container image, or often a set of RPMs. These get launched into a form that Ansible (and thus the test playbooks) can run against them:
The tutorial style documentation is here:
https://fedoraproject.org/wiki/CI/Tests
More documentation will be landing here about writing tests, preparing a dist-git repo for tests, and so on.
The information about the spec is here:
https://fedoraproject.org/wiki/Changes/InvokingTests
Note that older version of the spec used to have a test_local.yml file, and test subject specific playbooks. These test subject specific playbooks are no longer necessary with the inventory.
In addition test_local.yml may be used in place of tests.yml in the documentation. Once more people have looked over this, I'll open some pull requests to rename test_local.yml to tests.yml.
For the documentation to work, we do need to package the inventory scripts that are already present in standard-test-roles. In addition, I've opened a pull request to standard-test-roles for adding support for launching docker containers as dynamic inventory:
Hi, Stef,
Thank you for providing the documentation and examples at https://fedoraproject.org/wiki/CI/Tests.
PR#22 and several other fixe/improvement PRs have been merged.
I was able to experiment with the inventory scripts a bit, and there's a few improvements I'd like to see:
- The docker script is not compatible with Fedora 25 due to a
particular improvement in docker (combining "--rm" and "--detach" for "docker run ...") that's available on F26 that the script is taking advantage of. F25 could be accommodated (I think) by eliminating the "--rm" from "docker run ..." and doing a "docker rm -f ..." during cleanup.
Good catch thanks. Hope this fixes that:
Stef,
Yes it does. It has been merged. Thanks.
- The qcow2 script creates a log file in the current directory. That
should really be in the artifacts directory--or other directory of choice. What would you think of having the script honor a log directory environment variable? (eg., "LOGDIR")
Ah yes. It's such a shame that Ansible doesn't pass it's -e variables to inventory scripts. For this reason the spec mandates also passing the artifacts directory as an environment variable:
$TEST_ARTIFACTS
https://fedoraproject.org/wiki/Changes/InvokingTests#Invocation
I'll start to make use of it to accomplish this.
- The docker script should probably create a log file too. ("docker
logs ... > logfile")
Will do along with the above.
Thanks for https://pagure.io/standard-test-roles/pull-request/24 which accomplishes placing the inventory script logfiles in the $TEST_ARTIFACTS directory. It has also been merged.
To further make use of the $TEST_ARTIFACTS environment variable, I have raised https://pagure.io/standard-test-roles/pull-request/25 so the current roles will take their default artifacts directory from $TEST_ARTIFACTS--using "./artifacts" if not specified. Thus, specifying "-e artifacts=..." on the ansible-playbook command line is no longer necessary--but can still be provided as an override if desired.
Would you please let me know what you think about PR#25, Stef?
I'd be happy to take a shot at implementing the above improvements. Then, I think we have something ready to tag and package as v2.0.
Wonderful.
I would also be interested in seeing the example "test.yml" you refer to on the wiki page. I couldn't find it checked in anywhere.
See the gzip git repo. But you can typically substitute test_local.yml.
Got it. I thought maybe you had come up with a new and interesting standard format for a master test playbook. :)
Also, do you have any thoughts on how to have a single test.yml playbook cleanly pick and choose amongst which tests to run in which environments? Some tests simply won't run in a container and/or cloud image, and so should be skipped when they can't run properly. I also imagine there could be tests that should be run ONLY in a container and/or cloud image. I attempted to implement something like this with the "selector" role (https://pagure.io/standard-test-roles/blob/master/f/roles/standard-test-sele...), but it feels rather clumsy to use (eg., https://upstreamfirst.fedorainfracloud.org/sed/blob/master/f/test_local.yml).
I believe we're brainstorming this today. We could factor this aspect in.
Stef
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org
On 19.07.2017 22:56, Merlin Mathesius wrote:
On 07/19/2017 07:32 AM, Stef Walter wrote:
On 18.07.2017 16:38, Merlin Mathesius wrote:
On 07/17/2017 05:34 AM, Stef Walter wrote:
I've added documentation about launching test subjects using Ansible Dynamic inventory scripts. Test subjects are the thing to be tested. For example a QCow2, container image, or often a set of RPMs. These get launched into a form that Ansible (and thus the test playbooks) can run against them:
The tutorial style documentation is here:
https://fedoraproject.org/wiki/CI/Tests
More documentation will be landing here about writing tests, preparing a dist-git repo for tests, and so on.
The information about the spec is here:
https://fedoraproject.org/wiki/Changes/InvokingTests
Note that older version of the spec used to have a test_local.yml file, and test subject specific playbooks. These test subject specific playbooks are no longer necessary with the inventory.
In addition test_local.yml may be used in place of tests.yml in the documentation. Once more people have looked over this, I'll open some pull requests to rename test_local.yml to tests.yml.
For the documentation to work, we do need to package the inventory scripts that are already present in standard-test-roles. In addition, I've opened a pull request to standard-test-roles for adding support for launching docker containers as dynamic inventory:
Hi, Stef,
Thank you for providing the documentation and examples at https://fedoraproject.org/wiki/CI/Tests.
PR#22 and several other fixe/improvement PRs have been merged.
I was able to experiment with the inventory scripts a bit, and there's a few improvements I'd like to see:
- The docker script is not compatible with Fedora 25 due to a
particular improvement in docker (combining "--rm" and "--detach" for "docker run ...") that's available on F26 that the script is taking advantage of. F25 could be accommodated (I think) by eliminating the "--rm" from "docker run ..." and doing a "docker rm -f ..." during cleanup.
Good catch thanks. Hope this fixes that:
Stef,
Yes it does. It has been merged. Thanks.
- The qcow2 script creates a log file in the current directory. That
should really be in the artifacts directory--or other directory of choice. What would you think of having the script honor a log directory environment variable? (eg., "LOGDIR")
Ah yes. It's such a shame that Ansible doesn't pass it's -e variables to inventory scripts. For this reason the spec mandates also passing the artifacts directory as an environment variable:
$TEST_ARTIFACTS
https://fedoraproject.org/wiki/Changes/InvokingTests#Invocation
I'll start to make use of it to accomplish this.
- The docker script should probably create a log file too. ("docker
logs ... > logfile")
Will do along with the above.
Thanks for https://pagure.io/standard-test-roles/pull-request/24 which accomplishes placing the inventory script logfiles in the $TEST_ARTIFACTS directory. It has also been merged.
To further make use of the $TEST_ARTIFACTS environment variable, I have raised https://pagure.io/standard-test-roles/pull-request/25 so the current roles will take their default artifacts directory from $TEST_ARTIFACTS--using "./artifacts" if not specified. Thus, specifying "-e artifacts=..." on the ansible-playbook command line is no longer necessary--but can still be provided as an override if desired.
Would you please let me know what you think about PR#25, Stef?
I think it is useful for debugging, testing and making this easier to work with.
However the canonical way for a testing system to invoke a playbook is to set both the relevent environment variables *and* the ansible variables. The spec states this clearly:
https://fedoraproject.org/wiki/Changes/InvokingTests
I'd be happy to take a shot at implementing the above improvements. Then, I think we have something ready to tag and package as v2.0.
Wonderful.
I would also be interested in seeing the example "test.yml" you refer to on the wiki page. I couldn't find it checked in anywhere.
See the gzip git repo. But you can typically substitute test_local.yml.
Got it. I thought maybe you had come up with a new and interesting standard format for a master test playbook. :)
Heh, no. We really should be reaching the point where less changes, and not more.
As a matter of fact all the old ways of launching an image such as test_cloud.yml still work. We should decide whether we want to deprecate them and when.
Stef
Also, do you have any thoughts on how to have a single test.yml playbook cleanly pick and choose amongst which tests to run in which environments? Some tests simply won't run in a container and/or cloud image, and so should be skipped when they can't run properly. I also imagine there could be tests that should be run ONLY in a container and/or cloud image. I attempted to implement something like this with the "selector" role (https://pagure.io/standard-test-roles/blob/master/f/roles/standard-test-sele...), but it feels rather clumsy to use (eg., https://upstreamfirst.fedorainfracloud.org/sed/blob/master/f/test_local.yml).
I believe we're brainstorming this today. We could factor this aspect in.
Stef
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org
On 07/20/2017 12:17 AM, Stef Walter wrote:
On 19.07.2017 22:56, Merlin Mathesius wrote:
...
To further make use of the $TEST_ARTIFACTS environment variable, I have raised https://pagure.io/standard-test-roles/pull-request/25 so the current roles will take their default artifacts directory from $TEST_ARTIFACTS--using "./artifacts" if not specified. Thus, specifying "-e artifacts=..." on the ansible-playbook command line is no longer necessary--but can still be provided as an override if desired.
Would you please let me know what you think about PR#25, Stef?
I think it is useful for debugging, testing and making this easier to work with.
However the canonical way for a testing system to invoke a playbook is to set both the relevent environment variables *and* the ansible variables. The spec states this clearly:
Yes, it is stated in the spec. However, it is not show in the examples at https://fedoraproject.org/wiki/CI/Tests.
However, it is awkward and redundant to be required to specify the same parameter in multiple ways. While that shouldn't be a problem for an automated testing system, it does makes it more difficult and confusing for test developers. It's probably worth trying to eliminate the test subjects parameter redundancy requirement, too.
I'd be happy to take a shot at implementing the above improvements. Then, I think we have something ready to tag and package as v2.0.
Wonderful.
Do you feel the current master branch of standard-test-roles is in good shape to release as v2.0?
I would also be interested in seeing the example "test.yml" you refer to on the wiki page. I couldn't find it checked in anywhere.
See the gzip git repo. But you can typically substitute test_local.yml.
Got it. I thought maybe you had come up with a new and interesting standard format for a master test playbook. :)
Heh, no. We really should be reaching the point where less changes, and not more.
Agreed. I just wanted to see if you made any changes.
As discussed during brainstorming yesterday, we probably will want to make some updates to the "template" tests.yml so only tests applicable to the current environment are attempted to be run...
As a matter of fact all the old ways of launching an image such as test_cloud.yml still work. We should decide whether we want to deprecate them and when.
Yup. I agree.
Stef
Merlin
On 20.07.2017 16:13, Merlin Mathesius wrote:
On 07/20/2017 12:17 AM, Stef Walter wrote:
On 19.07.2017 22:56, Merlin Mathesius wrote:
...
To further make use of the $TEST_ARTIFACTS environment variable, I have raised https://pagure.io/standard-test-roles/pull-request/25 so the current roles will take their default artifacts directory from $TEST_ARTIFACTS--using "./artifacts" if not specified. Thus, specifying "-e artifacts=..." on the ansible-playbook command line is no longer necessary--but can still be provided as an override if desired.
Would you please let me know what you think about PR#25, Stef?
I think it is useful for debugging, testing and making this easier to work with.
However the canonical way for a testing system to invoke a playbook is to set both the relevent environment variables *and* the ansible variables. The spec states this clearly:
Yes, it is stated in the spec. However, it is not show in the examples at https://fedoraproject.org/wiki/CI/Tests.
However, it is awkward and redundant to be required to specify the same parameter in multiple ways. While that shouldn't be a problem for an automated testing system, it does makes it more difficult and confusing for test developers. It's probably worth trying to eliminate the test subjects parameter redundancy requirement, too.
Indeed. Can we just use environment variables? That would be nice. What would that look like. For example here in this simple example:
https://fedoraproject.org/wiki/CI/Tests#Writing_a_new_test
I'd be happy to take a shot at implementing the above improvements. Then, I think we have something ready to tag and package as v2.0.
Wonderful.
Do you feel the current master branch of standard-test-roles is in good shape to release as v2.0?
I think so. Although I believe we're currently backwards compatible, it may be worth tagging v1.1 ... but I'm probably straying into bikeshedding territory at this point :S
I would also be interested in seeing the example "test.yml" you refer to on the wiki page. I couldn't find it checked in anywhere.
See the gzip git repo. But you can typically substitute test_local.yml.
Got it. I thought maybe you had come up with a new and interesting standard format for a master test playbook. :)
Heh, no. We really should be reaching the point where less changes, and not more.
Agreed. I just wanted to see if you made any changes.
As discussed during brainstorming yesterday, we probably will want to make some updates to the "template" tests.yml so only tests applicable to the current environment are attempted to be run...
Exactly, the Ansible tags stuff will be another change in the spec that lands. So I guess another round of change to the spec is expected.
That should give us the leeway to make the above environment vs. ansible variable changes. But lets not wait on releasing a new standard-test-roles.
Cheers,
Stef
As a matter of fact all the old ways of launching an image such as test_cloud.yml still work. We should decide whether we want to deprecate them and when.
Yup. I agree.
Stef
Merlin _______________________________________________ CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org
On 07/20/2017 11:59 AM, Stef Walter wrote:
On 20.07.2017 16:13, Merlin Mathesius wrote:
On 07/20/2017 12:17 AM, Stef Walter wrote:
On 19.07.2017 22:56, Merlin Mathesius wrote:
...
To further make use of the $TEST_ARTIFACTS environment variable, I have raised https://pagure.io/standard-test-roles/pull-request/25 so the current roles will take their default artifacts directory from $TEST_ARTIFACTS--using "./artifacts" if not specified. Thus, specifying "-e artifacts=..." on the ansible-playbook command line is no longer necessary--but can still be provided as an override if desired.
Would you please let me know what you think about PR#25, Stef?
I think it is useful for debugging, testing and making this easier to work with.
However the canonical way for a testing system to invoke a playbook is to set both the relevent environment variables *and* the ansible variables. The spec states this clearly:
Yes, it is stated in the spec. However, it is not show in the examples at https://fedoraproject.org/wiki/CI/Tests.
However, it is awkward and redundant to be required to specify the same parameter in multiple ways. While that shouldn't be a problem for an automated testing system, it does makes it more difficult and confusing for test developers. It's probably worth trying to eliminate the test subjects parameter redundancy requirement, too.
Indeed. Can we just use environment variables? That would be nice. What would that look like. For example here in this simple example:
Yes, that would be nice. I can take a closer look at how that might play out.
I'd be happy to take a shot at implementing the above improvements. Then, I think we have something ready to tag and package as v2.0.
Wonderful.
Do you feel the current master branch of standard-test-roles is in good shape to release as v2.0?
I think so. Although I believe we're currently backwards compatible, it may be worth tagging v1.1 ... but I'm probably straying into bikeshedding territory at this point :S
I consider the inclusion of the inventory scripts worthy of bumping to v2.0. :)
Yes, everything should be backwards compatible when using the old calling conventions. However, I discovered that the standard-test-rhts role is not currently compatible with a docker image created using the dynamic inventory script. This is due to the restraint client requiring sshd and restraintd to be spun up--neither of which are currently done. But I can look at that later without holding up the package release.
However, there is one more thing I'd like to see included in the release: the ability to diagnose problems inside the containers/VMs spun up by the inventory scripts. The inventory scripts were doing such a good job of cleaning up after themselves that there was no opportunity to shell in and poke around. (I needed this to investigate the cause of the compatibility problem between the rhts role and docker inventory container I just mentioned.) So, I added hooks to both the qcow2 and docker inventory scripts to pause before cleaning up if TEST_DIAGNOSE_INVENTORY=1 is set in the environment.
https://pagure.io/standard-test-roles/pull-request/26
I would also be interested in seeing the example "test.yml" you refer to on the wiki page. I couldn't find it checked in anywhere.
See the gzip git repo. But you can typically substitute test_local.yml.
Got it. I thought maybe you had come up with a new and interesting standard format for a master test playbook. :)
Heh, no. We really should be reaching the point where less changes, and not more.
Agreed. I just wanted to see if you made any changes.
As discussed during brainstorming yesterday, we probably will want to make some updates to the "template" tests.yml so only tests applicable to the current environment are attempted to be run...
Exactly, the Ansible tags stuff will be another change in the spec that lands. So I guess another round of change to the spec is expected.
That should give us the leeway to make the above environment vs. ansible variable changes. But lets not wait on releasing a new standard-test-roles.
Indeed.
Cheers,
Stef
As a matter of fact all the old ways of launching an image such as test_cloud.yml still work. We should decide whether we want to deprecate them and when.
Yup. I agree.
Stef
Merlin
-
Merlin Mathesius -
Stef Walter