Summary: I'd like to use Ansible "inventory" [0] to describe test
subjects to be tested:
http://docs.ansible.com/ansible/intro_inventory.html
I've done initial exploratory work on this, pull requests below, more to
follow.
In our standard test invocation spec, we refer to "test subjects" as the
thing being tested:
https://fedoraproject.org/wiki/Changes/InvokingTests
Mechanisms related to these test subjects have been poorly thought out
in the spec. This is my fault. Now that we're trying to implement this,
I think we can take what we've learned and update the spec.
Two areas of interest:
1. Transforming test subjects into something testable by Ansible
By using an Ansible inventory directory we can have inventory
scripts launch or prepare a system ready for the Ansible based
tests to run against it. This directory is tests/subjects
The CI testing system will launch the tests like this:
# TEST_SUBJECTS='/path/to/atomic.qcow2 /path/to/sed.rpm' \
ansible-playbook -i tests/subjects tests/tests.yml
The effect of the above command would be to run tests/tests.yml
twice, once against the qcow2 image and once against installed
rpms (in-situ testing).
Ansible supports having a directory as its inventory. All
executable files in that directory are asked to produce
inventory.
Use of an environment variable as a way to pass information
to Ansible inventory scripts is standard practice.
Although each tests *could* have their own inventory scripts,
these will commonly be shared. I've started a pull request here
related to such shared default scripts:
https://pagure.io/standard-test-roles/pull-request/9
You can play with these scripts. One launches a qcow2 image as
a VM and makes that VM available to Ansible via inventory. A second
inventory script installs RPMs and then tells Ansible to run
locally.
Another not yet written inventory script would launch a docker image
and tell Ansible how to connect to it. Another would configure a
module repo ... and so on.
2. A dist-git repo should describe which test subjects are applicable
Following on from the above the tests/subjects directory should
either contain executable inventory scripts that the tests would
likely use to parse $TEST_SUBJECTS into something that Ansible
can execute tests against.
$TEST_SUBJECTS is a space separated list of paths of things
to test. This is a likely bike shed topic, and I'm open to ideas
here.
Each inventory script in tests/subjects directory consumes some
different part of the environment variable(s).
Most tests will *not* want to write their own inventory scripts
and will instead just include symlinks to well known inventory
scripts in standard-test-roles.
https://pagure.io/standard-test-roles/pull-request/9
The symlinks are necessary, as the test *must* be in control
of describing which types of subjects, and thus Ansible inventory
it supports.
Some advanced tests (such as the Cockpit tests or IPA tests) would
launch more complex local inventory and take control of what
inventory is reported to Ansible.
On the spec side, I strongly believe that both of these things should
remain in the firm control of the tests stored in the dist-git repo.
While still having useful shared code to implement the spec.
Related: The ansible_connection=local nonsense in the current roles and
spec would be dropped.
I aim watch for discussion here on this topic for the next days or two,
and create a new wiki page with an updated spec after that:
https://fedoraproject.org/wiki/Changes/InvokingTestsAnsibleTwo
Cheers,
Stef
Hi guys,
I started a landing page gathering info about this effort that I was
aware of:
https://fedoraproject.org/wiki/CI
It's independent of any specific Fedora change, and meant to answer
questions and get people started.
I don't yet have examples of how to write or port a test ... but I'll
work on that and display it prominently when I do. That's probably the
part that will interest the most people (like packagers).
Cheers,
Stef
InvokingTestsAnsible
<https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible> says:
The playbook and its test suite or test framework:
SHOULD drop privileges appropriately if the test suite should be run as
non-root.
I'm not clear how you do that. The test suite should not assume any
particular environment. So I can't easily switch to a known non-root user
account, because such account might not exist. Am I expected to switch to a
predefined locked account like 'nobody'? Or is there a different way to
drop privileges?
My envisioned use case: I want to run or develop a very simple test, e.g.
rpmlint. I want to run it locally on my system, because it's the easiest
hassle-free way. I know it's non-destructive and basically harmless, but
the test system mandates it's run as root (according to the spec). I feel
uneasy about running even a harmless test as root, so this test will try to
drop privileges immediately after start. But how exactly does it do that?
Thanks,
Kamil
Awesome. Thanks! I've started off the page.
Stef
On 18.06.2017 18:51, Abhradip Mukherjee wrote:
> Hi Stef,
>
> Thanks for reaching out. Please go ahead.
>
> I am also currently working in the DevOps domain as a solution architect
> and have been working on RedHat technologies as well.
>
> Would love to contribute in the same myself.
>
> Regards,
> Abhradip
>
>
> On Jun 18, 2017 22:05, "Stef Walter" <stefw(a)redhat.com
> <mailto:stefw@redhat.com>> wrote:
>
> Hi Abhradip,
>
> I noticed back in 2009 you created the following page on the Fedora
> wiki:
>
> https://fedoraproject.org/wiki/CI <https://fedoraproject.org/wiki/CI>
>
> Since that page hasn't had much activity, and since "CI" is often an
> abbreviation for "Continuous Integration" a testing and validation
> process ... I'd like to ask,
>
> Would it work for you if I reused that page as a landing page to
> describe continuous integration and testing efforts in Fedora?
>
> Thanks for considering,
>
> Stef
>
>
Hi Abhradip,
I noticed back in 2009 you created the following page on the Fedora wiki:
https://fedoraproject.org/wiki/CI
Since that page hasn't had much activity, and since "CI" is often an
abbreviation for "Continuous Integration" a testing and validation
process ... I'd like to ask,
Would it work for you if I reused that page as a landing page to
describe continuous integration and testing efforts in Fedora?
Thanks for considering,
Stef
As we're working more with the test invocation spec [1] and the
ansible implementation [2], there are a few things which aren't 100%
clear to me and I'm hoping to get some clarification on.
Why is "ansible_connection=local" a requirement for the testing system?
Why is obtaining the test subject solely the responsibility of the
testing system? This seems like something that could potentially have
different requirements for different usecases. For example, something
like abicheck needs to grab previous builds in addition to the latest
build of a given package.
I don't understand what we're trying to accomplish with all the
test_<subject>.yml. Is the idea that we'd run test_oci.yml for every
package used in building a container against that container?
What is the point in having test_local.yml and test_rpm.yml? I remember
the idea of in-situ tests being part of the discussion but I thought
that we were pretty limited on what we could do with those since we're
not going to be modifying rpm like debian was able to get changes made
for .deb packages.
Thanks,
Tim
[1] https://fedoraproject.org/wiki/Changes/InvokingTests
[2] https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible