CI June 2017

ci@lists.fedoraproject.org

8 participants
7 discussions

Using Ansible inventory scripts for Test Subjects
by Stef Walter 10 Jul '17

10 Jul '17

Summary: I'd like to use Ansible "inventory" [0] to describe test subjects to be tested: http://docs.ansible.com/ansible/intro_inventory.html I've done initial exploratory work on this, pull requests below, more to follow. In our standard test invocation spec, we refer to "test subjects" as the thing being tested: https://fedoraproject.org/wiki/Changes/InvokingTests Mechanisms related to these test subjects have been poorly thought out in the spec. This is my fault. Now that we're trying to implement this, I think we can take what we've learned and update the spec. Two areas of interest: 1. Transforming test subjects into something testable by Ansible By using an Ansible inventory directory we can have inventory scripts launch or prepare a system ready for the Ansible based tests to run against it. This directory is tests/subjects The CI testing system will launch the tests like this: # TEST_SUBJECTS='/path/to/atomic.qcow2 /path/to/sed.rpm' \ ansible-playbook -i tests/subjects tests/tests.yml The effect of the above command would be to run tests/tests.yml twice, once against the qcow2 image and once against installed rpms (in-situ testing). Ansible supports having a directory as its inventory. All executable files in that directory are asked to produce inventory. Use of an environment variable as a way to pass information to Ansible inventory scripts is standard practice. Although each tests *could* have their own inventory scripts, these will commonly be shared. I've started a pull request here related to such shared default scripts: https://pagure.io/standard-test-roles/pull-request/9 You can play with these scripts. One launches a qcow2 image as a VM and makes that VM available to Ansible via inventory. A second inventory script installs RPMs and then tells Ansible to run locally. Another not yet written inventory script would launch a docker image and tell Ansible how to connect to it. Another would configure a module repo ... and so on. 2. A dist-git repo should describe which test subjects are applicable Following on from the above the tests/subjects directory should either contain executable inventory scripts that the tests would likely use to parse $TEST_SUBJECTS into something that Ansible can execute tests against. $TEST_SUBJECTS is a space separated list of paths of things to test. This is a likely bike shed topic, and I'm open to ideas here. Each inventory script in tests/subjects directory consumes some different part of the environment variable(s). Most tests will *not* want to write their own inventory scripts and will instead just include symlinks to well known inventory scripts in standard-test-roles. https://pagure.io/standard-test-roles/pull-request/9 The symlinks are necessary, as the test *must* be in control of describing which types of subjects, and thus Ansible inventory it supports. Some advanced tests (such as the Cockpit tests or IPA tests) would launch more complex local inventory and take control of what inventory is reported to Ansible. On the spec side, I strongly believe that both of these things should remain in the firm control of the tests stored in the dist-git repo. While still having useful shared code to implement the spec. Related: The ansible_connection=local nonsense in the current roles and spec would be dropped. I aim watch for discussion here on this topic for the next days or two, and create a new wiki page with an updated spec after that: https://fedoraproject.org/wiki/Changes/InvokingTestsAnsibleTwo Cheers, Stef

2 5

Landing page
by Stef Walter 21 Jun '17

21 Jun '17

Hi guys, I started a landing page gathering info about this effort that I was aware of: https://fedoraproject.org/wiki/CI It's independent of any specific Fedora change, and meant to answer questions and get people started. I don't yet have examples of how to write or port a test ... but I'll work on that and display it prominently when I do. That's probably the part that will interest the most people (like packagers). Cheers, Stef

3 3

dropping privileges during test suite execution
by Kamil Paral 21 Jun '17

21 Jun '17

InvokingTestsAnsible <https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible> says: The playbook and its test suite or test framework: SHOULD drop privileges appropriately if the test suite should be run as non-root. I'm not clear how you do that. The test suite should not assume any particular environment. So I can't easily switch to a known non-root user account, because such account might not exist. Am I expected to switch to a predefined locked account like 'nobody'? Or is there a different way to drop privileges? My envisioned use case: I want to run or develop a very simple test, e.g. rpmlint. I want to run it locally on my system, because it's the easiest hassle-free way. I know it's non-destructive and basically harmless, but the test system mandates it's run as root (according to the spec). I feel uneasy about running even a harmless test as root, so this test will try to drop privileges immediately after start. But how exactly does it do that? Thanks, Kamil

2 1

Re: Reusing the URL https://fedoraproject.org/wiki/CI
by Stef Walter 18 Jun '17

18 Jun '17

Awesome. Thanks! I've started off the page. Stef On 18.06.2017 18:51, Abhradip Mukherjee wrote: > Hi Stef, > > Thanks for reaching out. Please go ahead. > > I am also currently working in the DevOps domain as a solution architect > and have been working on RedHat technologies as well. > > Would love to contribute in the same myself. > > Regards, > Abhradip > > > On Jun 18, 2017 22:05, "Stef Walter" <stefw(a)redhat.com > <mailto:stefw@redhat.com>> wrote: > > Hi Abhradip, > > I noticed back in 2009 you created the following page on the Fedora > wiki: > > https://fedoraproject.org/wiki/CI <https://fedoraproject.org/wiki/CI> > > Since that page hasn't had much activity, and since "CI" is often an > abbreviation for "Continuous Integration" a testing and validation > process ... I'd like to ask, > > Would it work for you if I reused that page as a landing page to > describe continuous integration and testing efforts in Fedora? > > Thanks for considering, > > Stef > >

2 1

Reusing the URL https://fedoraproject.org/wiki/CI
by Stef Walter 18 Jun '17

18 Jun '17

Hi Abhradip, I noticed back in 2009 you created the following page on the Fedora wiki: https://fedoraproject.org/wiki/CI Since that page hasn't had much activity, and since "CI" is often an abbreviation for "Continuous Integration" a testing and validation process ... I'd like to ask, Would it work for you if I reused that page as a landing page to describe continuous integration and testing efforts in Fedora? Thanks for considering, Stef

1 0

Invoking Tests and the Ansible Interface
by Tim Flink 13 Jun '17

13 Jun '17

As we're working more with the test invocation spec [1] and the ansible implementation [2], there are a few things which aren't 100% clear to me and I'm hoping to get some clarification on. Why is "ansible_connection=local" a requirement for the testing system? Why is obtaining the test subject solely the responsibility of the testing system? This seems like something that could potentially have different requirements for different usecases. For example, something like abicheck needs to grab previous builds in addition to the latest build of a given package. I don't understand what we're trying to accomplish with all the test_<subject>.yml. Is the idea that we'd run test_oci.yml for every package used in building a container against that container? What is the point in having test_local.yml and test_rpm.yml? I remember the idea of in-situ tests being part of the discussion but I thought that we were pretty limited on what we could do with those since we're not going to be modifying rpm like debian was able to get changes made for .deb packages. Thanks, Tim [1] https://fedoraproject.org/wiki/Changes/InvokingTests [2] https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible

3 4

Invoking Tests via Ansible
by Stef Walter 08 Jun '17

08 Jun '17

Looks like Ansible has been chosen as the way we're going to invoke tests stored in dist-git. https://fedoraproject.org/wiki/Changes/InvokingTests https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible Even for those who voted for alternatives, the good news is that almost all of both the packaged tests and the autopkgtest style tests can be wrapped in the Ansible style tests. I did some work to try out the Ansible proposal and finish it up where there were rough edges or mysterious parts. I talked about these changes with Pingou, Ari, and Martin. If you're interested in the details of what was changed or finished up in the Ansible proposal: https://fedoraproject.org/w/index.php?title=Changes%2FInvokingTestsAnsible&… In addition some examples have been written. In particular, the sed tests here work both in-situ and against a composed Fedora Atomic host image: https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible#Examples https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible#Example:_Beaker… There'll be more updates to the examples on those pages, including finishing them up for the standard as it is, and making some parts of them more Ansibley. Cheers, Stef

4 8

2024

2023

2022

2021

2020

2019

2018

2017

CI June 2017