July 2017 - CI - Fedora mailing-lists

Using tags to indicate context in which tests are able to run
by Stef Walter 26 Jul '17

26 Jul '17

Many of the tests that are being written for dist-git packages or modules have special assumptions about where they can execute. For example, tests may interact with packages not present in an Atomic Host, or run things that don't make sense in a container. While some of this info can be inferred from the "test subject" in the spec [0], there was a need for a rigorous way for packagers to what a test was designed for. To that end, Miroslav, Merlin and I worked on a change to the spec, which would allow for tests to indicate in which context they are designed to operate. We'd use Ansible Tags for this. So far there are three tags "atomic", "container" and "classic". Here's the updated page: https://fedoraproject.org/wiki/Changes/InvokingTestsThree And here's the specific changes: https://fedoraproject.org/w/index.php?title=Changes%2FInvokingTestsThree&di… And here's an example of it in use: https://upstreamfirst.fedorainfracloud.org/bash/pull-request/2 Broadly speaking the changes are backwards compatible, although tests will want to start to tag themselves to at least operate in the "classic" context. Does this make sense? If so, next steps are to: 1. Merge those changes back into the main spec: https://fedoraproject.org/wiki/Changes/InvokingTests 2. Update the tutorial style documentation: https://fedoraproject.org/wiki/CI/Tests 3. Work through the current set of staged tests and put tags in. Cheers, Stef [0] https://fedoraproject.org/wiki/Changes/InvokingTests

1 1

Re: Storage requirements for CI
by Paul W. Frields 25 Jul '17

25 Jul '17

On Tue, Jul 25, 2017 at 12:12:30PM -0400, Dennis Gregorovic wrote: > > > On 07/25/2017 10:59 AM, Paul W. Frields wrote: > > I'd meant to raise this question last week but it turned out several > > folks were out of pocket who'd probably want to discuss. One of the > > aspects of continuous integration[1] that impacts my team is the > > storage requirement. How much storage is required for keeping test > > results, composed trees and ostrees, and other artifacts? What is > > their retention policy? > > > > A policy of "keep everything ever made, forever" clearly isn't > > scalable. We don't do that in the non-CI realm either, e.g. with > > scratch builds. I do think that we must retain everything we > > officially ship, that's well understood. But atop that, anything we > > keep costs storage, and over time this storage costs money. So we > > need to draw some reasonable line that balances thrift and service. > > > > A. Retention > > ============ > > > > The second question is probably a good one to start with, so we can > > answer the first. So we need to answer the retention question for > > some combination of: > > > > 1. candidate builds that fail a CI pipeline > > 2. candidate builds that pass a CI pipeline > > 3. CI composed testables > > * a tree, ISO, AMI, other image, etc. that's a unit > > * ostree change which is more like a delta (AIUI) > > 4. CI generated logs > > 5. ...other stuff I may be forgetting > > The other big bucket is packages in the buildroot used to build the > builds. You may want to keep these as well if there is a desire to > be able to rebuild packages at a later point. Thanks for that. My bet is we'd want to set the retention dial for those no higher than "when that release goes EOL." > > My general thoughts are that these things are kept forever: > > > > * (2), but only if that build is promoted as an update or as part of a > > shipped tree/ostree/image > > * (3), but only if the output is shipped to users > > * (4), but only if corresponding to an item in (2) or (3) > > > > Outside that, artifacts and logs are kept only for a reasonable amount > > of troubleshooting time. Say 30 days, but I'm not too worried about > > the actual time period. It could be adjusted based on factors we have > > yet to encounter. > > How does this proposal compare the existing practice in Fedora? My initial guess is, pretty well in concept -- but that the *practice* is that we've not been very aggressive about trimming ancient shipped stuff. How many people are liable to seek Fedora <= 18 releases at this point, for example? -- Paul W. Frields http://paul.frields.org/ gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717 http://redhat.com/ - - - - http://pfrields.fedorapeople.org/ The open source story continues to grow: http://opensource.com

1 0

Storage requirements for CI
by Paul W. Frields 25 Jul '17

25 Jul '17

I'd meant to raise this question last week but it turned out several folks were out of pocket who'd probably want to discuss. One of the aspects of continuous integration[1] that impacts my team is the storage requirement. How much storage is required for keeping test results, composed trees and ostrees, and other artifacts? What is their retention policy? A policy of "keep everything ever made, forever" clearly isn't scalable. We don't do that in the non-CI realm either, e.g. with scratch builds. I do think that we must retain everything we officially ship, that's well understood. But atop that, anything we keep costs storage, and over time this storage costs money. So we need to draw some reasonable line that balances thrift and service. A. Retention ============ The second question is probably a good one to start with, so we can answer the first. So we need to answer the retention question for some combination of: 1. candidate builds that fail a CI pipeline 2. candidate builds that pass a CI pipeline 3. CI composed testables * a tree, ISO, AMI, other image, etc. that's a unit * ostree change which is more like a delta (AIUI) 4. CI generated logs 5. ...other stuff I may be forgetting My general thoughts are that these things are kept forever: * (2), but only if that build is promoted as an update or as part of a shipped tree/ostree/image * (3), but only if the output is shipped to users * (4), but only if corresponding to an item in (2) or (3) Outside that, artifacts and logs are kept only for a reasonable amount of troubleshooting time. Say 30 days, but I'm not too worried about the actual time period. It could be adjusted based on factors we have yet to encounter. B. Storage - How much? ====================== To get an idea of what this might look like, I think we might make estimates based on: * the number of builds currently happening per day * how many of these builds are within the definition for an officially shipped thing (like Atomic Host, Workstation, Server, etc.) * The average size of the sources + binaries, summed out over the ways we deliver them (SRPM + RPM, ostree binary, binary in another image), and multiplied out by arches * Then sum this out over the length of a Fedora release This is the part I think will need information from the rel-eng and CI contributors, working together. My assumption is there are gaping holes in this concept, so don't take this as a full-on proposal. Rather, looking for folks to help harden the concepts and fill in the missing pieces. I don't think we need a measurement down to the single GB; a broad estimate in 100s of GB (or even at the 1 TB order of magnitude) is likely good enough. I'm setting the follow-up to infrastructure(a)lists.fedoraproject.org, since that team has the most information about our existing storage and constraints. -- Paul W. Frields http://paul.frields.org/ gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717 http://redhat.com/ - - - - http://pfrields.fedorapeople.org/ The open source story continues to grow: http://opensource.com

1 0

Dependencies for standard-test-roles
by Stef Walter 25 Jul '17

25 Jul '17

The Ansible dynamic inventory scripts need certain dependencies in order to run successfully. I think we can handle this with Recommends in the standard-test-roles package: diff --git a/standard-test-roles.spec b/standard-test-roles.spec index 05d6cf5..5970bb9 100644 --- a/standard-test-roles.spec +++ b/standard-test-roles.spec @@ -10,6 +10,9 @@ Source0: http://releases.pagure.org/%{name}/%{name}-%{version}.tar.gz BuildArch: noarch BuildRequires: coreutils Requires: ansible +Recommends: docker +Recommends: genisoimage +Recommends: qemu-system-x86 %description Shared Ansible roles to support the Standard Test Interface as described What do you think? Once we need package this for RHEL or older Fedora we can make these a hard dependency for those operating systems using an %if block. Cheers, Stef

2 3

Re: Documentation on adding tests to dist-git
by Stef Walter 25 Jul '17

25 Jul '17

On 25.07.2017 08:11, Martin Pitt wrote: > Hello Neal, > > Neal Gompa [2017-07-20 8:58 -0400]: >> On Thu, Jul 20, 2017 at 8:53 AM, Stef Walter <stefw(a)redhat.com> wrote: >>> As a follow up to the 'Running tests' documentation I did, I've added >>> some documentation on adding new tests to dist-git: >>> >>> https://fedoraproject.org/wiki/CI/Tests#Adding_tests >>> >>> Anyone up for adding some documentation for beakerlib and restraint >>> wrapping? >>> >>> Merlin, Tim, do you guys think we should add an "example" repo to >>> upstreamfirst.fedorainfracloud.org with this example stuff in it? >>> >> >> It would be greatly helpful if there was some examples to go with the >> documentation. In part, I have been interested in the idea of having >> something "autopkgtests-ish" for snapd in Fedora. The package is >> super-complicated and I'd like to have some dedicated sanity checks >> for it whenever builds are made for it. > > Not sure what you mean, but the above wiki page mostly consists of examples, > and it also links to a concrete practical gzip test > (https://upstreamfirst.fedorainfracloud.org/gzip/c/d56637d71a8e8). That was > taken directly from the Ubuntu package's autopkgtest, i. e. the actual test > script is exactly the same. The meta-data differs between autopkgtest and > ansible, and arguably the ansible one currently contains some boilerplate[1], > but the spirit is not too different. > > snapd uses autopkgtests both for distro gating and for gating upstream PRs, so > these tests should work in a Fedora context too? I think so. I think a standard-test-autopkgtest Ansible role could be contributed to the standard-test-roles package [0] which just uses autopkgtest control files and staging and invocation logic directly: https://pagure.io/standard-test-roles This is similar to how there is a standard-test-beakerlib role and standard-test-rhts roles. Cheers, Stef > Thanks, > > Martin > > [1] I hope that the installation and execution of the test can be done in a > generic way without having to repeat this in every individual package - but > keep in mind that this is just a PoC. [0] https://pagure.io/standard-test-roles

1 0

Launching test subjects using inventory scripts
by Stef Walter 20 Jul '17

20 Jul '17

I've added documentation about launching test subjects using Ansible Dynamic inventory scripts. Test subjects are the thing to be tested. For example a QCow2, container image, or often a set of RPMs. These get launched into a form that Ansible (and thus the test playbooks) can run against them: The tutorial style documentation is here: https://fedoraproject.org/wiki/CI/Tests More documentation will be landing here about writing tests, preparing a dist-git repo for tests, and so on. The information about the spec is here: https://fedoraproject.org/wiki/Changes/InvokingTests Note that older version of the spec used to have a test_local.yml file, and test subject specific playbooks. These test subject specific playbooks are no longer necessary with the inventory. In addition test_local.yml may be used in place of tests.yml in the documentation. Once more people have looked over this, I'll open some pull requests to rename test_local.yml to tests.yml. For the documentation to work, we do need to package the inventory scripts that are already present in standard-test-roles. In addition, I've opened a pull request to standard-test-roles for adding support for launching docker containers as dynamic inventory: https://pagure.io/standard-test-roles/pull-request/22 Cheers, Stef

2 7

Documentation on adding tests to dist-git
by Stef Walter 20 Jul '17

20 Jul '17

As a follow up to the 'Running tests' documentation I did, I've added some documentation on adding new tests to dist-git: https://fedoraproject.org/wiki/CI/Tests#Adding_tests Anyone up for adding some documentation for beakerlib and restraint wrapping? Merlin, Tim, do you guys think we should add an "example" repo to upstreamfirst.fedorainfracloud.org with this example stuff in it? Cheers, Stef

3 2

Re: CI/CD requirements
by Paul W. Frields 18 Jul '17

18 Jul '17

Hi folks, I sent this email earlier to the infrastructure@ list but realized it might also be good to notify here: On Tue, Jul 11, 2017 at 12:18:55PM -0400, Paul W. Frields wrote: > One of the things I've been working on (along with several other > people, including mattdm, dgilmore and pingou) is sussing out Fedora > requirements on a CI/CD pipeline. I published the results to this > page: > > https://fedoraproject.org/wiki/Fedora_requirements_for_CI_and_CD > > Since Fedora infra is a stakeholder (along with release engineering, > QA, etc.), my thought is this page helps us clarify requirements and > avoid surprises down the road. Also this might be useful to show to > e.g. FESCo as a joint policy. > > Rather than just editing the page willy nilly, based on the number of > people it's already been through, it would be great if folks could use > the "discussion" (Talk) page to suggest changes. But I have a watch > on the page and can help resolve issues. Input is welcome on the wiki Discussion page, although hopefully there are no surprises here since this doc was assembled by a pretty broad team. -- Paul W. Frields http://paul.frields.org/ gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717 http://redhat.com/ - - - - http://pfrields.fedorapeople.org/ The open source story continues to grow: http://opensource.com

1 0

FYI changing of filename in fedora-atomic git repo
by Dusty Mabe 12 Jul '17

12 Jul '17

FYI: https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org…

1 0

Using Ansible inventory scripts for Test Subjects
by Stef Walter 10 Jul '17

10 Jul '17

Summary: I'd like to use Ansible "inventory" [0] to describe test subjects to be tested: http://docs.ansible.com/ansible/intro_inventory.html I've done initial exploratory work on this, pull requests below, more to follow. In our standard test invocation spec, we refer to "test subjects" as the thing being tested: https://fedoraproject.org/wiki/Changes/InvokingTests Mechanisms related to these test subjects have been poorly thought out in the spec. This is my fault. Now that we're trying to implement this, I think we can take what we've learned and update the spec. Two areas of interest: 1. Transforming test subjects into something testable by Ansible By using an Ansible inventory directory we can have inventory scripts launch or prepare a system ready for the Ansible based tests to run against it. This directory is tests/subjects The CI testing system will launch the tests like this: # TEST_SUBJECTS='/path/to/atomic.qcow2 /path/to/sed.rpm' \ ansible-playbook -i tests/subjects tests/tests.yml The effect of the above command would be to run tests/tests.yml twice, once against the qcow2 image and once against installed rpms (in-situ testing). Ansible supports having a directory as its inventory. All executable files in that directory are asked to produce inventory. Use of an environment variable as a way to pass information to Ansible inventory scripts is standard practice. Although each tests *could* have their own inventory scripts, these will commonly be shared. I've started a pull request here related to such shared default scripts: https://pagure.io/standard-test-roles/pull-request/9 You can play with these scripts. One launches a qcow2 image as a VM and makes that VM available to Ansible via inventory. A second inventory script installs RPMs and then tells Ansible to run locally. Another not yet written inventory script would launch a docker image and tell Ansible how to connect to it. Another would configure a module repo ... and so on. 2. A dist-git repo should describe which test subjects are applicable Following on from the above the tests/subjects directory should either contain executable inventory scripts that the tests would likely use to parse $TEST_SUBJECTS into something that Ansible can execute tests against. $TEST_SUBJECTS is a space separated list of paths of things to test. This is a likely bike shed topic, and I'm open to ideas here. Each inventory script in tests/subjects directory consumes some different part of the environment variable(s). Most tests will *not* want to write their own inventory scripts and will instead just include symlinks to well known inventory scripts in standard-test-roles. https://pagure.io/standard-test-roles/pull-request/9 The symlinks are necessary, as the test *must* be in control of describing which types of subjects, and thus Ansible inventory it supports. Some advanced tests (such as the Cockpit tests or IPA tests) would launch more complex local inventory and take control of what inventory is reported to Ansible. On the spec side, I strongly believe that both of these things should remain in the firm control of the tests stored in the dist-git repo. While still having useful shared code to implement the spec. Related: The ansible_connection=local nonsense in the current roles and spec would be dropped. I aim watch for discussion here on this topic for the next days or two, and create a new wiki page with an updated spec after that: https://fedoraproject.org/wiki/Changes/InvokingTestsAnsibleTwo Cheers, Stef

2 5

2024

2023

2022

2021

2020

2019

2018

2017

CI July 2017