CI January 2018

ci@lists.fedoraproject.org

5 participants
2 discussions

Code review for new CI test
by Tom Stellard 12 Feb '18

12 Feb '18

Hi, I wrote a CI test for the clang package, and I was wondering if someone could review it and let me know if I have done it correctly: https://src.fedoraproject.org/rpms/clang/pull-request/6 Thanks, Tom

2 1

Share Test Code
by Petr Splichal 31 Jan '18

31 Jan '18

Hi! Current Fedora CI documentation suggests: "The tests to be executed are stored in the dist-git repositories. The tests are stored or wrapped along-side the spec files..." [0] While working on porting tests to Fedora CI, several times we've noted concerns from developers/maintainers about placing test code directly into dist-git repositories. A common question is how to efficiently maintain tests & minimize test code duplication. There are some nice real-life examples available: Examples There are several shells which implement the POSIX specification: bash, ksh, mksh, zsh, dash. All of them share a significant amount of test coverage and it does not make sense to commit & maintain identical tests in five different repositories (+ possible branches). See the pull request [1] for a bit of context. Another example is Ruby: With about 80 packages related to Ruby on Rails it would be useful and efficient to have a single place for integration tests which verify that the framework is correctly working after updating any of these packages. Conversely, maintaining those tests in 80 repos would be a tedious task. Proposal: Share! So this is where the idea of shared test repository comes from. In general, tests define how the software works and the basic functionality of many packages doesn’t change that often. We try hard to keep the backward compatibility where possible. Thus it seems natural that, for such components, tests guarding the spec could change at a slower pace than the distribution branches. Main goals * Package source and tests are linked in a discoverable and unambiguous way (dist-git) * Prevent test code duplication (minimize test maintenance) * Catch incompatibilities early How it could work? A new namespace "tests" would be created in Fedora as suggested in [2], test code would reside there and the tests in the rpms dist git could directly reference a specific version of the tests to be run, e.g. with a pre_task: pre_tasks: - name: Fetch the tests git: repo: https://src.fedoraproject.org/tests/shell.git dest: tests version: master Also, standard-test-roles [3] should be enhanced to make it really easy to fetch tests from a remote repo. Ansible git module is quite simple but with standard-test-roles it could be even shorter. The tests.yml file could look like this (we could support fetching from multiple repositories): - role: standard-test-beakerlib tags: - classic - container - atomic tests: - smoke repo: - https://src.fedoraproject.org/tests/shell.git version: master A consistent location for shared tests is a good source of examples and helps drive automation in the future, e.g. trigger CI jobs on test changes. For stable packages providing specification-like functionality a single master branch could be used, for those components which change more, dist-git tests.yml could pick a proper branch or even a specific commit (if having concerns about updated tests breaking the continuous integration). This seems to give a nice flexibility while at the same time minimizing test code duplication. Referencing specific tests from rpms dist git ties tests and code together very well. Storing the test code in the tests namespace is only one option, any git repo accessible from the CI pipeline would suffice. Just One Place? We could possibly go one more step further: Shall we aim at having just a single test code for testing both Fedora and Red Hat Enterprise Linux? At least for those cases where this is possible: There might be some licensing or compatibility issues. For example, speaking about Beaker: restraint harness can also be instructed to fetch tests directly from git so the identical test code could perhaps be used for running tests in Beaker as well. A recent experience of a libselinux pull request [4] shows how cumbersome it can be to keep both upstream and downstream version of the test code in sync. Wouldn't it be great to have everything at a single place and keep the test code up-to-date only there? Instead of doing several rounds of "pull request - give feedback - update internal tests - update the pull request - find another issue - give feedback..." with many unnecessary steps and reminders just directly collaborating on a single test repository? When downstream testing is just a branch of upstream testing, we can use our standard open source workflows. Just as in upstream, there would likely be a downstream shared git repository that branches from the upstream one. Bonus As a bonus we would keep the dist-git for such repositories a bit more consistent as a place for metadata only: Build metadata (spec file = how to build the package) and test metadata (tests.yml = how to test the package). Plus we would get a more streamlined test maintenance as QE and volunteers willing to help with improving the test coverage could be granted direct access to individual package tests repo. Not every QE is a Fedora packager so contributing to dist-git is not that straightforward. Notes There are cons of this approach as well. It isn’t possible to have new feature code and tests submitted in a single commit or triggering test execution for every change in tests (CI for tests). There is a need for handling relevance of individual tests for different environments. But the maintenance simplification seems to be worth it, based on experience of the BaseOS QE team which has been maintaining thousands of tests for many years and using them for testing large number of supported versions of Red Hat Enterprise Linux. The overall benefits also depend on which test types we are speaking about. Tests should always be as far upstream as possible. Unit tests generally make more sense in upstream. For integration tests it might make more sense to live in the shared Fedora tests repository. Recommendations about which tests fit well into this "tests" namespace will evolve, but it will be up to each package to choose where to place its tests. So? What are your thoughts on this? Does it make sense to you? psss... [0] https://fedoraproject.org/wiki/CI#Tests_locations [1] https://src.fedoraproject.org/rpms/bash/pull-request/1 [2] https://pagure.io/fedora-infrastructure/issue/6478 [3] https://pagure.io/standard-test-roles/ [4] https://src.fedoraproject.org/rpms/libselinux/pull-request/1

6 6

2024

2023

2022

2021

2020

2019

2018

2017

CI January 2018