Hi,

On Wed, Mar 11, 2020 at 6:14 PM Sudhir D sdharane@redhat.com wrote:

On 3/11/20 7:14 PM, Aleksandra Fedorova wrote:

...

Hi, all.

Here is the summary of CI-related work happening in Fedora.

If you have questions or topics to discuss you can also join Fedora CI SIG bi-weekly meeting. Next session is today in #fedora-ci IRC channel at 15:30 UTC

https://apps.fedoraproject.org/calendar/SIGs/#m9618

========================================================

### Dist-git tests support multipackage updates

You can define package tests in dist-git via STR format

https://docs.fedoraproject.org/en-US/ci/standard-test-roles/

Note that dist-git/STR tests are different from running %check tests in the rpm building phase. STR tests are executed in a clean virtual machine with proper setup of repositories for the latest Fedora Rawhide packages. This environment is better suited for integration tests, which need to test the installed package, not the sources of it.

Dist-git tests are fully compatible with the dynamic sidetag approach: if you create a dynamic sidetag for the multi-package update, test environment will be created with the content of the entire sidetag, not an individual package.

Status: Ready to Use Contact: Bruno Goncalves (bgoncalv) and #fedora-ci IRC channel.

### New test: rpminspect

There is a new rpminspect test running in Fedora Rawhide gating enabled by default for all packages in a non-blocking mode.

More details: https://github.com/rpminspect/rpminspect

Status: Ready to Use Contact: David Cantrell (dcantrell)

Thanks for sending out all this information, Aleksandra :)

I also want to call out Tim Flink (irc: tflink) from Fedora QA team who worked extensively on getting rpminspect to run in a way that its results could show up in bodhi.

I somehow took it for granted that everyone knows Tim is there.

Thanks for clearing that out.

...

### Tests for pull-requests via Zuul

Zuul team has enabled Zuul CI engine to test Pagure pull requests.

You can opt-in to Zuul CI per package.

On every pull-request Zuul will

• run a scratch build

• run rpminspect on that build

• run dist-git test defined in STR format(if available)

• provide comment in the pull-request

• wait for you to put an manual approval on the PR

• merge the PR

• you can also get Zuul to handle merge events, so that it will

automatically build the regular koji build, after the merge.

Zuul now has support for EPEL8 branches.

More details: https://fedoraproject.org/wiki/Zuul-based-ci

Status: Ready to use Contact: Fabien Boucher (fbo)

### Koschei as a gating test

With sidetag gating feature enabled it is now possible to run Koschei for each dynamic sidetag and make it a part of the gating process.

We do have Koschei deployed on Fedora infrastructure. There is on-going work by Mikolaj Izdebski to get it integrated with the Fedora Messaging, so that sidetags are submitted in Koschei when created.

Status: research and prototyping Contact: Serhii Turivnyi (sturivny)

### Infra change: new Jenkins master

New Jenkins master to run generic tests and inherit Taskotron pipelines.

Our current Jenkins master, which is used for dist-git tests, was not updated for some time and it is by design bundled to the pipeline it runs. So adding new pipelines and separating pipelines from the Jenkins master configuration is problematic.

The goal here is to have a Jenkins master setup which is easy to update. It will have a set of plugins pre-installed and configured for Fedora infrastructure endpoints, but jobs configuration will be applied to it independently.

More details: Current work is done on a Communishift project. Code will be available soon at https://github.com/fedora-ci

Status: WIP Contact: Jim Bair (jbair)

### Infra change: common repository and common format for generic tests

We are refactoring the Groovy pipeline library so it is better suited to run generic tests.

One of the goals is that generic tests are all run in the same way, and you don’t need to add a lot of new Groovy code to add a certain bash script as a generic test.

We’d like people to be able to contribute new generic tests without prior knowledge of the Jenkins internal setup.

Current focus is rpmdeplint and rpminspect pipelines.

More details: https://github.com/fedora-ci

Status: WIP Contact: Michal Srb (msrb)

### Infra change: ODCS composes

We are updating ODCS staging infrastructure to the latest ODCS release. Once the Fedora instractructure freeze is over, we will also update the ODCS production instance. This work is preparation for possible further use of ODCS to generate composes used by Fedora CI as well as main Fedora composes.

Status: WIP Contact: Jan Kaluza (jkaluza)

### Infra change: Testing Farm Service

Testing Farm Team is working on open-sourcing parts of the RH internal CI infrastructure as a service, which will be used by Fedora CI's general tests and functional tests pipeline. The main input of the service will be test definitions in the TMT/FMF format.

TMT documentation: https://tmt.readthedocs.io/en/latest/ (recently added testcloud + podman provisioner)

Code is hosted at GitLab: https://gitlab.com/testing-farm/

Status: WIP (preview May 2020, GA August 2020) Contact: Miroslav Vadkerti (mvadkert)

### CI and Gating documentation

There is a repository of CI documentation

https://pagure.io/fedora-ci/docs/

The docs are published at

https://docs.fedoraproject.org/en-US/ci/

Then there is another repository with docs on Rawhide Gating:

https://pagure.io/cpe/rawhide-gating-docs/

And result is available at:

https://docs.fedoraproject.org/en-US/rawhide-gating/

There are some rather deep or generic items there, which are not always suitable for newcomers and are not easy to consume.

What I think we need is a smaller scale how-to’s answering specific questions and implementing specific use-cases, which hook the CI and gating into the packager workflow.

If you have experience with sidetag gating or CI in Fedora and figured out the way how _you_ work with it, please share.

You can drop me a mail or write a draft page and send a pull request to one of the repositories. We will figure out later in which section to land it.

Status: Needs help Contact: Aleksandra Fedorova (bookwar)

### Testing of GitHub PRs via Packit / Testing Farm on Fedora/CentOS

Packit service makes it possible to test copr builds built from GitHub PRs on all Fedora released (including Rawhide), CentOS 6/7 and CentOS stream via Testing Farm. Note that the Testing Farm deployed for Packit is different from the one we are open sourcing, and once that is ready i will replace this one.

Documentation: https://packit.dev/testing-farm/

Status: In production (since August 2019) Contact: Miroslav Vadkerti (mvadkert)

========================================================

For any of those topics you can contact Fedora CI SIG at #fedora-ci IRC channel.

On 12. 03. 20 10:44, Zbigniew Jędrzejewski-Szmek wrote:

There's a mention somewhere that this destructively modifies the host machine, and using a VM is suggested. But not in the "Quick Start Guide". I'd expect this to be prominently mentioned at the top.

This together with:

Simply following the procedure as described does not work, the command fails with "This command has to be run under the root user.".

Makes running he tests extremely painful locally. I've opened a ticket for this:

https://pagure.io/fedora-ci/general/issue/4

(Opened 2 years ago according to Pagure.)

I even voted against gating change at FESCo because I consider the (non)usability of the CI a blocker. My vote against was not enough -- gating is not coupled with the CI and an argument was made that enabling the gating will add more users of the CI and it will get better. It didn't.

On Thu, Mar 12, 2020 at 03:30:18PM +0100, Aleksandra Fedorova wrote:

Hi,

On Thu, Mar 12, 2020 at 10:46 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:

...

Hi, thanks for the update, it seems nice progress is being made.

...

### Dist-git tests support multipackage updates

You can define package tests in dist-git via STR format

https://docs.fedoraproject.org/en-US/ci/standard-test-roles/

The docs have a "Quick Start Guide", but it describes a relatively complex case — the bash package which shares tests with other packages using a separate repo, which obviously adds another layer of complexity. (The repo we are looking at has a playbook which just clones another repo with another playbook, which then runs the playbooks from standard-test-roles. Wow.)

Simply following the procedure as described does not work, the command fails with "This command has to be run under the root user.". I guess 'sudo ...' might be the answer, but it's not obvious, because of the layers of indirection. (Should ansible's "become" be used instead?)

The docs also talks about many different kinds of tests... For example something using docker. In F32, docker doesn't even run out of the box, so it probably shouldn't be mentioned in any kind of beginner docs.

In the "Examples" section, there's an example for "did" package. I copy-pasted that, and after running it (under sudo), the command does not fail, but it seems the tests failed based on the text output. This made me realize that the docs don't describe how to check if the tests pass. (I hope reading the transcript is not the answer.)

There's a mention somewhere that this destructively modifies the host machine, and using a VM is suggested. But not in the "Quick Start Guide". I'd expect this to be prominently mentioned at the top.

After looking at the docs as they are, this all seems soooo complex. Maybe this complexity is unavoidable, but I would love to see an introductory howto, from beginning to the checking of the test results and hooking up to bodhi gating, for a _simple_ case, in particular no separate tests repo cloned over the network from a server.

Thanks for the feedback. I will revise a Quick Start doc and make updates to it.

And I have just pushed the doc I had in drafts for too long, please take a look

https://docs.fedoraproject.org/en-US/ci/how-to-add-dist-git-test/

Thanks, that looks very promising.

Dist-git tests are actually easy, I agree we made them look unnecessarily complex.

Zbyszek

Hi,

...

### Tests for pull-requests via Zuul

Zuul team has enabled Zuul CI engine to test Pagure pull requests.

This is about pull requests in the upstream projects, not about dist-git / src.fedoraproject.org pull requests, right?

Both pagure.io and src.fedoraproject.org are supported. We also provide a set of jobs for dist-git. You will find more info in the wiki page the project: https://fedoraproject.org/wiki/Zuul-based-ci Here is a PR on a dist-git project tested by Zuul: https://src.fedoraproject.org/rpms/python3/pull-request/179

Fabien

On Wed, Mar 18, 2020 at 3:54 PM Jan Pazdziora jpazdziora@redhat.com wrote:

On Wed, Mar 18, 2020 at 03:26:14PM +0100, Fabien Boucher wrote:

...

Both pagure.io and src.fedoraproject.org are supported. We also provide a set of jobs for dist-git. You will find more info in the wiki page the project: https://fedoraproject.org/wiki/Zuul-based-ci Here is a PR on a dist-git project tested by Zuul: https://src.fedoraproject.org/rpms/python3/pull-request/179

In dist-git, how does Zuul relate to simple-koji-ci?

The Zuul CI instance is run by the Software Factory project, where we provide CI for RDO and some other projects as you can see here https://softwarefactory-project.io/zuul/tenants. Zuul CI (https://zuul-ci.org/) and simple-koji-ci (https://pagure.io/fedora-ci/simple-koji-ci) are two different projects.

The pull request in question had run a scratch build

    https://koji.fedoraproject.org/koji/taskinfo?taskID=42447845

done in koji via simple-koji-ci, but on the Zuul build page

    https://fedora.softwarefactory-project.io/zuul/buildset/58266604a5ca4f8297de2938c0de9da4

I also see rpm-scratch-build with link pointing to

    https://fedora.softwarefactory-project.io/zuul/build/37899d1668eb4716bce32616861c308a

which has x86_64 files under Logs > buildset ... but thore rpms seem to have different Build Host than those in koji.

rpm-scratch-build builds on koji. The related scratch build is https://koji.fedoraproject.org/koji/taskinfo?taskID=42447907 This link appears in the rpm-scratch-build's console log here https://fedora.softwarefactory-project.io/zuul/build/37899d1668eb4716bce3261... buildvm-26.phx2.fedoraproject.org is a koji builder rpm -qpi python3-3.7.7-1.fc30.x86_64.rpm | grep 'Build Host' Build Host : buildvm-26.phx2.fedoraproject.org

Isn't work being duplicated here?

Yes the scratch build part can be seen as duplication between Zuul and simple-koji-ci for this project's PR. TBH, I don't know how project maintainers can activate/deactivate simple-koji-ci. The Zuul CI runs a scratch build then runs some linter (rpmlint, rpminspect) jobs + the embedded tests.yml if exists against built rpms.

On Mon, 23 Mar 2020 16:27:32 +0100 Jan Pazdziora jpazdziora@redhat.com wrote:

On Wed, Mar 18, 2020 at 02:03:21PM +0100, Jan Pazdziora wrote:

...

On Wed, Mar 11, 2020 at 02:44:14PM +0100, Aleksandra Fedorova wrote:

...

### New test: rpminspect

There is a new rpminspect test running in Fedora Rawhide gating enabled by default for all packages in a non-blocking mode.

More details: https://github.com/rpminspect/rpminspect

Status: Ready to Use Contact: David Cantrell (dcantrell)

Hello,

I've checked random no-noarch rawhide build in Koji

https://koji.fedoraproject.org/koji/buildinfo?buildID=1478656

found its update in Bodhi

https://bodhi.fedoraproject.org/updates/FEDORA-2020-ccc90fa0aa

but I don't see rpminspect among its tests. Is that expected?

Could someone with more insight into the CI setup please check why rpminspect was not run?

Sorry, I managed to miss your previous email. The jenkins instance running rpminspect managed to crap itself last week. I didn't think there were still missing runs but I'll look into it and make sure that the missing ones are re-scheduled.

Thanks for bringing this to my attention.

Tim