CI March 2020

ci@lists.fedoraproject.org

12 participants
7 discussions

rpminspect Jenkins is effectively down
by Tim Flink 02 Apr '20

02 Apr '20

I figured I would start a new thread instead of continuing on with the old one. The Jenkins instance running rpminspect has crapped itself for the second time in less than a week and it's currently not processing the rpminspect runs it should be. There are also reports of missing rpminspect runs which likely stem from the Jenkins issues. I'm working on it and will send updates as things are fixed and skipped runs are re-scheduled. Tim

1 1

sidetags in Fedora (for CI purposes)
by Michal Srb 27 Mar '20

27 Mar '20

Hello, I am trying to understand sidetags in Fedora. $ koji list-tags | grep side ... f33-build-side-20525 ... The command above lists tags that look like could be sidetags, so I guess they are already in use. I am trying to understand who creates and when? Are they created automatically for example for each bodhi update, or how does it work? Thanks! Michal

2 2

Planned outage of fedora.softwarefactory-project.io: 2020-03-24 13:00 UTC
by Tristan Cacqueray 24 Mar '20

24 Mar '20

Hello folks, We plan to move all the control plane instances to a new cloud provided by Vexxhost on 2020-03-24 13:00 UTC. Major services interruption is expected during that day, including: * Zuul CI not running jobs for pagure.io or src.fedoraproject.org Let us know if you prefer a different time. Regards, -Tristan, on behalf of the Software Factory Operation Team

1 1

Jenkins unable to read config.xml
by Tim Flink 23 Mar '20

23 Mar '20

I'm having some fun with the Jenkins instance that's supposed to be running rpminspect and hope that someone here might have some suggestions. The instance is hosted in the CentOS CI OpenShift cluster and the configuration is here: https://github.com/tflink/fedora-ci-generic-checks/ At this point, every Jenkins master pod that attempts to start up gets an error like the following: 2020-03-23 21:21:31.509+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions 2020-03-23 21:21:31.535+0000 [id=28] SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading global config java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 16 at com.thoughtworks.xstream.core.util.FastStack.pop(FastStack.java:42) at com.thoughtworks.xstream.io.xml.AbstractPullReader.move(AbstractPullReader.java:125) at com.thoughtworks.xstream.io.xml.AbstractPullReader.moveDown(AbstractPullReader.java:103) at com.thoughtworks.xstream.io.xml.XppReader.<init>(XppReader.java:63) at com.thoughtworks.xstream.io.xml.AbstractXppDriver.createReader(AbstractXppDriver.java:54) at com.thoughtworks.xstream.io.xml.AbstractXppDriver.createReader(AbstractXppDriver.java:65) at hudson.XmlFile.unmarshal(XmlFile.java:180) Caused: java.io.IOException: Unable to read /var/lib/jenkins/config.xml at hudson.XmlFile.unmarshal(XmlFile.java:183) at hudson.XmlFile.unmarshal(XmlFile.java:163) at jenkins.model.Jenkins.loadConfig(Jenkins.java:3081) at jenkins.model.Jenkins.access$1200(Jenkins.java:315) at jenkins.model.Jenkins$13.run(Jenkins.java:3182) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1121) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 2020-03-23 21:21:31.536+0000 [id=21] SEVERE hudson.util.BootFailure#publish: Failed to initialize Jenkins java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 16 at com.thoughtworks.xstream.core.util.FastStack.pop(FastStack.java:42) at com.thoughtworks.xstream.io.xml.AbstractPullReader.move(AbstractPullReader.java:125) at com.thoughtworks.xstream.io.xml.AbstractPullReader.moveDown(AbstractPullReader.java:103) at com.thoughtworks.xstream.io.xml.XppReader.<init>(XppReader.java:63) at com.thoughtworks.xstream.io.xml.AbstractXppDriver.createReader(AbstractXppDriver.java:54) at com.thoughtworks.xstream.io.xml.AbstractXppDriver.createReader(AbstractXppDriver.java:65) at hudson.XmlFile.unmarshal(XmlFile.java:180) Caused: java.io.IOException: Unable to read /var/lib/jenkins/config.xml at hudson.XmlFile.unmarshal(XmlFile.java:183) at hudson.XmlFile.unmarshal(XmlFile.java:163) at jenkins.model.Jenkins.loadConfig(Jenkins.java:3081) at jenkins.model.Jenkins.access$1200(Jenkins.java:315) at jenkins.model.Jenkins$13.run(Jenkins.java:3182) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1121) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused: org.jvnet.hudson.reactor.ReactorException at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282) at jenkins.InitReactorRunner.run(InitReactorRunner.java:48) at jenkins.model.Jenkins.executeReactor(Jenkins.java:1154) at jenkins.model.Jenkins.<init>(Jenkins.java:965) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:233) Caused: hudson.util.HudsonFailedToLoad at hudson.WebAppMain$3.run(WebAppMain.java:250) 2020-03-23 21:21:31.550+0000 [id=21] INFO jenkins.model.Jenkins#cleanUp: Stopping Jenkins 2020-03-23 21:21:31.564+0000 [id=27] WARNING hudson.model.UpdateCenter#save: Failed to save /var/lib/jenkins/hudson.model.UpdateCenter.xml java.nio.channels.ClosedByInterruptException at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:199) at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:162) at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:285) at hudson.util.FileChannelWriter.write(FileChannelWriter.java:73) at java.base/java.io.Writer.write(Writer.java:290) at hudson.util.AtomicFileWriter.write(AtomicFileWriter.java:162) at java.base/java.io.Writer.write(Writer.java:249) at hudson.XmlFile.write(XmlFile.java:191) at hudson.model.UpdateCenter.save(UpdateCenter.java:893) at hudson.util.PersistedList.onModified(PersistedList.java:173) at hudson.util.PersistedList.replaceBy(PersistedList.java:85) at hudson.model.UpdateCenter.load(UpdateCenter.java:907) at hudson.model.UpdateCenter.init(UpdateCenter.java:2465) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104) at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1121) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 2020-03-23 21:21:31.714+0000 [id=21] INFO jenkins.model.Jenkins$18#onAttained: Started termination Has anyone seen this before? Any suggestions on things I could try? At this point, the only idea I have to fix the thing is to delete the persistent storage and rebuild the thing. If I do that, all the result links in rpminspect results would be broken and the logs from those runs lost. Thanks, Tim

1 0

Fedora CI update 2020-03-11
by Aleksandra Fedorova 23 Mar '20

23 Mar '20

Hi, all. Here is the summary of CI-related work happening in Fedora. If you have questions or topics to discuss you can also join Fedora CI SIG bi-weekly meeting. Next session is today in #fedora-ci IRC channel at 15:30 UTC https://apps.fedoraproject.org/calendar/SIGs/#m9618 ======================================================== ### Dist-git tests support multipackage updates You can define package tests in dist-git via STR format https://docs.fedoraproject.org/en-US/ci/standard-test-roles/ Note that dist-git/STR tests are different from running %check tests in the rpm building phase. STR tests are executed in a clean virtual machine with proper setup of repositories for the latest Fedora Rawhide packages. This environment is better suited for integration tests, which need to test the installed package, not the sources of it. Dist-git tests are fully compatible with the dynamic sidetag approach: if you create a dynamic sidetag for the multi-package update, test environment will be created with the content of the entire sidetag, not an individual package. Status: Ready to Use Contact: Bruno Goncalves (bgoncalv) and #fedora-ci IRC channel. ### New test: rpminspect There is a new rpminspect test running in Fedora Rawhide gating enabled by default for all packages in a non-blocking mode. More details: https://github.com/rpminspect/rpminspect Status: Ready to Use Contact: David Cantrell (dcantrell) ### Tests for pull-requests via Zuul Zuul team has enabled Zuul CI engine to test Pagure pull requests. You can opt-in to Zuul CI per package. On every pull-request Zuul will * run a scratch build * run rpminspect on that build * run dist-git test defined in STR format(if available) * provide comment in the pull-request * wait for you to put an manual approval on the PR * merge the PR * you can also get Zuul to handle merge events, so that it will automatically build the regular koji build, after the merge. Zuul now has support for EPEL8 branches. More details: https://fedoraproject.org/wiki/Zuul-based-ci Status: Ready to use Contact: Fabien Boucher (fbo) ### Koschei as a gating test With sidetag gating feature enabled it is now possible to run Koschei for each dynamic sidetag and make it a part of the gating process. We do have Koschei deployed on Fedora infrastructure. There is on-going work by Mikolaj Izdebski to get it integrated with the Fedora Messaging, so that sidetags are submitted in Koschei when created. Status: research and prototyping Contact: Serhii Turivnyi (sturivny) ### Infra change: new Jenkins master New Jenkins master to run generic tests and inherit Taskotron pipelines. Our current Jenkins master, which is used for dist-git tests, was not updated for some time and it is by design bundled to the pipeline it runs. So adding new pipelines and separating pipelines from the Jenkins master configuration is problematic. The goal here is to have a Jenkins master setup which is easy to update. It will have a set of plugins pre-installed and configured for Fedora infrastructure endpoints, but jobs configuration will be applied to it independently. More details: Current work is done on a Communishift project. Code will be available soon at https://github.com/fedora-ci Status: WIP Contact: Jim Bair (jbair) ### Infra change: common repository and common format for generic tests We are refactoring the Groovy pipeline library so it is better suited to run generic tests. One of the goals is that generic tests are all run in the same way, and you don’t need to add a lot of new Groovy code to add a certain bash script as a generic test. We’d like people to be able to contribute new generic tests without prior knowledge of the Jenkins internal setup. Current focus is rpmdeplint and rpminspect pipelines. More details: https://github.com/fedora-ci Status: WIP Contact: Michal Srb (msrb) ### Infra change: ODCS composes We are updating ODCS staging infrastructure to the latest ODCS release. Once the Fedora instractructure freeze is over, we will also update the ODCS production instance. This work is preparation for possible further use of ODCS to generate composes used by Fedora CI as well as main Fedora composes. Status: WIP Contact: Jan Kaluza (jkaluza) ### Infra change: Testing Farm Service Testing Farm Team is working on open-sourcing parts of the RH internal CI infrastructure as a service, which will be used by Fedora CI's general tests and functional tests pipeline. The main input of the service will be test definitions in the TMT/FMF format. TMT documentation: https://tmt.readthedocs.io/en/latest/ (recently added testcloud + podman provisioner) Code is hosted at GitLab: https://gitlab.com/testing-farm/ Status: WIP (preview May 2020, GA August 2020) Contact: Miroslav Vadkerti (mvadkert) ### CI and Gating documentation There is a repository of CI documentation https://pagure.io/fedora-ci/docs/ The docs are published at https://docs.fedoraproject.org/en-US/ci/ Then there is another repository with docs on Rawhide Gating: https://pagure.io/cpe/rawhide-gating-docs/ And result is available at: https://docs.fedoraproject.org/en-US/rawhide-gating/ There are some rather deep or generic items there, which are not always suitable for newcomers and are not easy to consume. What I think we need is a smaller scale how-to’s answering specific questions and implementing specific use-cases, which hook the CI and gating into the packager workflow. If you have experience with sidetag gating or CI in Fedora and figured out the way how _you_ work with it, please share. You can drop me a mail or write a draft page and send a pull request to one of the repositories. We will figure out later in which section to land it. Status: Needs help Contact: Aleksandra Fedorova (bookwar) ### Testing of GitHub PRs via Packit / Testing Farm on Fedora/CentOS Packit service makes it possible to test copr builds built from GitHub PRs on all Fedora released (including Rawhide), CentOS 6/7 and CentOS stream via Testing Farm. Note that the Testing Farm deployed for Packit is different from the one we are open sourcing, and once that is ready i will replace this one. Documentation: https://packit.dev/testing-farm/ Status: In production (since August 2019) Contact: Miroslav Vadkerti (mvadkert) ======================================================== For any of those topics you can contact Fedora CI SIG at #fedora-ci IRC channel. -- Aleksandra Fedorova bookwar

7 12

Syncing Up on Generic Checks
by Tim Flink 18 Mar '20

18 Mar '20

I brought this up in the last ci meeting but as I'm learning more, I think this topic could use a broader discussion to be more sure that I and maybe others aren't misunderstanding what the broader plan is. Below, I've written up a summary of my understanding of what's going on around generic checks. I figure that's the fastest way to see what I'm missing and doubles as a decent summary of current work beyond what bookwar sent out earlier this week. I should have been more clear on what I was planning to do so that there aren't overlaps and I'm looking to start correcting that now. Please let me know if I've misunderstood any of the bits below. Thanks, Tim ---------- rpmdeplint ---------- I know that msrb is working on a pipeline to get rpmdeplint running for Fedora builds. That work is being tracked in taiga [1] and for the moment, lives in github [2]. Once that prototype is complete, there will be more discussion on where rpmdeplint will be run going forwards. I don't know if the prototype is deployed anywhere at the moment. [1] https://teams.fedoraproject.org/project/ci/us/71 [2] https://github.com/fedora-ci/rawhide-rpmdeplint-pipeline ------------------ pipeline libraries ------------------ msrb is also working on a new (?) pipeline library for the use of more generic, lint-ish checks going forward [3]. I am unclear on how or if there will be any relationship with the existing pipeline libraries or what pipelines/checks will be using the new library going forward. [3] https://github.com/fedora-ci/jenkins-pipeline-library ------- Jenkins ------- Outside the pipelines which are used for running package-specific CI tests, I am aware of at least two Jenkins masters which are being used for something related to CI. rpminspect production running in the CentOS CI cluster [5] the thing jbair is working on in Fedora communishift [no link] msrb may have an instance for rpmdeplint I also have a second project in the CentOS CI openshift cluster which I was planning to use as a staging instance for the rpminspect pipeline and the other generic checks that will go away when Taskotron goes EOL at the end of April. I'm unclear on if there are plans around combining all these Jenkins masters or what those plans might be. I'm hesitant to continue work on a staging instance if it's not going to live very long. As far as I know, the docker images used by these various efforts are all different to varying degrees. It sounds like there could be plans to do some unification based on a ticket in taiga [4] but I'm not clear on those plans [4] https://teams.fedoraproject.org/project/ci/us/70 ---------------------------------- My Previous Plans for Jenkins Work ---------------------------------- I had been planning the following work on the rpminspect setup. It turns out that some of the work is already being done by others and I'm unclear on the rest. - Move rpmdeplint from Taskotron to Jenkins until testing farm is running it so that we don't have gaps in coverage after Taskotron goes EOL. * AFAIK, msrb is working on this and I will not continue the work I had been doing - Complete the transition of the generic task setup to JCasC * jbair is working on this for a different project now? I'm unclear on what's going on here - Set up a custom tag for the Jenkins master so that we have more control over when upgrades happen * also improve the image tagging that is currently used so that stuff can be tested before it shows up in production - Set up a staging instance of the generic master which is currently running rpminspect. This would enable better testing of: * changes to how rpminspect and other generic checks are run without significant risk to production results * new versions of the actual tools like rpminspect or rpmdeplint * This would also enable better plugin upgrade testing and new Jenkins master version testing I say previous plans because I'm not very clear on what the larger plans are going forward. As far as I know, all generic checks will be handled by Testing Farm starting later this summer. To me, this means that most of what I've outlined above is not worth working on and going beyond "good enough for now" is not going to be worth the effort for things which are not directly contributing to Testing Farm's efforts.

4 4

Can some one help me to understand why Fedora CI is not working currently?
by Parag Nemade 13 Mar '20

13 Mar '20

Hi, I see Fedora CI is not running on the PR I am submitting for packages. Can someone help me for the issues I am seeing? I have reported my issue https://pagure.io/fedora-ci/general/issue/99 Regards, Parag

2 2

2024

2023

2022

2021

2020

2019

2018

2017

CI March 2020