Hello,
I've built a package today for rawhide, it got errata
https://bodhi.fedoraproject.org/updates/FEDORA-2020-a9b4fc5c9d
created automatically and
org.centos.prod.ci.pipeline.allpackages-build.package.test.functional.complete
finished for it in half an hour or so.
The I've done the same build for Fedora 32, I've filed
https://bodhi.fedoraproject.org/updates/FEDORA-2020-8f05f05489
for it but since then
org.centos.prod.ci.pipeline.allpackages-build.package.test.functional.complete
is stuck as missing. The red rectangle is not clickable, so it's hard to see if the test was actually scheduled, if CI knows about the new build, what is going on.
Could anyone with insight into the Fedora CI process take a look to see where things got stuck, and also recommend how to do the check in the future myself?
Thank you,
On Thu, Feb 27, 2020 at 4:19 PM Jan Pazdziora jpazdziora@redhat.com wrote:
Hello,
I've built a package today for rawhide, it got errata
https://bodhi.fedoraproject.org/updates/FEDORA-2020-a9b4fc5c9dcreated automatically and
org.centos.prod.ci.pipeline.allpackages-build.package.test.functional.completefinished for it in half an hour or so.
The I've done the same build for Fedora 32, I've filed
https://bodhi.fedoraproject.org/updates/FEDORA-2020-8f05f05489for it but since then
org.centos.prod.ci.pipeline.allpackages-build.package.test.functional.completeis stuck as missing. The red rectangle is not clickable, so it's hard to see if the test was actually scheduled, if CI knows about the new build, what is going on.
That is interesting, it looks like the pipeline didn't trigger because bodhi didn't send a message for it.
At least the only recent message that it sent about this package recently was for rawhide.... https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod....
Bruno
Could anyone with insight into the Fedora CI process take a look to see where things got stuck, and also recommend how to do the check in the future myself?
Thank you,
-- Jan Pazdziora Product Owner, Platform Security Readiness, Red Hat _______________________________________________ CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
On Thu, Feb 27, 2020 at 04:56:28PM +0100, Bruno Goncalves wrote:
That is interesting, it looks like the pipeline didn't trigger because bodhi didn't send a message for it.
At least the only recent message that it sent about this package recently was for rawhide.... https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod....
Thank you for looking into it.
Does that sound like a bug in bodhi, so something to file an issue at
https://github.com/fedora-infra/bodhi/issues
for?
Is the only way to force that CI job to run to make bodhi send the message?
On Thu, Feb 27, 2020 at 5:04 PM Jan Pazdziora jpazdziora@redhat.com wrote:
On Thu, Feb 27, 2020 at 04:56:28PM +0100, Bruno Goncalves wrote:
That is interesting, it looks like the pipeline didn't trigger because bodhi didn't send a message for it.
At least the only recent message that it sent about this package recently was for rawhide.... https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod....
Thank you for looking into it.
Does that sound like a bug in bodhi, so something to file an issue at
https://github.com/fedora-infra/bodhi/issuesfor?
I'm not sure either, I'd rather if someone with more experience with bodhi could say when "org.fedoraproject.prod.bodhi.update.status.testing.koji-build-group.build.complete" is sent. Or if we should trigger with some other topic...
I've been triggering on it since https://pagure.io/fedora-ci/general/issue/70
Is the only way to force that CI job to run to make bodhi send the message?
I've rescheduled it manually, let's see how it goes... https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-build...
-- Jan Pazdziora Product Owner, Platform Security Readiness, Red Hat _______________________________________________ CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
On Thu, Feb 27, 2020 at 05:49:07PM +0100, Bruno Goncalves wrote:
I've rescheduled it manually, let's see how it goes... https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-build...
It passed and
https://bodhi.fedoraproject.org/updates/FEDORA-2020-8f05f05489
is now happy with six passing tests.
Thank you!
On Thu, 2020-02-27 at 17:49 +0100, Bruno Goncalves wrote:
On Thu, Feb 27, 2020 at 5:04 PM Jan Pazdziora jpazdziora@redhat.com wrote:
On Thu, Feb 27, 2020 at 04:56:28PM +0100, Bruno Goncalves wrote:
That is interesting, it looks like the pipeline didn't trigger because bodhi didn't send a message for it.
At least the only recent message that it sent about this package recently was for rawhide.... https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod....
Thank you for looking into it.
Does that sound like a bug in bodhi, so something to file an issue at
https://github.com/fedora-infra/bodhi/issuesfor?
I'm not sure either, I'd rather if someone with more experience with bodhi could say when "org.fedoraproject.prod.bodhi.update.status.testing.koji-build-group.build.complete" is sent.
Per the relevant doc string:
" Signal that the update has been moved to testing.
This happens in the following cases: - for stable releases: the update lands in the testing repository - for rawhide: all packages in an update have been built by koji "
https://github.com/fedora-infra/bodhi/blob/develop/bodhi/server/models.py#L3... (that function does nothing but publish the message in question, and nothing else publishes it).
Note that "the update lands in the testing repository" definitely is *not* the same point in time as "the update is created". It will usually be at least several minutes later, more usually several hours later (pushes to updates-testing usually happen once a day per release, more can be done manually if requested for some reason).
This was worse for F32 up until now - there haven't been *any* updates- testing pushes for it yet. I just asked releng, and Mohan is going to do one now. From then on there should be one a day.
So the 'typical' experience, as things stand, will (I expect) be that updates come up as failing gating shortly after they're created; then a few hours later, an updates-testing push should happen, the tests should be scheduled, and then the status should change to passing gating (as long as the tests pass).
Depending on how the pipeline actually *does* testing of updates, you *may* be able to schedule on the update being created or edited instead/as well (this is how the openQA scheduler does it; we don't use the koji-build-group.build.complete topic at all, we use bodhi.update.request.testing (published when an update is created) and bodhi.update.edit (published when an update is edited) instead. However, you can only do that if the pipeline does not rely on the packages actually being in the updates-testing repo, but retrieves them itself. At the point those messages are published, the package files cannot be relied upon to be present anywhere outside of Koji.
HTH!
On Thu, Feb 27, 2020 at 01:27:46PM -0800, Adam Williamson wrote:
On Thu, 2020-02-27 at 17:49 +0100, Bruno Goncalves wrote:
On Thu, Feb 27, 2020 at 5:04 PM Jan Pazdziora jpazdziora@redhat.com wrote:
On Thu, Feb 27, 2020 at 04:56:28PM +0100, Bruno Goncalves wrote:
That is interesting, it looks like the pipeline didn't trigger because bodhi didn't send a message for it.
At least the only recent message that it sent about this package recently was for rawhide.... https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod....
Thank you for looking into it.
Does that sound like a bug in bodhi, so something to file an issue at
https://github.com/fedora-infra/bodhi/issuesfor?
I'm not sure either, I'd rather if someone with more experience with bodhi could say when "org.fedoraproject.prod.bodhi.update.status.testing.koji-build-group.build.complete" is sent.
Per the relevant doc string:
" Signal that the update has been moved to testing.
This happens in the following cases: - for stable releases: the update lands in the testing repository - for rawhide: all packages in an update have been built by koji"
https://github.com/fedora-infra/bodhi/blob/develop/bodhi/server/models.py#L3... (that function does nothing but publish the message in question, and nothing else publishes it).
Note that "the update lands in the testing repository" definitely is *not* the same point in time as "the update is created". It will usually be at least several minutes later, more usually several hours later (pushes to updates-testing usually happen once a day per release, more can be done manually if requested for some reason).
This was worse for F32 up until now - there haven't been *any* updates- testing pushes for it yet. I just asked releng, and Mohan is going to do one now. From then on there should be one a day.
This is all correct :)
So the 'typical' experience, as things stand, will (I expect) be that updates come up as failing gating shortly after they're created; then a few hours later, an updates-testing push should happen, the tests should be scheduled, and then the status should change to passing gating (as long as the tests pass).
Correct again, it "fails" gating because the expected tests aren't present yet indeed.
Depending on how the pipeline actually *does* testing of updates, you *may* be able to schedule on the update being created or edited instead/as well (this is how the openQA scheduler does it; we don't use the koji-build-group.build.complete topic at all, we use bodhi.update.request.testing (published when an update is created) and bodhi.update.edit (published when an update is edited) instead. However, you can only do that if the pipeline does not rely on the packages actually being in the updates-testing repo, but retrieves them itself. At the point those messages are published, the package files cannot be relied upon to be present anywhere outside of Koji.
Well, when the update is created the RPMs are signed yet and we want to test the signed RPM to make sure that we test what is being pushed to the users.
From reading this, I think things work as designed, it's just that there were no push to update-testing for F32 yet.
Pierre
On Fri, Feb 28, 2020 at 09:20:16AM +0100, Pierre-Yves Chibon wrote:
Depending on how the pipeline actually *does* testing of updates, you *may* be able to schedule on the update being created or edited instead/as well (this is how the openQA scheduler does it; we don't use the koji-build-group.build.complete topic at all, we use bodhi.update.request.testing (published when an update is created) and bodhi.update.edit (published when an update is edited) instead. However, you can only do that if the pipeline does not rely on the packages actually being in the updates-testing repo, but retrieves them itself. At the point those messages are published, the package files cannot be relied upon to be present anywhere outside of Koji.
Well, when the update is created the RPMs are signed yet and we want to test the signed RPM to make sure that we test what is being pushed to the users.
From reading this, I think things work as designed, it's just that there were no push to update-testing for F32 yet.
The question is, could the CI run based on some other messages, like those that Adam mentioned? Will the test setup be able to pull the bits from koji or does it rely on updates-testing? Bruno was able to schedule the CI job for my Fedora 32 update and it passed so it looks like it was able to get the bits in spite of no update-testing pushes for Fedora 32.
In general it seems like a suboptimal user experience from maintainers' point of view when they create an errata and there is an undefined time for which the errata will show failing tests because the gating tests haven't been even scheduled. Any change we can make to get the erratas all ready (with tests run and passed) will increase the chance that maintainers will be willing to add gating tests to their packages.
On Fri, Feb 28, 2020 at 10:06:18AM +0100, Jan Pazdziora wrote:
On Fri, Feb 28, 2020 at 09:20:16AM +0100, Pierre-Yves Chibon wrote:
Depending on how the pipeline actually *does* testing of updates, you *may* be able to schedule on the update being created or edited instead/as well (this is how the openQA scheduler does it; we don't use the koji-build-group.build.complete topic at all, we use bodhi.update.request.testing (published when an update is created) and bodhi.update.edit (published when an update is edited) instead. However, you can only do that if the pipeline does not rely on the packages actually being in the updates-testing repo, but retrieves them itself. At the point those messages are published, the package files cannot be relied upon to be present anywhere outside of Koji.
Well, when the update is created the RPMs are signed yet and we want to test the signed RPM to make sure that we test what is being pushed to the users.
From reading this, I think things work as designed, it's just that there were no push to update-testing for F32 yet.
The question is, could the CI run based on some other messages, like those that Adam mentioned? Will the test setup be able to pull the bits from koji or does it rely on updates-testing? Bruno was able to schedule the CI job for my Fedora 32 update and it passed so it looks like it was able to get the bits in spite of no update-testing pushes for Fedora 32.
Checking the
https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenki...
the rpms well pulled in from the package-specific repo
swid-tools.noarch 0.8.9-1.fc32 @test-swid-tools
So it seems like the gating tests don't have to wait for the updates-testing push and some other message could be used.
On Fri, Feb 28, 2020 at 4:02 PM Jan Pazdziora jpazdziora@redhat.com wrote:
On Fri, Feb 28, 2020 at 10:06:18AM +0100, Jan Pazdziora wrote:
On Fri, Feb 28, 2020 at 09:20:16AM +0100, Pierre-Yves Chibon wrote:
Depending on how the pipeline actually *does* testing of updates, you *may* be able to schedule on the update being created or edited instead/as well (this is how the openQA scheduler does it; we don't use the koji-build-group.build.complete topic at all, we use bodhi.update.request.testing (published when an update is created) and bodhi.update.edit (published when an update is edited) instead. However, you can only do that if the pipeline does not rely on the packages actually being in the updates-testing repo, but retrieves them itself. At the point those messages are published, the package files cannot be relied upon to be present anywhere outside of Koji.
Well, when the update is created the RPMs are signed yet and we want to test the signed RPM to make sure that we test what is being pushed to the users.
From reading this, I think things work as designed, it's just that there were no push to update-testing for F32 yet.
The question is, could the CI run based on some other messages, like those that Adam mentioned? Will the test setup be able to pull the bits from koji or does it rely on updates-testing? Bruno was able to schedule the CI job for my Fedora 32 update and it passed so it looks like it was able to get the bits in spite of no update-testing pushes for Fedora 32.
Checking the
https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-f32-build-pipeline/detail/fedora-f32-build-pipeline/26/pipeline/the rpms well pulled in from the package-specific repo
swid-tools.noarch 0.8.9-1.fc32 @test-swid-toolsSo it seems like the gating tests don't have to wait for the updates-testing push and some other message could be used.
It might be some other components require some newer version of package as dependency that are not still on stable repositories. We could probably discuss to trigger on different message, but it would be nice to have all gating tests triggering on the same topic.
Jan, in case you want suggest to trigger on a different message, could you request it as an issue on https://pagure.io/fedora-ci/general/issues ?
Bruno
-- Jan Pazdziora Product Owner, Platform Security Readiness, Red Hat _______________________________________________ CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
On Fri, 2020-02-28 at 10:06 +0100, Jan Pazdziora wrote:
On Fri, Feb 28, 2020 at 09:20:16AM +0100, Pierre-Yves Chibon wrote:
Depending on how the pipeline actually *does* testing of updates, you *may* be able to schedule on the update being created or edited instead/as well (this is how the openQA scheduler does it; we don't use the koji-build-group.build.complete topic at all, we use bodhi.update.request.testing (published when an update is created) and bodhi.update.edit (published when an update is edited) instead. However, you can only do that if the pipeline does not rely on the packages actually being in the updates-testing repo, but retrieves them itself. At the point those messages are published, the package files cannot be relied upon to be present anywhere outside of Koji.
Well, when the update is created the RPMs are signed yet and we want to test the signed RPM to make sure that we test what is being pushed to the users.
From reading this, I think things work as designed, it's just that there were no push to update-testing for F32 yet.
The question is, could the CI run based on some other messages, like those that Adam mentioned? Will the test setup be able to pull the bits from koji or does it rely on updates-testing? Bruno was able to schedule the CI job for my Fedora 32 update and it passed so it looks like it was able to get the bits in spite of no update-testing pushes for Fedora 32.
Pierre was answering that question, I think: his answer is that he doesn't want CI to pull direct from Koji because when you do that, you get RPMs which have not been through the signing process, and he thinks it's important to test exactly what actually goes out to users. It's a reasonable point.
In general it seems like a suboptimal user experience from maintainers' point of view when they create an errata and there is an undefined time for which the errata will show failing tests because the gating tests haven't been even scheduled. Any change we can make to get the erratas all ready (with tests run and passed) will increase the chance that maintainers will be willing to add gating tests to their packages.
I mean, it should be possible to fudge this, though it may involve the sort of magic some folks tend to be unhappy with. If we know this is a thing that happens, we can maybe make the UI look less angry and red when the state is 'tests not yet scheduled' as opposed to 'a test actually failed', right? We can even start doing more speculative stuff like having a timer - if all the required tests aren't scheduled *24 hours after the update is submitted*, that's more worrying that if they aren't scheduled 5 minutes after it's submitted. Or we could teach Greenwave or Bodhi or something about updates-testing pushes in some way...
-
Adam Williamson -
Bruno Goncalves -
Jan Pazdziora -
Pierre-Yves Chibon