I'm trying to wrap up the work to get rpminspect running on all builds coming out of koji and I've run into some questions about the messages I'm emitting and what should be in resultsdb.
Do error messages (koji-build.test.error) belong in resultsdb?
If an error condition is reached, should both a completion (koji-build.test.complete) and an error message be emitted? I'm unclear if exiting with an error condition constitutes completion or not.
I'd appreciate input that folks have on this.
Tim
Hi Tim,
On Wed, Feb 5, 2020 at 5:09 PM Tim Flink tflink@redhat.com wrote:
I'm trying to wrap up the work to get rpminspect running on all builds coming out of koji and I've run into some questions about the messages I'm emitting and what should be in resultsdb.
Do error messages (koji-build.test.error) belong in resultsdb?
I think they do.
If an error condition is reached, should both a completion (koji-build.test.complete) and an error message be emitted? I'm unclear if exiting with an error condition constitutes completion or not.
AFAIK, it's either "error" or "complete". Not both. "error" when CI couldn't finish testing from some reason and "complete" when CI finished testing successfully. Test failures still count as "complete".
Thanks, Michal
I'd appreciate input that folks have on this.
Tim _______________________________________________ CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
On Wed, 5 Feb 2020 17:52:56 +0100 Michal Srb msrb@redhat.com wrote:
Hi Tim,
On Wed, Feb 5, 2020 at 5:09 PM Tim Flink tflink@redhat.com wrote:
I'm trying to wrap up the work to get rpminspect running on all builds coming out of koji and I've run into some questions about the messages I'm emitting and what should be in resultsdb.
Do error messages (koji-build.test.error) belong in resultsdb?
I think they do.
Can you elaborate a bit more on why you think that's the case. I'm more of the mind that they don't belong in resultsdb because "error" as defined in the messaging standard isn't a test result - it's an execution state. I can't see an error state ever being used in a gating decision and thus don't see why errors should be in resultsdb.
That being said, I'm not so strongly attached to not putting them in resultsdb.
FWIW, it appears as if the all-packages pipeline's error messages are not stored in resultsdb:
https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/ci_resu...
Lines 32-36 are for production.
If an error condition is reached, should both a completion (koji-build.test.complete) and an error message be emitted? I'm unclear if exiting with an error condition constitutes completion or not.
AFAIK, it's either "error" or "complete". Not both. "error" when CI couldn't finish testing from some reason and "complete" when CI finished testing successfully. Test failures still count as "complete".
That logic makes sense but there is also some sense in saying that a run is complete if it errors out.
Is that what other systems/pipelines are doing? The all-packages pipeline seems to send out both when an error condition is reached. I'm mostly not sure how other systems are reacting to the messages sent out by CI systems.
Tim
Thanks, Michal
I'd appreciate input that folks have on this.
Tim _______________________________________________ CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
On Wed, 2020-02-05 at 11:19 -0700, Tim Flink wrote:
Can you elaborate a bit more on why you think that's the case. I'm more of the mind that they don't belong in resultsdb because "error" as defined in the messaging standard isn't a test result - it's an execution state. I can't see an error state ever being used in a gating decision and thus don't see why errors should be in resultsdb.
I agree with you. However, I do think execution states would be good to store somewhere, so that UIs can show users the state of the tests as they are running (queued, running, error). It makes sense to me not to store them in the same place as results.
On Wed, 2020-02-05 at 14:01 -0500, Randy Barlow wrote:
On Wed, 2020-02-05 at 11:19 -0700, Tim Flink wrote:
Can you elaborate a bit more on why you think that's the case. I'm more of the mind that they don't belong in resultsdb because "error" as defined in the messaging standard isn't a test result - it's an execution state. I can't see an error state ever being used in a gating decision and thus don't see why errors should be in resultsdb.
I agree with you. However, I do think execution states would be good to store somewhere, so that UIs can show users the state of the tests as they are running (queued, running, error). It makes sense to me not to store them in the same place as results.
We've kicked around the idea of an "execdb" before, I think...
On Wed, 2020-02-05 at 11:47 -0800, Adam Williamson wrote:
We've kicked around the idea of an "execdb" before, I think...
Yeah, or it could just be a new table/API on resultsdb, to avoid complexities of starting and deploying a whole new project.
The data would have a different schema than test results, which is why I advocate for at least giving it its own API and database table.
On Wed, 05 Feb 2020 15:03:19 -0500 Randy Barlow bowlofeggs@fedoraproject.org wrote:
On Wed, 2020-02-05 at 11:47 -0800, Adam Williamson wrote:
We've kicked around the idea of an "execdb" before, I think...
Yeah, or it could just be a new table/API on resultsdb, to avoid complexities of starting and deploying a whole new project.
I'd push back on this pretty hard. Resultsdb is as useful as it is because it's so simple and "dumb". Adding features on top of what's already there starts to defeat that utility.
I'm personally of the mind that two small, simple services are easier to manage than one larger, more complex service.
Taskotron's execdb is a good example in my mind. Most of the dev/maintenance time that we ever put into that service was the frontend and how the data was presented to human users. That work would have to be done no matter where the backend functionality landed and keeping that functionality separate made it easier to maintain both codebases.
That being said, I have no real dog in this fight as someone else would end up doing the implementation.
The data would have a different schema than test results, which is why I advocate for at least giving it its own API and database table.
It sounds like we're of the same mind on this. I'm an advocate of keeping results and execution status separate because the primary audience/consumer for them are not looking for the same thing.
Anyhow, there may well be plans for something like this already. Hopefully someone will chime in if that's the case.
Tim
On Wed, 05 Feb 2020 14:01:04 -0500 Randy Barlow bowlofeggs@fedoraproject.org wrote:
On Wed, 2020-02-05 at 11:19 -0700, Tim Flink wrote:
Can you elaborate a bit more on why you think that's the case. I'm more of the mind that they don't belong in resultsdb because "error" as defined in the messaging standard isn't a test result - it's an execution state. I can't see an error state ever being used in a gating decision and thus don't see why errors should be in resultsdb.
I agree with you. However, I do think execution states would be good to store somewhere, so that UIs can show users the state of the tests as they are running (queued, running, error). It makes sense to me not to store them in the same place as results.
Back in the Taskotron days, we had a dashboard to show the execution status of jobs separate from test results called execdb.
I'm under the impression that there's a dashboard project in RH that accomplishes a similar purpose. It will be open sourced if it hasn't been already but I don't know what the plans are around using it.
Tim
On Wed, Feb 5, 2020 at 7:20 PM Tim Flink tflink@redhat.com wrote:
On Wed, 5 Feb 2020 17:52:56 +0100 Michal Srb msrb@redhat.com wrote:
Hi Tim,
On Wed, Feb 5, 2020 at 5:09 PM Tim Flink tflink@redhat.com wrote:
I'm trying to wrap up the work to get rpminspect running on all builds coming out of koji and I've run into some questions about the messages I'm emitting and what should be in resultsdb.
Do error messages (koji-build.test.error) belong in resultsdb?
I think they do.
Can you elaborate a bit more on why you think that's the case. I'm more of the mind that they don't belong in resultsdb because "error" as defined in the messaging standard isn't a test result - it's an execution state. I can't see an error state ever being used in a gating decision and thus don't see why errors should be in resultsdb.
That being said, I'm not so strongly attached to not putting them in resultsdb.
FWIW, it appears as if the all-packages pipeline's error messages are not stored in resultsdb:
https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/ci_resu...
Lines 32-36 are for production.
Michal answer is correct for the current situation, the error topic is stored in resultsdb and it is also used by Greenwave. the all-packages pipeline is also sends messages using the new format (lines 37-40), but still sends old format as well as there are components that are gating on old testcase name.
Here is an example of error message sent by the pipeline https://apps.fedoraproject.org/datagrepper/id?id=2020-5f7780c8-fbd8-46c8-9a2...
And you can see it shows in bodhi under "Automated Tests" with testcase name "fedora-ci.koji-build.tier0.functional" https://bodhi.fedoraproject.org/updates/FEDORA-2020-803c9175b9
If an error condition is reached, should both a completion (koji-build.test.complete) and an error message be emitted? I'm unclear if exiting with an error condition constitutes completion or not.
AFAIK, it's either "error" or "complete". Not both. "error" when CI couldn't finish testing from some reason and "complete" when CI finished testing successfully. Test failures still count as "complete".
That logic makes sense but there is also some sense in saying that a run is complete if it errors out.
Is that what other systems/pipelines are doing? The all-packages pipeline seems to send out both when an error condition is reached. I'm mostly not sure how other systems are reacting to the messages sent out by CI systems.
Tim
Thanks, Michal
I'd appreciate input that folks have on this.
Tim _______________________________________________ CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
-
Adam Williamson -
Bruno Goncalves -
Michal Srb -
Randy Barlow -
Tim Flink