I am trying to get a wrapped test script to work for squashfs-tools and the artifacts don't appear to be properly saved. I think I might be misunderstanding the wrapper example. Is the idea really that the test should run ansible with the basic role referring to the wrapper? I have been trying to run the wrapper as the test and it runs the shell script, but the test results don't end up where the CI can find them and the test fails. I also don't see a way for the wrapper to ask for extra packages other than to run dnf in the script.
There is stuff to look at in the pipeline pages if you want to see what I tried to do. _______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org
On Tue, May 21, 2019 at 15:57:21 -0500, Bruno Wolff III bruno@wolff.to wrote:
I am trying to get a wrapped test script to work for squashfs-tools and the artifacts don't appear to be properly saved. I think I might be misunderstanding the wrapper example. Is the idea really that the test should run ansible with the basic role referring to the wrapper? I have been trying to run the wrapper as the test and it runs the shell script, but the test results don't end up where the CI can find them and the test fails.
It looks like the wrapper documentation is wrong about something. Artifacts don't seem to get copied back to where they are expected to be. I'm going to test if remote_artifacts is defined for the wrapper playbook. I'm guessing that only works if you explicitly use the base role. Then I'm going to try .., but I'm expecting that to be blocked by security checks.
On Wed, May 22, 2019 at 03:25:07 -0500, Bruno Wolff III bruno@wolff.to wrote:
On Tue, May 21, 2019 at 15:57:21 -0500, Bruno Wolff III bruno@wolff.to wrote:
I am trying to get a wrapped test script to work for squashfs-tools and the artifacts don't appear to be properly saved. I think I might be misunderstanding the wrapper example. Is the idea really that the test should run ansible with the basic role referring to the wrapper? I have been trying to run the wrapper as the test and it runs the shell script, but the test results don't end up where the CI can find them and the test fails.
It looks like the wrapper documentation is wrong about something. Artifacts don't seem to get copied back to where they are expected to be. I'm going to test if remote_artifacts is defined for the wrapper playbook. I'm guessing that only works if you explicitly use the base role. Then I'm going to try .., but I'm expecting that to be blocked by security checks.
I'm getting closer, but am getting close to done for the night.
It looks like test.log and results.yml need to get copied to ../../logs/ instead of {{ artifacts }}. It would be nice if a good way to add extra packages to the wrapper config existed. I'm using dnf -y --nogpgcheck in the command instead. I was a bit surprised the redirected outout from the command included a lot of set up and not just the output from my test script. I'm testing to see if parens can help that. I think I can make the test for squashfs-tools work. The last part is probably going to be figuring out why some mounts don't work. I may need kernel-modules-extra and in the process of testing that. But I'm pretty sure that's just a matter of figuring out what the missing package is.
Hi Bruno,
for your use case (I see you have a single shell test script in your pull request [1]) I would recommend to use the basic role:
https://docs.fedoraproject.org/en-US/ci/standard-test-roles/#_basic
It would take care of correctly copying artifacts and also handle test script return code to report appropriate result. Also adding a list of required packages is easy there. You can also have a look at a few more examples here:
https://docs.fedoraproject.org/en-US/ci/examples/
Hope this helps.
psss...
[1] https://src.fedoraproject.org/rpms/squashfs-tools/pull-request/5
On Wed, 22 May 2019 at 12:05, Bruno Wolff III bruno@wolff.to wrote:
On Wed, May 22, 2019 at 03:25:07 -0500, Bruno Wolff III bruno@wolff.to wrote:
On Tue, May 21, 2019 at 15:57:21 -0500, Bruno Wolff III bruno@wolff.to wrote:
I am trying to get a wrapped test script to work for squashfs-tools and the artifacts don't appear to be properly saved. I think I might be misunderstanding the wrapper example. Is the idea really that the test should run ansible with the basic role referring to the wrapper? I have been trying to run the wrapper as the test and it runs the shell script, but the test results don't end up where the CI can find them and the test fails.
It looks like the wrapper documentation is wrong about something. Artifacts don't seem to get copied back to where they are expected to be. I'm going to test if remote_artifacts is defined for the wrapper playbook. I'm guessing that only works if you explicitly use the base role. Then I'm going to try .., but I'm expecting that to be blocked by security checks.
I'm getting closer, but am getting close to done for the night.
It looks like test.log and results.yml need to get copied to ../../logs/ instead of {{ artifacts }}. It would be nice if a good way to add extra packages to the wrapper config existed. I'm using dnf -y --nogpgcheck in the command instead. I was a bit surprised the redirected outout from the command included a lot of set up and not just the output from my test script. I'm testing to see if parens can help that. I think I can make the test for squashfs-tools work. The last part is probably going to be figuring out why some mounts don't work. I may need kernel-modules-extra and in the process of testing that. But I'm pretty sure that's just a matter of figuring out what the missing package is. _______________________________________________ CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
On Wed, May 22, 2019 at 13:25:17 +0200, Petr Šplíchal psplicha@redhat.com wrote:
Hi Bruno,
for your use case (I see you have a single shell test script in your pull request [1]) I would recommend to use the basic role:
https://docs.fedoraproject.org/en-US/ci/standard-test-roles/#_basic
It would take care of correctly copying artifacts and also handle test script return code to report appropriate result. Also adding a list of required packages is easy there. You can also have a look at a few more examples here:
The script doesn't return a failure unless it has a problem with something outside what is being tested. You need to run grep to see if any of the tests, run in loops, fails. That messes with the output. I could probably string something together not much different than I did for the wrapper case though.
I'm close enough with the wrapper set up now, that I'll probably see if I can finish with it. However, the documentation being off could cause a lot of confusion for others trying to create tests for their packages and something should be done to fix it or warn people off.
I've also been seeing random failures with set up of the environment. Not always the same thing. I'm worried there might be issues with gating because of this. I'm not sure how to rerun a test that has been used for gating.
On Wed, 22 May 2019 at 14:55, Bruno Wolff III bruno@wolff.to wrote:
On Wed, May 22, 2019 at 13:25:17 +0200, Petr Šplíchal psplicha@redhat.com wrote:
Hi Bruno,
for your use case (I see you have a single shell test script in your pull request [1]) I would recommend to use the basic role:
https://docs.fedoraproject.org/en-US/ci/standard-test-roles/#_basic
It would take care of correctly copying artifacts and also handle test script return code to report appropriate result. Also adding a list of required packages is easy there. You can also have a look at a few more examples here:
The script doesn't return a failure unless it has a problem with something outside what is being tested. You need to run grep to see if any of the tests, run in loops, fails. That messes with the output. I could probably string something together not much different than I did for the wrapper case though.
You could do something like this:
- hosts: localhost roles: - role: standard-test-basic tags: - classic tests: - compression: dir: . run: "./Squashfs-compression-test.sh 2>&1 | tee /tmp/test.log && ! grep -iq failed /tmp/test.log" required_packages: - kernel-modules-extra - rpmdevtools
Which looks much more concise.
I'm close enough with the wrapper set up now, that I'll probably see if I can finish with it. However, the documentation being off could cause a lot of confusion for others trying to create tests for their packages and something should be done to fix it or warn people off.
What exactly in the docs should cause the confusion? Should any of these pages be updated?
https://docs.fedoraproject.org/en-US/ci/quick-start-guide/ https://docs.fedoraproject.org/en-US/ci/standard-test-roles/ https://docs.fedoraproject.org/en-US/ci/examples/
Or are you missing some essential content there?
I've also been seeing random failures with set up of the environment. Not always the same thing. I'm worried there might be issues with gating because of this. I'm not sure how to rerun a test that has been used for gating.
Including the "[citest]" string in a pull request comment does rerun all enabled tests.
psss...
On Wed, May 22, 2019 at 16:25:16 +0200, Petr Šplíchal psplicha@redhat.com wrote:
On Wed, 22 May 2019 at 14:55, Bruno Wolff III bruno@wolff.to wrote:
You could do something like this:
- hosts: localhost
roles:
- role: standard-test-basic tags:
tests:
- classic
- compression: dir: . run: "./Squashfs-compression-test.sh 2>&1 | tee /tmp/test.log && !
grep -iq failed /tmp/test.log" required_packages:
- kernel-modules-extra
- rpmdevtools
That's pretty close to what I was thinking and I may switch to it after I figure out what appears to be a missing required package. I also think I'll need to negate the gtrep test, but there is a way to do that that I've used in the past, but don't remember off hand.
Which looks much more concise.
I agree that it will be easier to understand. So I'll probably switch.
What exactly in the docs should cause the confusion? Should any of these pages be updated?
The wrapper stuff has you copying the log and results to {{ artifacts }}, which isn't correct. The correct location is ../../logs/ . I don't know if that is in a variable someplace. I figured it out by looking at the pipeline logs. It was very confusing trying to figure out what was going on and I wasted a fair amount of time before deciding the documentation must be wrong.
Including the "[citest]" string in a pull request comment does rerun all enabled tests.
I found that and it is useful, but not all builds come through pull requests. I'm not sure how the gate is connecting builds and tests. If it is by latest build of a commit, then redoing a pull test will work. But I forgot there is also a waive process, so worst case I can waive a test if need be.
On Wed, May 22, 2019 at 13:25:17 +0200, Petr Šplíchal psplicha@redhat.com wrote:
It would take care of correctly copying artifacts and also handle test script return code to report appropriate result. Also adding a list of required packages is easy there. You can also have a look at a few more examples here:
It turns out there is a dnf module for ansible that can install packages in the wrapper.
So for now I'm going to use a wrapper style, just so there will be a good example of one.
I have one last thing I should need to figure out before the test is done. It looks like squashfs file systems won't auto detect properly. I'm hoping if I force the type they will mount. If not, I'll still have a bit of work figuring this puzzle out.
On Wed, May 22, 2019 at 13:46:20 -0500, Bruno Wolff III bruno@wolff.to wrote:
I have one last thing I should need to figure out before the test is done. It looks like squashfs file systems won't auto detect properly. I'm hoping if I force the type they will mount. If not, I'll still have a bit of work figuring this puzzle out.
I needed to include kernel-modules matching the running kernel and figured out how to do that. The test ran OK on rawhide, f30 and f29. f28 had a problem, but it looked like it was with getting a consistant set of packages installed. With only a week left for it, I'm not going to worry about it too much.
The script I started with, was from a test case meant to be used for doing prerelease testing manually, that I wrote a few years ago. The new system is much nicer. For stuff that is mostly hardware independent this will save a lot of work. This is a case where it would be nice to test on alternate arches someday.
On Wed, 22 May 2019 at 16:44, Bruno Wolff III bruno@wolff.to wrote:
On Wed, May 22, 2019 at 16:25:16 +0200, Petr Šplíchal psplicha@redhat.com wrote:
On Wed, 22 May 2019 at 14:55, Bruno Wolff III bruno@wolff.to wrote:
You could do something like this:
- hosts: localhost
roles:
- role: standard-test-basic tags:
tests:
- classic
- compression: dir: . run: "./Squashfs-compression-test.sh 2>&1 | tee /tmp/test.log && !
grep -iq failed /tmp/test.log" required_packages:
- kernel-modules-extra
- rpmdevtools
That's pretty close to what I was thinking and I may switch to it after I figure out what appears to be a missing required package. I also think I'll need to negate the gtrep test, but there is a way to do that that I've used in the past, but don't remember off hand.
Which looks much more concise.
I agree that it will be easier to understand. So I'll probably switch.
What exactly in the docs should cause the confusion? Should any of these pages be updated?
The wrapper stuff has you copying the log and results to {{ artifacts }}, which isn't correct. The correct location is ../../logs/ . I don't know if that is in a variable someplace. I figured it out by looking at the pipeline logs. It was very confusing trying to figure out what was going on and I wasted a fair amount of time before deciding the documentation must be wrong.
This is weird. The "artifacts" variable should be set by the pipeline and thus using it instead of hardcoding some path used in the current implementation should be more reliable.
Bruno (Goncalves), could you please confirm that "artifacts" are correctly set in the Fedora pipeline? Thanks.
psss...
Including the "[citest]" string in a pull request comment does rerun all enabled tests.
I found that and it is useful, but not all builds come through pull requests. I'm not sure how the gate is connecting builds and tests. If it is by latest build of a commit, then redoing a pull test will work. But I forgot there is also a waive process, so worst case I can waive a test if need be. _______________________________________________ CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
On Wed, 22 May 2019 at 16:44, Bruno Wolff III <bruno(a)wolff.to> wrote:
This is weird. The "artifacts" variable should be set by the pipeline and thus using it instead of hardcoding some path used in the current implementation should be more reliable.
Bruno (Goncalves), could you please confirm that "artifacts" are correctly set in the Fedora pipeline? Thanks.
by artifacts you mean the TEST_ARTIFACTS variable as defined on [1]? the PR that we are talking about is [2]? If so, the ansible {{ artifacts }} variable is not defined in the playbook at all. The pipeline doesn't set it before running the playbook.
I believe Petr suggestion to use standard-test-basic role is the best, as it will save you the work on handling all these variables. But in case you want you want use your PR, I think you just need to set artifacts based on TEST_ARTIFACTS value. Like on [3].
[1] https://docs.fedoraproject.org/en-US/ci/standard-test-interface/#_invocation [2] https://src.fedoraproject.org/rpms/squashfs-tools/pull-request/4 [3] https://pagure.io/standard-test-roles/blob/master/f/roles/str-common-init/de...
psss...
Including the "[citest]" string in a pull request comment does rerun all enabled tests.
I found that and it is useful, but not all builds come through pull requests. I'm not sure how the gate is connecting builds and tests. If it is by latest build of a commit, then redoing a pull test will work. But I forgot there is also a waive process, so worst case I can waive a test if need be. _______________________________________________ CI mailing list -- ci(a)lists.fedoraproject.org To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
Thanks for clarifying, Bruno. I've updated docs accordingly: https://pagure.io/fedora-ci/docs/c/a5aebfe
psss...
On Thu, 23 May 2019 at 15:16, Bruno Goncalves bgoncalv@redhat.com wrote:
On Wed, 22 May 2019 at 16:44, Bruno Wolff III <bruno(a)wolff.to> wrote:
This is weird. The "artifacts" variable should be set by the pipeline and thus using it instead of hardcoding some path used in the current implementation should be more reliable.
Bruno (Goncalves), could you please confirm that "artifacts" are correctly set in the Fedora pipeline? Thanks.
by artifacts you mean the TEST_ARTIFACTS variable as defined on [1]? the PR that we are talking about is [2]? If so, the ansible {{ artifacts }} variable is not defined in the playbook at all. The pipeline doesn't set it before running the playbook.
I believe Petr suggestion to use standard-test-basic role is the best, as it will save you the work on handling all these variables. But in case you want you want use your PR, I think you just need to set artifacts based on TEST_ARTIFACTS value. Like on [3].
[1] https://docs.fedoraproject.org/en-US/ci/standard-test-interface/#_invocation [2] https://src.fedoraproject.org/rpms/squashfs-tools/pull-request/4 [3] https://pagure.io/standard-test-roles/blob/master/f/roles/str-common-init/de...
psss...
Including the "[citest]" string in a pull request comment does rerun all enabled tests.
I found that and it is useful, but not all builds come through pull requests. I'm not sure how the gate is connecting builds and tests. If it is by latest build of a commit, then redoing a pull test will work. But I forgot there is also a waive process, so worst case I can waive a test if need be. _______________________________________________ CI mailing list -- ci(a)lists.fedoraproject.org To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
On Thu, May 23, 2019 at 15:44:37 +0200, Petr Šplíchal psplicha@redhat.com wrote:
Thanks for clarifying, Bruno. I've updated docs accordingly: https://pagure.io/fedora-ci/docs/c/a5aebfe
Thanks. I'll update my test case to use that. I'm going to keep it as a wrapper for now, in case anyone else wants a known working example to look at. It also demos adding packages with the dnf module and installing kernel-modules matching the running kernel (I actually got mismatched ones in some of my testing.)
Thanks for updating the docs.
ci@lists.stg.fedoraproject.org
-
Bruno Goncalves -
Bruno Wolff III -
Petr Šplíchal