Looks like Ansible has been chosen as the way we're going to invoke tests stored in dist-git.
https://fedoraproject.org/wiki/Changes/InvokingTests https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible
Even for those who voted for alternatives, the good news is that almost all of both the packaged tests and the autopkgtest style tests can be wrapped in the Ansible style tests.
I did some work to try out the Ansible proposal and finish it up where there were rough edges or mysterious parts. I talked about these changes with Pingou, Ari, and Martin.
If you're interested in the details of what was changed or finished up in the Ansible proposal:
https://fedoraproject.org/w/index.php?title=Changes%2FInvokingTestsAnsible&a...
In addition some examples have been written. In particular, the sed tests here work both in-situ and against a composed Fedora Atomic host image:
https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible#Examples
https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible#Example:_Beakerl...
There'll be more updates to the examples on those pages, including finishing them up for the standard as it is, and making some parts of them more Ansibley.
Cheers,
Stef
On Wed, Apr 19, 2017 at 10:54 AM, Stef Walter stefw@redhat.com wrote:
Looks like Ansible has been chosen as the way we're going to invoke tests stored in dist-git.
https://fedoraproject.org/wiki/Changes/InvokingTests https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible
Even for those who voted for alternatives, the good news is that almost all of both the packaged tests and the autopkgtest style tests can be wrapped in the Ansible style tests.
I did some work to try out the Ansible proposal and finish it up where there were rough edges or mysterious parts. I talked about these changes with Pingou, Ari, and Martin.
If you're interested in the details of what was changed or finished up in the Ansible proposal:
https://fedoraproject.org/w/index.php?title=Changes% 2FInvokingTestsAnsible&diff=491284&oldid=490960
In addition some examples have been written. In particular, the sed tests here work both in-situ and against a composed Fedora Atomic host image:
https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible#Examples
https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible#Example:_ Beakerlib_based_test
There'll be more updates to the examples on those pages, including finishing them up for the standard as it is, and making some parts of them more Ansibley.
The proposition I proposed hasn't been really reviewed. I don't know the exact process for taking decisions like this but it seems rushed.
I'm insisting on my proposal because that's more a superset of the 2 other propositions than an alternative: you can package your tests or write them in ansible without any problem with the added values of having metadata about the tests like needing a real system, needing root access etc. and also being able to reuse tests from the Debian and Ubuntu communities.
Fred
On 19.04.2017 11:29, Frederic Lepied wrote:
On Wed, Apr 19, 2017 at 10:54 AM, Stef Walter <stefw@redhat.com mailto:stefw@redhat.com> wrote:
Looks like Ansible has been chosen as the way we're going to invoke tests stored in dist-git. https://fedoraproject.org/wiki/Changes/InvokingTests <https://fedoraproject.org/wiki/Changes/InvokingTests> https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible <https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible> Even for those who voted for alternatives, the good news is that almost all of both the packaged tests and the autopkgtest style tests can be wrapped in the Ansible style tests. I did some work to try out the Ansible proposal and finish it up where there were rough edges or mysterious parts. I talked about these changes with Pingou, Ari, and Martin. If you're interested in the details of what was changed or finished up in the Ansible proposal: https://fedoraproject.org/w/index.php?title=Changes%2FInvokingTestsAnsible&diff=491284&oldid=490960 <https://fedoraproject.org/w/index.php?title=Changes%2FInvokingTestsAnsible&diff=491284&oldid=490960> In addition some examples have been written. In particular, the sed tests here work both in-situ and against a composed Fedora Atomic host image: https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible#Examples <https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible#Examples> https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible#Example:_Beakerlib_based_test <https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible#Example:_Beakerlib_based_test> There'll be more updates to the examples on those pages, including finishing them up for the standard as it is, and making some parts of them more Ansibley.The proposition I proposed hasn't been really reviewed. I don't know the exact process for taking decisions like this but it seems rushed.
I'll add my review. Have you reached out folks to review it? Make sure to email them?
The conclusion here is one driven by those who are going to do the work migrating and wrapping the tests. By and large they have indicated they prefer doing that with Ansible as the standard interface. It's up to alternatives to reach out to them (see those who voted) and talk with them to change their minds.
I'm insisting on my proposal because that's more a superset of the 2 other propositions than an alternative: you can package your tests or write them in ansible without any problem with the added values of having metadata about the tests like needing a real system, needing root access etc. and also being able to reuse tests from the Debian and Ubuntu communities.
This works with the three proposals in various combinations. The staging, invocation and results can be wrapped either the following ways:
* autopkgtest + control can wrap and execute tests written in Ansible * Ansible can execute tests wrapped in autopkgtest + control files * Ansible tests can execute packaged tests * Packaged tests can execute Ansible tests * autopkgtest + control can execute packaged tests
In fact we already see some of these combinations in the examples. For example GLib2 or Cockpit package their own tests in RPMs. These are then executed by the tests laid out as Ansible tests.
Cheers,
Stef
On Wed, Apr 19, 2017 at 12:26 PM, Stef Walter stefw@redhat.com wrote:
On 19.04.2017 11:29, Frederic Lepied wrote:
On Wed, Apr 19, 2017 at 10:54 AM, Stef Walter <stefw@redhat.com mailto:stefw@redhat.com> wrote:
Looks like Ansible has been chosen as the way we're going to invoke tests stored in dist-git. https://fedoraproject.org/wiki/Changes/InvokingTests <https://fedoraproject.org/wiki/Changes/InvokingTests> https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible <https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible> Even for those who voted for alternatives, the good news is thatalmost
all of both the packaged tests and the autopkgtest style tests can be wrapped in the Ansible style tests. I did some work to try out the Ansible proposal and finish it upwhere
there were rough edges or mysterious parts. I talked about thesechanges
with Pingou, Ari, and Martin. If you're interested in the details of what was changed or finishedup
in the Ansible proposal: https://fedoraproject.org/w/index.php?title=Changes%2FInvokingTestsAnsible&diff=491284&oldid=490960
<https://fedoraproject.org/w/index.php?title=Changes%2FInvokingTestsAnsible&diff=491284&oldid=490960>
In addition some examples have been written. In particular, the sed tests here work both in-situ and against a composed Fedora Atomichost
image: https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible#Examples <https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible#Examples>
https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible#Example:_Beakerlib_based_test
<https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible#Example:_Beakerlib_based_test>
There'll be more updates to the examples on those pages, including finishing them up for the standard as it is, and making some parts of them more Ansibley.The proposition I proposed hasn't been really reviewed. I don't know the exact process for taking decisions like this but it seems rushed.
I'll add my review. Have you reached out folks to review it? Make sure to email them?
Yes I sent an email but got a few out of office replies :-(
Shouldn't we send an email to fedora-devel to have more feedback?
The conclusion here is one driven by those who are going to do the work migrating and wrapping the tests. By and large they have indicated they prefer doing that with Ansible as the standard interface. It's up to alternatives to reach out to them (see those who voted) and talk with them to change their minds.
I'm insisting on my proposal because that's more a superset of the 2 other propositions than an alternative: you can package your tests or write them in ansible without any problem with the added values of having metadata about the tests like needing a real system, needing root access etc. and also being able to reuse tests from the Debian and Ubuntu communities.
This works with the three proposals in various combinations. The staging, invocation and results can be wrapped either the following ways:
- autopkgtest + control can wrap and execute tests written in Ansible
- Ansible can execute tests wrapped in autopkgtest + control files
- Ansible tests can execute packaged tests
- Packaged tests can execute Ansible tests
- autopkgtest + control can execute packaged tests
In fact we already see some of these combinations in the examples. For example GLib2 or Cockpit package their own tests in RPMs. These are then executed by the tests laid out as Ansible tests.
I agree with this except that we cannot discover which packages are involved for a particular test using the ansible approach. The only package that will trigger the test will be the package that is being built while with the other 2 approaches you can have a finer granularity in the other packages needed for a particular test (with the control approach) or for the all the tests (with the packaging approach).
Fred
On Wed, Apr 19, 2017 at 12:34:56PM +0200, Frederic Lepied wrote:
On Wed, Apr 19, 2017 at 12:26 PM, Stef Walter stefw@redhat.com wrote:
Shouldn't we send an email to fedora-devel to have more feedback?
We announced the present mailing list on devel-announce which is automatically CC'd to devel itself. So folks interested on CI know that this is the place to be.
I think we all recognized the Ansible proposal has some limitations, but looking at the people who reviewed, the majority has a preference for it.
In fact we already see some of these combinations in the examples. For example GLib2 or Cockpit package their own tests in RPMs. These are then executed by the tests laid out as Ansible tests.I agree with this except that we cannot discover which packages are involved for a particular test using the ansible approach. The only package that will trigger the test will be the package that is being built while with the other 2 approaches you can have a finer granularity in the other packages needed for a particular test  (with the control approach) or for the all the tests (with the packaging approach).
Having reviewed the Control approach and having asked questions¹ on the wiki page, I must say that the control approach seems very close to me to the ansible one, with one big pro: re-using existing tests from other distros and one big con: entirely new tool in the Fedora ecosystem at large. The other level of granularity (allowing to specify if it requires root, needs a dedicated box) don't really matter since afaiu the idea is to invoke the tests as root anyway (up to the tests to change to another user if they want to) in a different VM every time.
Another thing I am missing with control is how would we test different artifact? (an RPM, a OCI container, a qcow image, an iso, an Atomic Host image).
Finally, as Stef said, what matters most if the opinion of the persons who will be actually writing new tests and porting existing tests to this system and looking at the wiki page they seemed quite in favor of the Ansible based proposal.
Pierre
¹: I see that Martin replied to the questions, I'll read the links he gave shortly.
On Wed, Apr 19, 2017 at 2:01 PM, Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Wed, Apr 19, 2017 at 12:34:56PM +0200, Frederic Lepied wrote:
On Wed, Apr 19, 2017 at 12:26 PM, Stef Walter stefw@redhat.com
wrote:
Shouldn't we send an email to fedora-devel to have more feedback?
We announced the present mailing list on devel-announce which is automatically CC'd to devel itself. So folks interested on CI know that this is the place to be.
Yes I know that but we didn't really try to broader the discussion to announce the vote because that's not a CI only discussion: packagers and QA should be involved too.
I think we all recognized the Ansible proposal has some limitations, but looking at the people who reviewed, the majority has a preference for it.
It was a little bit rushed as nobody except Stef and you gave feedback on the new proposition.
In fact we already see some of these combinations in the examples.For
example GLib2 or Cockpit package their own tests in RPMs. These arethen
executed by the tests laid out as Ansible tests.I agree with this except that we cannot discover which packages are involved for a particular test using the ansible approach. The only package that will trigger the test will be the package that is being
built
while with the other 2 approaches you can have a finer granularity in
the
other packages needed for a particular test  (with the control
approach)
or for the all the tests (with the packaging approach).
Having reviewed the Control approach and having asked questions¹ on the wiki page, I must say that the control approach seems very close to me to the ansible one, with one big pro: re-using existing tests from other distros and one big con: entirely new tool in the Fedora ecosystem at large. The other level of granularity (allowing to specify if it requires root, needs a dedicated box) don't really matter since afaiu the idea is to invoke the tests as root anyway (up to the tests to change to another user if they want to) in a different VM every time.
Another thing I am missing with control is how would we test different artifact? (an RPM, a OCI container, a qcow image, an iso, an Atomic Host image).
There is no such support to my knowledge but I can see some ways to implement it. Regarding the Ansible proposition, I don't see either how you'll declare that you are going to test a different artifact than an rpm.
Fred
On Wed, Apr 19, 2017 at 02:45:44PM +0200, Frederic Lepied wrote:
There is no such support to my knowledge but I can see some ways to implement it.
That is one con for the control proposal as it means that it'll need work before people can use it and I have absolutely no idea how much work is involved in this.
I now realized that this was something discussed early on that never made it to the requirement list: rely on existing tooling as much as possible.
Any idea how much work it would be? And would upstream take such patches?
Regarding the Ansible proposition, I don't see either how you'll declare that you are going to test a different artifact than an rpm.
Stef added this part in the diff he linked to earlier. The idea is to have multiple playbooks, one per artifact tested. Potentially re-using shared roles/playbooks for the common parts.
Does that sound reasonable?
Pierre
On 19.04.2017 15:05, Pierre-Yves Chibon wrote:
On Wed, Apr 19, 2017 at 02:45:44PM +0200, Frederic Lepied wrote:
There is no such support to my knowledge but I can see some ways to implement it.
That is one con for the control proposal as it means that it'll need work before people can use it and I have absolutely no idea how much work is involved in this.
I now realized that this was something discussed early on that never made it to the requirement list: rely on existing tooling as much as possible.
Any idea how much work it would be? And would upstream take such patches?
Regarding the Ansible proposition, I don't see either how you'll declare that you are going to test a different artifact than an rpm.
Stef added this part in the diff he linked to earlier. The idea is to have multiple playbooks, one per artifact tested. Potentially re-using shared roles/playbooks for the common parts.
Does that sound reasonable?
There's a working example of this linked in my previous email. I'm playing with the implementation and examples a bit more, but the interface for handling multiple artifacts remains as documented:
https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible
Stef
Hi all,
Apologies for the necropost. If the Ansible Tests have been declared as the way forward, can we update the main page[1] to reflect this? Or is it still being decided?
[1] https://fedoraproject.org/wiki/Changes/InvokingTests
----- Original Message -----
Looks like Ansible has been chosen as the way we're going to invoke tests stored in dist-git.
https://fedoraproject.org/wiki/Changes/InvokingTests https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible
Even for those who voted for alternatives, the good news is that almost all of both the packaged tests and the autopkgtest style tests can be wrapped in the Ansible style tests.
I did some work to try out the Ansible proposal and finish it up where there were rough edges or mysterious parts. I talked about these changes with Pingou, Ari, and Martin.
If you're interested in the details of what was changed or finished up in the Ansible proposal:
https://fedoraproject.org/w/index.php?title=Changes%2FInvokingTestsAnsible&a...
In addition some examples have been written. In particular, the sed tests here work both in-situ and against a composed Fedora Atomic host image:
https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible#Examples
https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible#Example:_Beakerl...
There'll be more updates to the examples on those pages, including finishing them up for the standard as it is, and making some parts of them more Ansibley.
Cheers,
Stef
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org
-
Frederic Lepied -
Jonathan Lebon -
Pierre-Yves Chibon -
Stef Walter