Hi!
While working on adding CI tests [0] using the Standard Test Interface a need arose to have a shared git repository where tests could be stored:
* A large number of test files makes a dist-git repository more difficult to maintain
* Tests might follow a different branching pattern than the dist-git repo, leading to code duplication
* Shared maintenance for tests sometimes benefits from different access levels than the release dist-git repository
The plan is to create a new “tests” namespace in Fedora git/pagure dedicated to storing the shared test code. To enable execution of these tests by the CI pipeline, tests.yml file in dist-git will be used to link the tests in the standard way as defined by the Standard Test Interface [1].
This approach should help to efficiently maintain tests & minimize test code duplication. Using a dedicated git repo for test code also means to keep dist-git more as a place for storing metadata only: Build metadata (spec file = how to build the package) and test metadata (tests.yml = how to test the package) rather than mixing spec files with test code itself.
Please note that this does not mean that all tests should now go into this new namespace. You can still link tests directly from upstream (like GitHub) or any other source. Also, for unit tests it makes more sense to be kept directly with the project source and executed there. Shared tests namespace in Fedora will be suitable especially for functional and integration testing which should help to continuously ensure the OS works as a whole.
For more detailed motivation and real-life examples see the Share Test Code proposal on the Fedora CI list [2]. If you have any questions or comments feel free to share them here or in the pagure issue requesting the new namespace:
https://pagure.io/fedora-infrastructure/issue/6478
Thanks.
psss...
[0] https://fedoraproject.org/wiki/CI [1] https://fedoraproject.org/wiki/Changes/InvokingTests [2] https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/thr...
Hi!
During the last days there have been concerns raised regarding what is an appropriate content for the tests namespace. [1] My original idea was to enable sharing tests even across branches of the same component, not only for tests to be used by completely different packages. The initial examples might have been a bit misleading in this respect. One of the main points still holds:
- Tests might follow a different branching pattern than the dist-git repo, leading to code duplication
From the feedback from developers I feel they always keep on mind and care a lot about the maintenance costs. So it perfectly makes sense to me if they want to keep and maintain tests in a separate repo instead of merging/cherry-picking between dist-git branches.
When, for a particular package, it is the most efficient way to maintain tests in a separate repo why should we discourage from this approach? There are packages where it makes more sense to store test code directly in dist-git. And it is still an option. But why should we enforce this for all?
Please share your thoughts and real-life examples. For those who are not familiar with the topic there is a new wiki page with a summary of the Share Test Code approach [2].
psss...
[1] https://pagure.io/fedora-infrastructure/issue/6695 [2] https://fedoraproject.org/wiki/CI/Share_Test_Code
On 7 December 2017 at 10:38, Petr Splichal psplicha@redhat.com wrote:
Hi!
While working on adding CI tests [0] using the Standard Test Interface a need arose to have a shared git repository where tests could be stored:
A large number of test files makes a dist-git repository more difficult to maintain
Tests might follow a different branching pattern than the dist-git repo, leading to code duplication
Shared maintenance for tests sometimes benefits from different access levels than the release dist-git repository
The plan is to create a new “tests” namespace in Fedora git/pagure dedicated to storing the shared test code. To enable execution of these tests by the CI pipeline, tests.yml file in dist-git will be used to link the tests in the standard way as defined by the Standard Test Interface [1].
This approach should help to efficiently maintain tests & minimize test code duplication. Using a dedicated git repo for test code also means to keep dist-git more as a place for storing metadata only: Build metadata (spec file = how to build the package) and test metadata (tests.yml = how to test the package) rather than mixing spec files with test code itself.
Please note that this does not mean that all tests should now go into this new namespace. You can still link tests directly from upstream (like GitHub) or any other source. Also, for unit tests it makes more sense to be kept directly with the project source and executed there. Shared tests namespace in Fedora will be suitable especially for functional and integration testing which should help to continuously ensure the OS works as a whole.
For more detailed motivation and real-life examples see the Share Test Code proposal on the Fedora CI list [2]. If you have any questions or comments feel free to share them here or in the pagure issue requesting the new namespace:
https://pagure.io/fedora-infrastructure/issue/6478
Thanks.
psss...
[0] https://fedoraproject.org/wiki/CI [1] https://fedoraproject.org/wiki/Changes/InvokingTests [2] https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/thr...
On 02/14/2018 05:28 PM, Petr Šplíchal wrote:
Hi!
During the last days there have been concerns raised regarding what is an appropriate content for the tests namespace. [1] My original idea was to enable sharing tests even across branches of the same component, not only for tests to be used by completely different packages. The initial examples might have been a bit misleading in this respect. One of the main points still holds:
- Tests might follow a different branching pattern than the dist-git repo, leading to code duplication
From the feedback from developers I feel they always keep on mind and care a lot about the maintenance costs. So it perfectly makes sense to me if they want to keep and maintain tests in a separate repo instead of merging/cherry-picking between dist-git branches.
+1
From my PoV, it makes perfect sense to share tests between branches, or between different versions of packages available in modules streams (soon).
Honza
When, for a particular package, it is the most efficient way to maintain tests in a separate repo why should we discourage from this approach? There are packages where it makes more sense to store test code directly in dist-git. And it is still an option. But why should we enforce this for all?
Please share your thoughts and real-life examples. For those who are not familiar with the topic there is a new wiki page with a summary of the Share Test Code approach [2].
psss...
[1] https://pagure.io/fedora-infrastructure/issue/6695 [2] https://fedoraproject.org/wiki/CI/Share_Test_Code
On 7 December 2017 at 10:38, Petr Splichal psplicha@redhat.com wrote:
Hi!
While working on adding CI tests [0] using the Standard Test Interface a need arose to have a shared git repository where tests could be stored:
A large number of test files makes a dist-git repository more difficult to maintain
Tests might follow a different branching pattern than the dist-git repo, leading to code duplication
Shared maintenance for tests sometimes benefits from different access levels than the release dist-git repository
The plan is to create a new “tests” namespace in Fedora git/pagure dedicated to storing the shared test code. To enable execution of these tests by the CI pipeline, tests.yml file in dist-git will be used to link the tests in the standard way as defined by the Standard Test Interface [1].
This approach should help to efficiently maintain tests & minimize test code duplication. Using a dedicated git repo for test code also means to keep dist-git more as a place for storing metadata only: Build metadata (spec file = how to build the package) and test metadata (tests.yml = how to test the package) rather than mixing spec files with test code itself.
Please note that this does not mean that all tests should now go into this new namespace. You can still link tests directly from upstream (like GitHub) or any other source. Also, for unit tests it makes more sense to be kept directly with the project source and executed there. Shared tests namespace in Fedora will be suitable especially for functional and integration testing which should help to continuously ensure the OS works as a whole.
For more detailed motivation and real-life examples see the Share Test Code proposal on the Fedora CI list [2]. If you have any questions or comments feel free to share them here or in the pagure issue requesting the new namespace:
https://pagure.io/fedora-infrastructure/issue/6478
Thanks.
psss...
[0] https://fedoraproject.org/wiki/CI [1] https://fedoraproject.org/wiki/Changes/InvokingTests [2] https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/thr...
On Wed, 2018-02-14 at 17:28 +0100, Petr Šplíchal wrote:
Hi!
During the last days there have been concerns raised regarding what is an appropriate content for the tests namespace. [1] My original idea was to enable sharing tests even across branches of the same component, not only for tests to be used by completely different packages. The initial examples might have been a bit misleading in this respect. One of the main points still holds:
- Tests might follow a different branching pattern than the dist-git repo, leading to code duplication
From the feedback from developers I feel they always keep on mind and care a lot about the maintenance costs. So it perfectly makes sense to me if they want to keep and maintain tests in a separate repo instead of merging/cherry-picking between dist-git branches.
When, for a particular package, it is the most efficient way to maintain tests in a separate repo why should we discourage from this approach? There are packages where it makes more sense to store test code directly in dist-git. And it is still an option. But why should we enforce this for all?
Please share your thoughts and real-life examples. For those who are not familiar with the topic there is a new wiki page with a summary of the Share Test Code approach [2].
I don't have any problem with the concept *in itself*; in fact I know of another reason to have something like it. That is 'generic' tests, tests we want to run on all packages, or at least a very large set of packages - like the tests currently running in Taskotron, which are generally run on all packages (rpmgrill, rpmdeplint...) or a large subset (the Python tests).
What I did think was maybe a concern is that we should figure out in advance the squishy human consequences of different technical approaches.
Basically this boiled down to "who is responsible for these 'shared' tests, and who gets to decide which 'shared' tests can/should block packages?"
Looking at the proposal, I think it actually has at least workable initial explicit/implicit answers to this, if I'm reading it correctly. Anyone can create a shared test repository (and is therefore "responsible" for maintaining it), but the definition of which tests are run on which packages remains in the package repositories. Thus the package maintainer(s) retain the ultimate choice over which tests are run against their packages (and thus can pick which shared tests are run, and disable shared tests if they are no longer properly testing their package).
The question 'who decides which tests block which packages' is left a bit up in the air, but in fact no more so than it already was for package-specific tests...
On 14 February 2018 at 18:18, Adam Williamson adamwill@fedoraproject.org wrote:
On Wed, 2018-02-14 at 17:28 +0100, Petr Šplíchal wrote:
Hi!
During the last days there have been concerns raised regarding what is an appropriate content for the tests namespace. [1] My original idea was to enable sharing tests even across branches of the same component, not only for tests to be used by completely different packages. The initial examples might have been a bit misleading in this respect. One of the main points still holds:
- Tests might follow a different branching pattern than the
dist-git repo, leading to code duplication
From the feedback from developers I feel they always keep on mind and care a lot about the maintenance costs. So it perfectly makes sense to me if they want to keep and maintain tests in a separate repo instead of merging/cherry-picking between dist-git branches.
When, for a particular package, it is the most efficient way to maintain tests in a separate repo why should we discourage from this approach? There are packages where it makes more sense to store test code directly in dist-git. And it is still an option. But why should we enforce this for all?
Please share your thoughts and real-life examples. For those who are not familiar with the topic there is a new wiki page with a summary of the Share Test Code approach [2].
I don't have any problem with the concept *in itself*; in fact I know of another reason to have something like it. That is 'generic' tests, tests we want to run on all packages, or at least a very large set of packages - like the tests currently running in Taskotron, which are generally run on all packages (rpmgrill, rpmdeplint...) or a large subset (the Python tests).
Yes, sharing tests useful for multiple packages makes sense in general. Test code for generic tests can be stored basically anywhere but not in specific components. However I see some additional benefits of having a dedicated tests namespace:
* Source of inspiration and real-life examples (at one place) * Potential for workflow automation (tests CI, fedpkg --tests) * Easy cherry-picking/backporting upstream/downstream tests
The difference I see when comparing these shared tests to generic tests mentioned above like rpmdeplint is that you don't want to reference these from the individual package repos (e.g. using the Standard Test Interface), but just run them on all packages.
If the expectation is to run tests only on a subset of rpms then a question comes: How to store the list of relevant components or criteria to decide which packages are to be tested or not. Here the Flexible Metadata Format might come handy in the future:
What I did think was maybe a concern is that we should figure out in advance the squishy human consequences of different technical approaches.
Basically this boiled down to "who is responsible for these 'shared' tests, and who gets to decide which 'shared' tests can/should block packages?"
Looking at the proposal, I think it actually has at least workable initial explicit/implicit answers to this, if I'm reading it correctly. Anyone can create a shared test repository (and is therefore "responsible" for maintaining it), but the definition of which tests are run on which packages remains in the package repositories. Thus the package maintainer(s) retain the ultimate choice over which tests are run against their packages (and thus can pick which shared tests are run, and disable shared tests if they are no longer properly testing their package).
Yes. This is exactly true. The maintainer of the package repo has always the last word on what is to be executed for qualification. It can be done by referencing only suitable tests from the shared repo or selecting an appropriate branch or even a specific commit to ensure that future test development does not break stable gating (and only manually update reference after review of shared test updates) or even completely disable tests by removing the reference.
The question 'who decides which tests block which packages' is left a bit up in the air, but in fact no more so than it already was for package-specific tests...
Right. Do we want to change this? Specify this more strictly? Like exactly defining requirements which an Always Ready Operating System has to meet? And then mapping these requirements to the test coverage? (Here again FMF mentioned above might come handy.)
psss...
-- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net
On Thu, 2018-02-15 at 13:11 +0100, Petr Šplíchal wrote:
The question 'who decides which tests block which packages' is left a bit up in the air, but in fact no more so than it already was for package-specific tests...
Right. Do we want to change this? Specify this more strictly? Like exactly defining requirements which an Always Ready Operating System has to meet? And then mapping these requirements to the test coverage? (Here again FMF mentioned above might come handy.)
Maybe we should just *formalize* it a bit, so folks know where they stand. So far we've basically done three things that implicitly describe the policy on this:
1) Bodhi has a mechanism by which the person submitting an update can specify some tests that block the update
2) FESCo accepted, voted on and approved a ticket which involved a decision that certain tests blocked some/all updates
3) We built and deployed WaiverDB and Greenwave such that Greenwave currently accepts waivers submitted to WaiverDB by...(anybody??? this point I'm not sure on)
So, effectively, we have communicated that FESCo can set mandatory policies here, and packagers can also make voluntary choices. Point 3) is rather a crucial one, and I don't know that it's been formally discussed anywhere; I think we have been rather heavily focused on building the tools as opposed to setting the policy so far. But we've at least clearly implied so far that FESCo claims to some extent the power to set automated test gating requirements for updates, and that packagers have some ability to extend those requirements voluntarily, and also submit waivers. I guess the natural thing to do from here, if we want to be a bit more explicit about it, would be to extend the Updates Policy:
https://fedoraproject.org/wiki/Updates_Policy
either in-line or with a subsidiary document, to define this stuff a bit more clearly/formally.
On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
Hi!
During the last days there have been concerns raised regarding what is an appropriate content for the tests namespace. [1] My original idea was to enable sharing tests even across branches of the same component, not only for tests to be used by completely different packages. The initial examples might have been a bit misleading in this respect. One of the main points still holds:
- Tests might follow a different branching pattern than the dist-git repo, leading to code duplication
From the feedback from developers I feel they always keep on mind and care a lot about the maintenance costs. So it perfectly makes sense to me if they want to keep and maintain tests in a separate repo instead of merging/cherry-picking between dist-git branches.
When, for a particular package, it is the most efficient way to maintain tests in a separate repo why should we discourage from this approach? There are packages where it makes more sense to store test code directly in dist-git. And it is still an option. But why should we enforce this for all?
My worries are basically that this mechanism is hiding the tests from the package maintainers. It splits the concerns between people maintaining the artifacts and people maintaining the tests, which is exactly what the original plan to bring CI was *not* about. The idea has always been to bring the tests where the code lives, so that both can be worked on as one, to make tests be a concern of the package maintainer very much like updating the artifact (rpms, image..) is, while getting support from QE for them (the tests).
In addition, this is what I fear most: The tests for the package are stored elsewhere, we're not sure where (the tests namespace, a random git repo on the internet...), the packager update package and the tests fail. What do you think will happen? a) the packager will look for $random_place_of_the_internet where tests are and spend time trying to fix them. b) the packager will turn off the tests because they get in the way
If the packager wants to turn off the tests, they will have to go through dist-git to do it, they may very well end up turning the tests off anyway, but if the tests are right there, they may as well have a quick look at them to see if they can fix them quickly before deciding.
In addition, if the packager turn the tests off and the people maintaining the tests do not realize that, they will end up spending time maintaining $somewhere_else tests that aren't being used. Having them interact directly with the dist-git repo will increase the chances that they see/realize when something is turned off.
If that means we have less tests in dist-git because the maintainers do not want them, I'd say so be it. In the long term this is their loss, they are the ones who will get the bug reports and will have to deal with them, they are the ones who will have to dive into old code rather than going back into something that is still fresh in their mind. As long as, it is clear that they have been approached and that it is their choice to not merge pull-requests adding tests, I think the people offering to help should not be the ones blamed.
Pierre
On 22 February 2018 at 15:50, Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
Hi!
During the last days there have been concerns raised regarding what is an appropriate content for the tests namespace. [1] My original idea was to enable sharing tests even across branches of the same component, not only for tests to be used by completely different packages. The initial examples might have been a bit misleading in this respect. One of the main points still holds:
- Tests might follow a different branching pattern than the dist-git repo, leading to code duplication
From the feedback from developers I feel they always keep on mind and care a lot about the maintenance costs. So it perfectly makes sense to me if they want to keep and maintain tests in a separate repo instead of merging/cherry-picking between dist-git branches.
When, for a particular package, it is the most efficient way to maintain tests in a separate repo why should we discourage from this approach? There are packages where it makes more sense to store test code directly in dist-git. And it is still an option. But why should we enforce this for all?
Thanks for sharing your thoughts, Pierre.
My worries are basically that this mechanism is hiding the tests from the package maintainers. It splits the concerns between people maintaining the artifacts and people maintaining the tests, which is exactly what the original plan to bring CI was *not* about.
I do not see a split here but rather potential for enhanced or extended collaboration. Repositories in the tests namespace can be set up with more open access than rpms git. In this way devel can invite qe for direct collaboration on tests. That's one of the reasons why we should try to fix the non-packager access as soon as possible.
https://pagure.io/fedora-infrastructure/issue/6361
The idea has always been to bring the tests where the code lives, so that both can be worked on as one, to make tests be a concern of the package maintainer very much like updating the artifact (rpms, image..) is, while getting support from QE for them (the tests).
I see the benefits of having code with tests at the very same place, I find this very useful especially for unit tests in the project upstream. However, the situation here is a bit different in two respects:
1) The rpms dist git does not include package code itself but rather build metadata only (mainly spec files describing how to build the package) so it makes very good sense to host at this place testing metadata only (how to test the package). This is quite nicely consistent approach.
2) As far as I can see, within the Always Ready Operating System effort we are not looking to cover every and each feature in all packages but rather ensuring the OS works as a whole. Thus basic functionality and integration tests which guard the stability of the packages are more appropriate for testing. These do not need to be kept so close to the package code.
I believe we don't want to duplicate upstream unit testing here. We should probably clarify more which test types are appropriate or expected in the Fedora CI. As part of the docs consolidation I added a brief section about test types to the main CI portal:
https://fedoraproject.org/wiki/CI#Test_Types
Does it decribe well what we are looking for? Shall we extend it?
In addition, this is what I fear most: The tests for the package are stored elsewhere, we're not sure where (the tests namespace, a random git repo on the internet...), the packager update package and the tests fail.
I see no hiding of tests from maintainers: Tests are clearly referenced and discoverable from the rpms dist git as defined by the Standard Test Interface and devels have access, see:
https://fedoraproject.org/wiki/CI/Share_Test_Code#Examples
Devel decides which tests are good for testing, it does not matter where the test code lives. However there needs to be trust to the referenced repo. But this is the same as with any other open source project.
What do you think will happen? a) the packager will look for $random_place_of_the_internet where tests are and spend time trying to fix them. b) the packager will turn off the tests because they get in the way
If the packager wants to turn off the tests, they will have to go through dist-git to do it, they may very well end up turning the tests off anyway, but if the tests are right there, they may as well have a quick look at them to see if they can fix them quickly before deciding.
It's not random_place_of_the_internet, it's well selected and trusted source where devel can (usually) directly contribute. I meet devels around who see benefit in having tests enabled. So I don't share your concern about turning the test off so easily.
In addition, if the packager turn the tests off and the people maintaining the tests do not realize that, they will end up spending time maintaining $somewhere_else tests that aren't being used.
Having them interact directly with the dist-git repo will increase the chances that they see/realize when something is turned off.
Working on something that is not being used supposes there is no communication/collaboration between devel and qe. And I doubt that enforcing inefficient workflow with test code duplication and less ACL flexibility will help to increase it. I believe we should set up the workflow as smooth as possible to encourage better collaboration.
If that means we have less tests in dist-git because the maintainers do not want them, I'd say so be it.
In the long term this is their loss, they are the ones who will get the bug reports and will have to deal with them, they are the ones who will have to dive into old code rather than going back into something that is still fresh in their mind.
As long as, it is clear that they have been approached and that it is their choice to not merge pull-requests adding tests, I think the people offering to help should not be the ones blamed.
The question is not about devels who do not want tests enabled in dist-git but about those who do not want to inefficiently maintain the test code there. So diving into old code does not apply here.
psss...
On Fri, Feb 23, 2018 at 10:19:35AM +0100, Petr Šplíchal wrote:
On 22 February 2018 at 15:50, Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
Hi!
During the last days there have been concerns raised regarding what is an appropriate content for the tests namespace. [1] My original idea was to enable sharing tests even across branches of the same component, not only for tests to be used by completely different packages. The initial examples might have been a bit misleading in this respect. One of the main points still holds:
- Tests might follow a different branching pattern than the dist-git repo, leading to code duplication
From the feedback from developers I feel they always keep on mind and care a lot about the maintenance costs. So it perfectly makes sense to me if they want to keep and maintain tests in a separate repo instead of merging/cherry-picking between dist-git branches.
When, for a particular package, it is the most efficient way to maintain tests in a separate repo why should we discourage from this approach? There are packages where it makes more sense to store test code directly in dist-git. And it is still an option. But why should we enforce this for all?
Thanks for sharing your thoughts, Pierre.
My worries are basically that this mechanism is hiding the tests from the package maintainers. It splits the concerns between people maintaining the artifacts and people maintaining the tests, which is exactly what the original plan to bring CI was *not* about.
I do not see a split here but rather potential for enhanced or extended collaboration. Repositories in the tests namespace can be set up with more open access than rpms git. In this way devel can invite qe for direct collaboration on tests.
I don't see how that's not possible with tests being in dist-git.
That's one of the reasons why we should try to fix the non-packager access as soon as possible.
https://pagure.io/fedora-infrastructure/issue/6361
Agreed and being worked on.
The idea has always been to bring the tests where the code lives, so that both can be worked on as one, to make tests be a concern of the package maintainer very much like updating the artifact (rpms, image..) is, while getting support from QE for them (the tests).
I see the benefits of having code with tests at the very same place, I find this very useful especially for unit tests in the project upstream. However, the situation here is a bit different in two respects:
- The rpms dist git does not include package code itself but
rather build metadata only (mainly spec files describing how to build the package) so it makes very good sense to host at this place testing metadata only (how to test the package). This is quite nicely consistent approach.
I still believe it creates a separation of concerns that is in opposition with the original idea.
- As far as I can see, within the Always Ready Operating System
effort we are not looking to cover every and each feature in all packages but rather ensuring the OS works as a whole. Thus basic functionality and integration tests which guard the stability of the packages are more appropriate for testing. These do not need to be kept so close to the package code.
I believe we don't want to duplicate upstream unit testing here. We should probably clarify more which test types are appropriate or expected in the Fedora CI. As part of the docs consolidation I added a brief section about test types to the main CI portal:
https://fedoraproject.org/wiki/CI#Test_TypesDoes it decribe well what we are looking for? Shall we extend it?
In addition, this is what I fear most: The tests for the package are stored elsewhere, we're not sure where (the tests namespace, a random git repo on the internet...), the packager update package and the tests fail.
I see no hiding of tests from maintainers: Tests are clearly referenced and discoverable
Referenced maybe, discoverable, much less than if they are right there.
from the rpms dist git as defined by the Standard Test Interface and devels have access, see:
If the tests are defined in dist-git and adding new tests means committing these changes to dist-git's yaml file, how does that solve the concern about "noise" in the git commits?
It seems to me that whether the commit touches just the yaml file or the yaml and tweak the tests themselves, the outcome is the same.
Devel decides which tests are good for testing, it does not matter where the test code lives. However there needs to be trust to the referenced repo. But this is the same as with any other open source project.
What do you think will happen? a) the packager will look for $random_place_of_the_internet where tests are and spend time trying to fix them. b) the packager will turn off the tests because they get in the way
If the packager wants to turn off the tests, they will have to go through dist-git to do it, they may very well end up turning the tests off anyway, but if the tests are right there, they may as well have a quick look at them to see if they can fix them quickly before deciding.
It's not random_place_of_the_internet, it's well selected and trusted source where devel can (usually) directly contribute. I meet devels around who see benefit in having tests enabled. So I don't share your concern about turning the test off so easily.
In addition, if the packager turn the tests off and the people maintaining the tests do not realize that, they will end up spending time maintaining $somewhere_else tests that aren't being used.
Having them interact directly with the dist-git repo will increase the chances that they see/realize when something is turned off.
Working on something that is not being used supposes there is no communication/collaboration between devel and qe. And I doubt that enforcing inefficient workflow with test code duplication and less
I don't see how this duplication works, if you need to touch the yaml file anyway, there will be just as much "duplication".
ACL flexibility will help to increase it.
How are ACLs less flexible than with any of the open source project out there? You start by submitting pull-request and at one point the people trust you enough to grant you direct commit access.
I believe we should set up the workflow as smooth as possible to encourage better collaboration.
If that means we have less tests in dist-git because the maintainers do not want them, I'd say so be it.
In the long term this is their loss, they are the ones who will get the bug reports and will have to deal with them, they are the ones who will have to dive into old code rather than going back into something that is still fresh in their mind.
As long as, it is clear that they have been approached and that it is their choice to not merge pull-requests adding tests, I think the people offering to help should not be the ones blamed.
The question is not about devels who do not want tests enabled in dist-git but about those who do not want to inefficiently maintain the test code there. So diving into old code does not apply here.
I don't see what's inefficient here sorry.
Pierre
My worries are basically that this mechanism is hiding the tests from the package maintainers. It splits the concerns between people maintaining the artifacts and people maintaining the tests, which is exactly what the original plan to bring CI was *not* about.
I do not see a split here but rather potential for enhanced or extended collaboration. Repositories in the tests namespace can be set up with more open access than rpms git. In this way devel can invite qe for direct collaboration on tests.
I don't see how that's not possible with tests being in dist-git.
What I've heard from devels is that they often don't feel like giving direct commit access to the rpms dist git to QE as spec files and packaging in general is a big responsiblity. But they are quite OK with directly collaborating with QE on test coverage.
If the tests are defined in dist-git and adding new tests means committing these changes to dist-git's yaml file, how does that solve the concern about "noise" in the git commits?
Not necessarily. You could use advanced metadata to select desired tests only once. The very first proof of concept of using Flexible Metadata Format for testing selinux quite nicely illustrates this:
https://src.fedoraproject.org/tests/selinux/pull-request/1 https://github.com/psss/fmf
Instead of listing individual tests you could say: Run all tests for this selinux component + all Tier1 selinux integration tests:
fmf --key test --filter 'component:libselinux | tags:Tier1'
For tests covering multiple components the benefit of the tests namespace approach is unquestionable. However, I see benefit of the approach for single components as well (and developers seem to often feel the same): That is not having to cherry-pick/merge test coverage between supported branches. So even for these cases there is the benefit of sharing the test code.
I believe that extending the integration testing coverage is one the most essential goals of Fedora CI. This is the crucial piece of the puzzle how to make an Always Ready Operating System: It is to ensure that packages play nicely together. So I guess there will be quite many tests which will benefit from this approach.
Do we really want/need to assess every new request for a tests repo if it is good-enough for the test namespace? Will there be a committee to decide whether a particular set of tests has a potential to be used by other components in the future or not? :)
To sum up what I've heard so far from the developer side:
* I would like to enable tests for my component (yes, I want) * I will take care of them (really, I see the benefit in CI) * I want to easily collaborate on tests with qe (direct commits) * I don't want to give qe commit access to the rpms dist git * I want to efficiently maintain tests long-term * I want to have just a single place for my tests (no duplication) * I want to keep rpms dist git clean (just metadata & patches) * I want to easily enable testing for all supported branches * I don't want to backport new tests to old branches (enable once) * I want to run all available relevant tests (not to list them) * I want to execute set of tests based on a tag (e.g. Tier1) * I want an easy way to clone tests (fedpkg clone tests/pkg)
I believe the tests namespace resolves them all.
psss...
On Thu, Mar 01, 2018 at 02:44:37PM +0100, Petr Šplíchal wrote:
My worries are basically that this mechanism is hiding the tests from the package maintainers. It splits the concerns between people maintaining the artifacts and people maintaining the tests, which is exactly what the original plan to bring CI was *not* about.
I do not see a split here but rather potential for enhanced or extended collaboration. Repositories in the tests namespace can be set up with more open access than rpms git. In this way devel can invite qe for direct collaboration on tests.
I don't see how that's not possible with tests being in dist-git.
What I've heard from devels is that they often don't feel like giving direct commit access to the rpms dist git to QE as spec files and packaging in general is a big responsiblity. But they are quite OK with directly collaborating with QE on test coverage.
If the tests are defined in dist-git and adding new tests means committing these changes to dist-git's yaml file, how does that solve the concern about "noise" in the git commits?
Not necessarily. You could use advanced metadata to select desired tests only once. The very first proof of concept of using Flexible Metadata Format for testing selinux quite nicely illustrates this:
https://src.fedoraproject.org/tests/selinux/pull-request/1 https://github.com/psss/fmfInstead of listing individual tests you could say: Run all tests for this selinux component + all Tier1 selinux integration tests:
fmf --key test --filter 'component:libselinux | tags:Tier1'For tests covering multiple components the benefit of the tests namespace approach is unquestionable. However, I see benefit of the approach for single components as well (and developers seem to often feel the same): That is not having to cherry-pick/merge test coverage between supported branches. So even for these cases there is the benefit of sharing the test code.
I believe that extending the integration testing coverage is one the most essential goals of Fedora CI. This is the crucial piece of the puzzle how to make an Always Ready Operating System: It is to ensure that packages play nicely together. So I guess there will be quite many tests which will benefit from this approach.
Do we really want/need to assess every new request for a tests repo if it is good-enough for the test namespace? Will there be a committee to decide whether a particular set of tests has a potential to be used by other components in the future or not? :)
To sum up what I've heard so far from the developer side:
- I would like to enable tests for my component (yes, I want)
- I will take care of them (really, I see the benefit in CI)
- I want to easily collaborate on tests with qe (direct commits)
- I don't want to give qe commit access to the rpms dist git
This is quite likely the crux of the problem.
Personally, I'm perfectly happy to give write access to any repo to people who have shown that they know what they are doing. We have pull requests in dist-git pagure now, and I think this is a better approach: 1. ask QE contributors to submit PRs 2. if enough cooperation happens and trust is established, give privileges to write to the repo directly, possibly with an agreement that this specific person should only touch tests, and not the packaging.
I think it's perfectly fine to never get to point 2.: many many upstream projects require a review from a second person, or sometimes even two reviews before a PR is merged, which means that one _never_ merges their own PR, and another contributor is always involved. We usually don't do this in dist-git, but I'm quite sure that requiring reviews for dist-git changes would raise quality and catch many silly mistakes. Either way, it's nowadays possible to cooperate using a single repo without fully trusting the other person, frictionlessly.
- I want to efficiently maintain tests long-term
- I want to have just a single place for my tests (no duplication)
- I don't want to backport new tests to old branches (enable once)
- I want to easily enable testing for all supported branches
Those four items depend strongly on the package. My thinking is biased by some specific usecases (mainly systemd), but I'm sure many other packages are like that: a lot of tests would be for new functionality, and then the tests _are_ tied to the branch being tested.
While I agree that keeping tests separate avoids a bit of effort required to update multiple branches, this effort isn't very big. If the tests indeed apply cleanly to all branches, then it's just a matter of doing 'git merge' or 'git cherry-pick' once per branch.
- I want to keep rpms dist git clean (just metadata & patches)
Meh.
- I want to run all available relevant tests (not to list them)
- I want to execute set of tests based on a tag (e.g. Tier1)
Those two sound like stuff that should be solved in the tooling, whatever is used to run tests.
- I want an easy way to clone tests (fedpkg clone tests/pkg)
Tests alone make no sense, you need something to test, and cloning one repo is easier than cloning two repos, so there's no advantage to the split here.
I believe the tests namespace resolves them all.
None of those arguments convince me that separate repos for tests are a good default. Sure, there are specific packages where this will make sense, or specific packagers with idiosyncratic workflows, but I'd rather "start small", with the simple configuration, and only move specific packages to the more complicated setup once that proves to be required.
Zbyszek
To sum up what I've heard so far from the developer side:
- I would like to enable tests for my component (yes, I want)
- I will take care of them (really, I see the benefit in CI)
- I want to easily collaborate on tests with qe (direct commits)
- I don't want to give qe commit access to the rpms dist git
This is quite likely the crux of the problem.
Personally, I'm perfectly happy to give write access to any repo to people who have shown that they know what they are doing. We have pull requests in dist-git pagure now, and I think this is a better approach:
- ask QE contributors to submit PRs
- if enough cooperation happens and trust is established, give
privileges to write to the repo directly, possibly with an agreement that this specific person should only touch tests, and not the packaging.
I think it's perfectly fine to never get to point 2.: many many upstream projects require a review from a second person, or sometimes even two reviews before a PR is merged, which means that one _never_ merges their own PR, and another contributor is always involved. We usually don't do this in dist-git, but I'm quite sure that requiring reviews for dist-git changes would raise quality and catch many silly mistakes. Either way, it's nowadays possible to cooperate using a single repo without fully trusting the other person, frictionlessly.
Good point. However, if there is a qe/devel team which prefers to collaborate on tests in a separate repo because this is the most efficient way they found so far, why should we stop them and try to enforce a less efficient workflow? (Which they can workaround by a different git repo.)
- I want to efficiently maintain tests long-term
- I want to have just a single place for my tests (no duplication)
- I don't want to backport new tests to old branches (enable once)
- I want to easily enable testing for all supported branches
Those four items depend strongly on the package. My thinking is biased by some specific usecases (mainly systemd), but I'm sure many other packages are like that: a lot of tests would be for new functionality, and then the tests _are_ tied to the branch being tested.
While I agree that keeping tests separate avoids a bit of effort required to update multiple branches, this effort isn't very big. If the tests indeed apply cleanly to all branches, then it's just a matter of doing 'git merge' or 'git cherry-pick' once per branch.
- I want to keep rpms dist git clean (just metadata & patches)
Meh.
- I want to run all available relevant tests (not to list them)
- I want to execute set of tests based on a tag (e.g. Tier1)
Those two sound like stuff that should be solved in the tooling, whatever is used to run tests.
- I want an easy way to clone tests (fedpkg clone tests/pkg)
Tests alone make no sense, you need something to test, and cloning one repo is easier than cloning two repos, so there's no advantage to the split here.
But "fedpkg clone tests" is easier than cloning from a "random" git repository where I was forced to save my tests because I was not allowed to save them in Fedora tests namespace.
I believe the tests namespace resolves them all.
None of those arguments convince me that separate repos for tests are a good default. Sure, there are specific packages where this will make sense, or specific packagers with idiosyncratic workflows, but I'd rather "start small", with the simple configuration, and only move specific packages to the more complicated setup once that proves to be required.
Why default? Test namespace should be an option. Not the default. Storing tests directly in dist git is and will be possible. Anybody who finds this as a better way can do so. But why enforcing this approach to all?
psss...
On 03/01/2018 04:00 PM, Petr Šplíchal wrote:
To sum up what I've heard so far from the developer side:
- I would like to enable tests for my component (yes, I want)
- I will take care of them (really, I see the benefit in CI)
- I want to easily collaborate on tests with qe (direct commits)
- I don't want to give qe commit access to the rpms dist git
This is quite likely the crux of the problem.
Personally, I'm perfectly happy to give write access to any repo to people who have shown that they know what they are doing. We have pull requests in dist-git pagure now, and I think this is a better approach:
- ask QE contributors to submit PRs
- if enough cooperation happens and trust is established, give privileges to write to the repo directly, possibly with an agreement that this specific person should only touch tests, and not the packaging.
I think it's perfectly fine to never get to point 2.: many many upstream projects require a review from a second person, or sometimes even two reviews before a PR is merged, which means that one _never_ merges their own PR, and another contributor is always involved. We usually don't do this in dist-git, but I'm quite sure that requiring reviews for dist-git changes would raise quality and catch many silly mistakes. Either way, it's nowadays possible to cooperate using a single repo without fully trusting the other person, frictionlessly.
Good point. However, if there is a qe/devel team which prefers to collaborate on tests in a separate repo because this is the most efficient way they found so far, why should we stop them and try to enforce a less efficient workflow? (Which they can workaround by a different git repo.)
- I want to efficiently maintain tests long-term
- I want to have just a single place for my tests (no duplication)
- I don't want to backport new tests to old branches (enable once)
- I want to easily enable testing for all supported branches
Those four items depend strongly on the package. My thinking is biased by some specific usecases (mainly systemd), but I'm sure many other packages are like that: a lot of tests would be for new functionality, and then the tests _are_ tied to the branch being tested.
While I agree that keeping tests separate avoids a bit of effort required to update multiple branches, this effort isn't very big. If the tests indeed apply cleanly to all branches, then it's just a matter of doing 'git merge' or 'git cherry-pick' once per branch.
- I want to keep rpms dist git clean (just metadata & patches)
Meh.
- I want to run all available relevant tests (not to list them)
- I want to execute set of tests based on a tag (e.g. Tier1)
Those two sound like stuff that should be solved in the tooling, whatever is used to run tests.
- I want an easy way to clone tests (fedpkg clone tests/pkg)
Tests alone make no sense, you need something to test, and cloning one repo is easier than cloning two repos, so there's no advantage to the split here.
But "fedpkg clone tests" is easier than cloning from a "random" git repository where I was forced to save my tests because I was not allowed to save them in Fedora tests namespace.
I believe the tests namespace resolves them all.
None of those arguments convince me that separate repos for tests are a good default. Sure, there are specific packages where this will make sense, or specific packagers with idiosyncratic workflows, but I'd rather "start small", with the simple configuration, and only move specific packages to the more complicated setup once that proves to be required.
Why default? Test namespace should be an option. Not the default. Storing tests directly in dist git is and will be possible. Anybody who finds this as a better way can do so. But why enforcing this approach to all?
+1 Exactly! I don't really see any reason why to force people to follow one way when there are obvious disadvantages. All concerns against tests/ namespace in this thread are based on expectations that it won't work, but how can anybody know at this point? The only proof we have is that the tests/ namespace works internally.
I believe that what we really need here is flexibility -- let people find the way that works best for them.
Honza
Hi!
We've had some more discussions regarding this topic within the CI team. Here's a brief summary of the recommendation we agreed upon:
Tests may be written in different ways, but are enabled in a standard way directly in the package git repository as defined by the Standard Test Interface. [1]
Test code can be stored directly in the package git repository (recommended as default) or fetched from another location hosted in the Fedora infrastructure such as the Tests Namespace. [2] The decision where the test code is stored is up to the package maintainer.
[1] https://fedoraproject.org/wiki/CI/Standard_Test_Interface [2] https://fedoraproject.org/wiki/CI/Share_Test_Code
psss...
On 1 March 2018 at 18:06, Honza Horak hhorak@redhat.com wrote:
On 03/01/2018 04:00 PM, Petr Šplíchal wrote:
To sum up what I've heard so far from the developer side:
- I would like to enable tests for my component (yes, I want)
- I will take care of them (really, I see the benefit in CI)
- I want to easily collaborate on tests with qe (direct commits)
- I don't want to give qe commit access to the rpms dist git
This is quite likely the crux of the problem.
Personally, I'm perfectly happy to give write access to any repo to people who have shown that they know what they are doing. We have pull requests in dist-git pagure now, and I think this is a better approach:
- ask QE contributors to submit PRs
- if enough cooperation happens and trust is established, give privileges to write to the repo directly, possibly with an agreement that this specific person should only touch tests, and not the packaging.
I think it's perfectly fine to never get to point 2.: many many upstream projects require a review from a second person, or sometimes even two reviews before a PR is merged, which means that one _never_ merges their own PR, and another contributor is always involved. We usually don't do this in dist-git, but I'm quite sure that requiring reviews for dist-git changes would raise quality and catch many silly mistakes. Either way, it's nowadays possible to cooperate using a single repo without fully trusting the other person, frictionlessly.
Good point. However, if there is a qe/devel team which prefers to collaborate on tests in a separate repo because this is the most efficient way they found so far, why should we stop them and try to enforce a less efficient workflow? (Which they can workaround by a different git repo.)
- I want to efficiently maintain tests long-term
- I want to have just a single place for my tests (no duplication)
- I don't want to backport new tests to old branches (enable once)
- I want to easily enable testing for all supported branches
Those four items depend strongly on the package. My thinking is biased by some specific usecases (mainly systemd), but I'm sure many other packages are like that: a lot of tests would be for new functionality, and then the tests _are_ tied to the branch being tested.
While I agree that keeping tests separate avoids a bit of effort required to update multiple branches, this effort isn't very big. If the tests indeed apply cleanly to all branches, then it's just a matter of doing 'git merge' or 'git cherry-pick' once per branch.
- I want to keep rpms dist git clean (just metadata & patches)
Meh.
- I want to run all available relevant tests (not to list them)
- I want to execute set of tests based on a tag (e.g. Tier1)
Those two sound like stuff that should be solved in the tooling, whatever is used to run tests.
- I want an easy way to clone tests (fedpkg clone tests/pkg)
Tests alone make no sense, you need something to test, and cloning one repo is easier than cloning two repos, so there's no advantage to the split here.
But "fedpkg clone tests" is easier than cloning from a "random" git repository where I was forced to save my tests because I was not allowed to save them in Fedora tests namespace.
I believe the tests namespace resolves them all.
None of those arguments convince me that separate repos for tests are a good default. Sure, there are specific packages where this will make sense, or specific packagers with idiosyncratic workflows, but I'd rather "start small", with the simple configuration, and only move specific packages to the more complicated setup once that proves to be required.
Why default? Test namespace should be an option. Not the default. Storing tests directly in dist git is and will be possible. Anybody who finds this as a better way can do so. But why enforcing this approach to all?
+1 Exactly! I don't really see any reason why to force people to follow one way when there are obvious disadvantages. All concerns against tests/ namespace in this thread are based on expectations that it won't work, but how can anybody know at this point? The only proof we have is that the tests/ namespace works internally.
I believe that what we really need here is flexibility -- let people find the way that works best for them.
Honza
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On Thursday, February 22, 2018 3:50:38 PM CET Pierre-Yves Chibon wrote:
On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
When, for a particular package, it is the most efficient way to maintain tests in a separate repo why should we discourage from this approach? There are packages where it makes more sense to store test code directly in dist-git. And it is still an option. But why should we enforce this for all?
My worries are basically that this mechanism is hiding the tests from the package maintainers. It splits the concerns between people maintaining the artifacts and people maintaining the tests, which is exactly what the original plan to bring CI was *not* about.
Since we'll have to "hook" the remote repo into dist-git code, I must be missing something. Where do you see the hidden magic?
The idea has always been to bring the tests where the code lives, so that both can be worked on as one, to make tests be a concern of the package maintainer very much like updating the artifact (rpms, image..) is, while getting support from QE for them (the tests).
If the tests live in separate repo, they still can "work as one".
In addition, this is what I fear most: The tests for the package are stored elsewhere, we're not sure where (the tests namespace, a random git repo on the internet...), the packager update package and the tests fail. What do you think will happen? a) the packager will look for $random_place_of_the_internet where tests are and spend time trying to fix them.
Yes, please don't force us to use random_place_of_the_internet, but rather allow maintainers to use standardized place.
Then e.g. 'rpmtests/<PKGNAME>' repo can automatically grant commit access to package maintainers of 'rpms/<PKGNAME>' to not overcomplicate things, but still, we can grant non-packagers access to 'rpmtests/<PKGNAME>'.
What about proventesters, could we grant them access to all tests, or could we create new role?
b) the packager will turn off the tests because they get in the way
Polemics (even below) about turning-off the tests is FUD. Since tests are opt-in feature, why would anyone thought this way?
If the packager wants to turn off the tests, they will have to go through dist-git to do it, they may very well end up turning the tests off anyway, but if the tests are right there, they may as well have a quick look at them to see if they can fix them quickly before deciding.
In addition, if the packager turn the tests off and the people maintaining the tests do not realize that, they will end up spending time maintaining $somewhere_else tests that aren't being used. Having them interact directly with the dist-git repo will increase the chances that they see/realize when something is turned off.
If that means we have less tests in dist-git because the maintainers do not want them, I'd say so be it. In the long term this is their loss, they are the ones who will get the bug reports and will have to deal with them, they are the ones who will have to dive into old code rather than going back into something that is still fresh in their mind.
There's no loss, no. Everything is as transparent as with in-dist-git testing, except it gives us more flexibility - to allow more people to contribute. - keep the package's git smaller and concentrated on packaging
As long as, it is clear that they have been approached and that it is their choice to not merge pull-requests adding tests, I think the people offering to help should not be the ones blamed.
The help here is very much welcome, I wouldn't blame anyone though (I for one only wait for a bit more flexible form of CI).
Btw., the far more important problem is that we can not setup CI for non-atomic host packages, and that we can not use the CI for Rawhide gating.
Pavel
On Fri, Feb 23, 2018 at 10:33:57AM +0100, Pavel Raiskup wrote:
On Thursday, February 22, 2018 3:50:38 PM CET Pierre-Yves Chibon wrote:
On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
When, for a particular package, it is the most efficient way to maintain tests in a separate repo why should we discourage from this approach? There are packages where it makes more sense to store test code directly in dist-git. And it is still an option. But why should we enforce this for all?
My worries are basically that this mechanism is hiding the tests from the package maintainers. It splits the concerns between people maintaining the artifacts and people maintaining the tests, which is exactly what the original plan to bring CI was *not* about.
Since we'll have to "hook" the remote repo into dist-git code, I must be missing something. Where do you see the hidden magic?
The idea has always been to bring the tests where the code lives, so that both can be worked on as one, to make tests be a concern of the package maintainer very much like updating the artifact (rpms, image..) is, while getting support from QE for them (the tests).
If the tests live in separate repo, they still can "work as one".
How do they work as one if they are both working in different places?
In addition, this is what I fear most: The tests for the package are stored elsewhere, we're not sure where (the tests namespace, a random git repo on the internet...), the packager update package and the tests fail. What do you think will happen? a) the packager will look for $random_place_of_the_internet where tests are and spend time trying to fix them.
Yes, please don't force us to use random_place_of_the_internet, but rather allow maintainers to use standardized place.
Then e.g. 'rpmtests/<PKGNAME>' repo can automatically grant commit access to package maintainers of 'rpms/<PKGNAME>' to not overcomplicate things, but still, we can grant non-packagers access to 'rpmtests/<PKGNAME>'.
We're working on figuring a way to give non-packagers commit to dist-git, so giving more access would work on the main repo as well.
What about proventesters, could we grant them access to all tests, or could we create new role?
b) the packager will turn off the tests because they get in the way
Polemics (even below) about turning-off the tests is FUD. Since tests are opt-in feature, why would anyone thought this way?
I may very well be miss-seeing this, but what I see is people refusing to add tests to their repository because it doesn't suit their workflow. So I am thinking that if they finally think "whatever let's just merge" because they don't want to cause troubles, whenever the tests are going to impact that workflow even more because they fail, chances are high that said person will just turn them off. I'd be happy to be wrong about that though.
If the packager wants to turn off the tests, they will have to go through dist-git to do it, they may very well end up turning the tests off anyway, but if the tests are right there, they may as well have a quick look at them to see if they can fix them quickly before deciding.
In addition, if the packager turn the tests off and the people maintaining the tests do not realize that, they will end up spending time maintaining $somewhere_else tests that aren't being used. Having them interact directly with the dist-git repo will increase the chances that they see/realize when something is turned off.
If that means we have less tests in dist-git because the maintainers do not want them, I'd say so be it. In the long term this is their loss, they are the ones who will get the bug reports and will have to deal with them, they are the ones who will have to dive into old code rather than going back into something that is still fresh in their mind.
There's no loss, no. Everything is as transparent as with in-dist-git testing, except it gives us more flexibility
- to allow more people to contribute.
Pull-requests and improvement to our dist-git set up seems to me better ways to achieve this.
- keep the package's git smaller and concentrated on packaging
That one is fair.
As long as, it is clear that they have been approached and that it is their choice to not merge pull-requests adding tests, I think the people offering to help should not be the ones blamed.
The help here is very much welcome, I wouldn't blame anyone though (I for one only wait for a bit more flexible form of CI).
Btw., the far more important problem is that we can not setup CI for non-atomic host packages, and that we can not use the CI for Rawhide
Definitely agreed on this one, I've been pushing for it for the better part of a year, I'd be happy to see more people pushing for it.
Pierre
On Thu, Feb 22, 2018 at 03:50:38PM +0100, Pierre-Yves Chibon wrote:
On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
Hi!
During the last days there have been concerns raised regarding what is an appropriate content for the tests namespace. [1] My original idea was to enable sharing tests even across branches of the same component, not only for tests to be used by completely different packages. The initial examples might have been a bit misleading in this respect. One of the main points still holds:
- Tests might follow a different branching pattern than the dist-git repo, leading to code duplication
From the feedback from developers I feel they always keep on mind and care a lot about the maintenance costs. So it perfectly makes sense to me if they want to keep and maintain tests in a separate repo instead of merging/cherry-picking between dist-git branches.
When, for a particular package, it is the most efficient way to maintain tests in a separate repo why should we discourage from this approach? There are packages where it makes more sense to store test code directly in dist-git. And it is still an option. But why should we enforce this for all?
My worries are basically that this mechanism is hiding the tests from the package maintainers. It splits the concerns between people maintaining the artifacts and people maintaining the tests, which is exactly what the original plan to bring CI was *not* about. The idea has always been to bring the tests where the code lives, so that both can be worked on as one, to make tests be a concern of the package maintainer very much like updating the artifact (rpms, image..) is, while getting support from QE for them (the tests).
In addition, this is what I fear most: The tests for the package are stored elsewhere, we're not sure where (the tests namespace, a random git repo on the internet...), the packager update package and the tests fail. What do you think will happen? a) the packager will look for $random_place_of_the_internet where tests are and spend time trying to fix them. b) the packager will turn off the tests because they get in the way
If the packager wants to turn off the tests, they will have to go through dist-git to do it, they may very well end up turning the tests off anyway, but if the tests are right there, they may as well have a quick look at them to see if they can fix them quickly before deciding.
In addition, if the packager turn the tests off and the people maintaining the tests do not realize that, they will end up spending time maintaining $somewhere_else tests that aren't being used. Having them interact directly with the dist-git repo will increase the chances that they see/realize when something is turned off.
If that means we have less tests in dist-git because the maintainers do not want them, I'd say so be it. In the long term this is their loss, they are the ones who will get the bug reports and will have to deal with them, they are the ones who will have to dive into old code rather than going back into something that is still fresh in their mind. As long as, it is clear that they have been approached and that it is their choice to not merge pull-requests adding tests, I think the people offering to help should not be the ones blamed.
Hi,
pingou, I share your thinking in general, but I think your concern is overstated. Petr's original e-mail suggests that the new separate namespace should be used *in* *preference* to the in-dist-git, and your reply concentrates on that. But I think that we want to have both possibilities, because both will work best for different cases, and it's just the emphasis that needs to change.
I agree that having tests in a separate namespace is not any simpler or more manageable for most packages. Thus, I think that *by* *default* we should put tests in dist-git. IMHO this will apply to 90% of all tests and packages.
OTOH, there will be cases where it'll be useful to split the tests out into a separate area. The documentation for tests/ has the example of shared tests for bash/zhs/dash. I'm sure that there will be other cases, like testing of stacks consisting of multiple packages apache+mysql+php or glibc+gcc+binutils, etc, where it'll be beneficial to keep the test separate. The management of permissions in this case will be _much_ more complicated, because to meaningfully update the tests sometimes it'll be necessary to change the test repo and _each_ of the packages. But having them separate is better than randomly sticking the shared tests in dist-git for one of the involved packages. Nevertheless, because of the overhead of more repos and additional permissions, IMHO this should never be advertised as the default approach, but as an "advanced" possibility.
Zbyszek
On Sun, Feb 25, 2018 at 12:12:33PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Thu, Feb 22, 2018 at 03:50:38PM +0100, Pierre-Yves Chibon wrote:
On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
Hi!
During the last days there have been concerns raised regarding what is an appropriate content for the tests namespace. [1] My original idea was to enable sharing tests even across branches of the same component, not only for tests to be used by completely different packages. The initial examples might have been a bit misleading in this respect. One of the main points still holds:
- Tests might follow a different branching pattern than the dist-git repo, leading to code duplication
From the feedback from developers I feel they always keep on mind and care a lot about the maintenance costs. So it perfectly makes sense to me if they want to keep and maintain tests in a separate repo instead of merging/cherry-picking between dist-git branches.
When, for a particular package, it is the most efficient way to maintain tests in a separate repo why should we discourage from this approach? There are packages where it makes more sense to store test code directly in dist-git. And it is still an option. But why should we enforce this for all?
My worries are basically that this mechanism is hiding the tests from the package maintainers. It splits the concerns between people maintaining the artifacts and people maintaining the tests, which is exactly what the original plan to bring CI was *not* about. The idea has always been to bring the tests where the code lives, so that both can be worked on as one, to make tests be a concern of the package maintainer very much like updating the artifact (rpms, image..) is, while getting support from QE for them (the tests).
In addition, this is what I fear most: The tests for the package are stored elsewhere, we're not sure where (the tests namespace, a random git repo on the internet...), the packager update package and the tests fail. What do you think will happen? a) the packager will look for $random_place_of_the_internet where tests are and spend time trying to fix them. b) the packager will turn off the tests because they get in the way
If the packager wants to turn off the tests, they will have to go through dist-git to do it, they may very well end up turning the tests off anyway, but if the tests are right there, they may as well have a quick look at them to see if they can fix them quickly before deciding.
In addition, if the packager turn the tests off and the people maintaining the tests do not realize that, they will end up spending time maintaining $somewhere_else tests that aren't being used. Having them interact directly with the dist-git repo will increase the chances that they see/realize when something is turned off.
If that means we have less tests in dist-git because the maintainers do not want them, I'd say so be it. In the long term this is their loss, they are the ones who will get the bug reports and will have to deal with them, they are the ones who will have to dive into old code rather than going back into something that is still fresh in their mind. As long as, it is clear that they have been approached and that it is their choice to not merge pull-requests adding tests, I think the people offering to help should not be the ones blamed.
pingou, I share your thinking in general, but I think your concern is overstated. Petr's original e-mail suggests that the new separate namespace should be used *in* *preference* to the in-dist-git, and your reply concentrates on that. But I think that we want to have both possibilities, because both will work best for different cases, and it's just the emphasis that needs to change.
I agree that having tests in a separate namespace is not any simpler or more manageable for most packages. Thus, I think that *by* *default* we should put tests in dist-git. IMHO this will apply to 90% of all tests and packages.
OTOH, there will be cases where it'll be useful to split the tests out into a separate area. The documentation for tests/ has the example of shared tests for bash/zhs/dash. I'm sure that there will be other cases, like testing of stacks consisting of multiple packages apache+mysql+php or glibc+gcc+binutils, etc, where it'll be beneficial to keep the test separate. The management of permissions in this case will be _much_ more complicated, because to meaningfully update the tests sometimes it'll be necessary to change the test repo and _each_ of the packages. But having them separate is better than randomly sticking the shared tests in dist-git for one of the involved packages. Nevertheless, because of the overhead of more repos and additional permissions, IMHO this should never be advertised as the default approach, but as an "advanced" possibility.
I violently agree with this :) I am very much advocating for using the test namespace to share tests across multiple packages. What trigger this discussion has been the request to create repositories in the test namespace to store the tests of a single package (ie: instead of just adding them to the corresponding dist-git repo).
Pierre
On 02/25/2018 08:59 PM, Pierre-Yves Chibon wrote:
On Sun, Feb 25, 2018 at 12:12:33PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Thu, Feb 22, 2018 at 03:50:38PM +0100, Pierre-Yves Chibon wrote:
On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
Hi!
During the last days there have been concerns raised regarding what is an appropriate content for the tests namespace. [1] My original idea was to enable sharing tests even across branches of the same component, not only for tests to be used by completely different packages. The initial examples might have been a bit misleading in this respect. One of the main points still holds:
- Tests might follow a different branching pattern than the dist-git repo, leading to code duplication
From the feedback from developers I feel they always keep on mind and care a lot about the maintenance costs. So it perfectly makes sense to me if they want to keep and maintain tests in a separate repo instead of merging/cherry-picking between dist-git branches.
When, for a particular package, it is the most efficient way to maintain tests in a separate repo why should we discourage from this approach? There are packages where it makes more sense to store test code directly in dist-git. And it is still an option. But why should we enforce this for all?
My worries are basically that this mechanism is hiding the tests from the package maintainers. It splits the concerns between people maintaining the artifacts and people maintaining the tests, which is exactly what the original plan to bring CI was *not* about. The idea has always been to bring the tests where the code lives, so that both can be worked on as one, to make tests be a concern of the package maintainer very much like updating the artifact (rpms, image..) is, while getting support from QE for them (the tests).
In addition, this is what I fear most: The tests for the package are stored elsewhere, we're not sure where (the tests namespace, a random git repo on the internet...), the packager update package and the tests fail. What do you think will happen? a) the packager will look for $random_place_of_the_internet where tests are and spend time trying to fix them. b) the packager will turn off the tests because they get in the way
If the packager wants to turn off the tests, they will have to go through dist-git to do it, they may very well end up turning the tests off anyway, but if the tests are right there, they may as well have a quick look at them to see if they can fix them quickly before deciding.
In addition, if the packager turn the tests off and the people maintaining the tests do not realize that, they will end up spending time maintaining $somewhere_else tests that aren't being used. Having them interact directly with the dist-git repo will increase the chances that they see/realize when something is turned off.
If that means we have less tests in dist-git because the maintainers do not want them, I'd say so be it. In the long term this is their loss, they are the ones who will get the bug reports and will have to deal with them, they are the ones who will have to dive into old code rather than going back into something that is still fresh in their mind. As long as, it is clear that they have been approached and that it is their choice to not merge pull-requests adding tests, I think the people offering to help should not be the ones blamed.
pingou, I share your thinking in general, but I think your concern is overstated. Petr's original e-mail suggests that the new separate namespace should be used *in* *preference* to the in-dist-git, and your reply concentrates on that. But I think that we want to have both possibilities, because both will work best for different cases, and it's just the emphasis that needs to change.
I agree that having tests in a separate namespace is not any simpler or more manageable for most packages. Thus, I think that *by* *default* we should put tests in dist-git. IMHO this will apply to 90% of all tests and packages.
OTOH, there will be cases where it'll be useful to split the tests out into a separate area. The documentation for tests/ has the example of shared tests for bash/zhs/dash. I'm sure that there will be other cases, like testing of stacks consisting of multiple packages apache+mysql+php or glibc+gcc+binutils, etc, where it'll be beneficial to keep the test separate. The management of permissions in this case will be _much_ more complicated, because to meaningfully update the tests sometimes it'll be necessary to change the test repo and _each_ of the packages. But having them separate is better than randomly sticking the shared tests in dist-git for one of the involved packages. Nevertheless, because of the overhead of more repos and additional permissions, IMHO this should never be advertised as the default approach, but as an "advanced" possibility.
I violently agree with this :) I am very much advocating for using the test namespace to share tests across multiple packages. What trigger this discussion has been the request to create repositories in the test namespace to store the tests of a single package (ie: instead of just adding them to the corresponding dist-git repo).
Even a single package is then maintained in multiple Fedora branches (and don't forget about modules in the picture) and while RPM spec files and sources might be easily different release to release, the tests have much bigger chance to be same across the releases, so having one copy of them is what I expect in most cases. That's what I like about the shared place variant.
Honza
-
Adam Williamson -
Honza Horak -
Pavel Raiskup -
Petr Splichal -
Petr Šplíchal
-
Pierre-Yves Chibon -
Zbigniew Jędrzejewski-Szmek