Hey all- I was starting to setup CI for one of my packages in Fedora (cscope), which requires that I have access to the sources to run my test (cscope uses its own source tree to search for various symbols to confirm that its working properly). Getting the sources in the CI environment is a bit of a pain, so I started working on trying to do this by creating a test subpackage (specifically named -citest) to package up the sources solely for the purpose of getting them installed and available during CI runs. It occured to me that this offers several advantages, among them: 1) the ability to codify dependencies within the ame spec file, rather than having to copy them to the test.yml file, and keep them in sync
2) The ability to use a file format (rpm spec files) that I'm more familiar with
3) Easy access to tests that are embedded in the source tree
4) minimizing the test harness setup in test.yml
For anyone interested, I've got a pull request started here: https://src.fedoraproject.org/rpms/cscope/pull-request/2
If anyone wants to take a look at the changes I had to make to do this (fair warning, its still very rough).
That all said, I was wondering if perhaps there was general interest in making this kind of test model somewhat more formal (i.e. creating an rpm macro library to make test package generation a bit easier, creating a standard entry point to run tests, etc).
Thoughts welcome
I am skeptical about this proposal. While this might work for your package, I am afraid it won't work generally and trying to do something like this is wasted energy. Let me explain.
When RubyGems were designed in 2004, TDD, BDD and testing in general was becoming good practice. Therefore they decided that it is good idea to ship code together with its test suite and also execute the tests during installation. So you were supposed to be able to run something like "gem install -t foo". But it proven to be problematic. Generally this "-t" option never worked and could be used just for the most naive test suites. So the option was removed [1]. Nowadays, more and more gem packages does not ship their tests suites and even the support for shipping the test suites is deprecated [2].
If you want to understand how complex it might be to run the test suite of some packages, I welcome you to explore the %check sections of rubygem- packages. Some are simple, but most are not that simple, starting with small differences as load path, ending with test suites which needs to run various servers or check against cloud services. Not mentioning test matrices.
Also, for all these tests we usually try to simulate the environment as similar as possible to upstream repository, because this is how these test suites are designed. It would be even harder to execute the test suites against installed packages.
Vít
[1] https://github.com/rubygems/rubygems/commit/429f883210f8b2b38ea310f7fc6636cd...
[2] https://github.com/rubygems/rubygems/issues/735
Dne 05. 07. 19 v 19:49 Neil Horman napsal(a):
Hey all- I was starting to setup CI for one of my packages in Fedora (cscope), which requires that I have access to the sources to run my test (cscope uses its own source tree to search for various symbols to confirm that its working properly). Getting the sources in the CI environment is a bit of a pain, so I started working on trying to do this by creating a test subpackage (specifically named -citest) to package up the sources solely for the purpose of getting them installed and available during CI runs. It occured to me that this offers several advantages, among them:
- the ability to codify dependencies within the ame spec file, rather than
having to copy them to the test.yml file, and keep them in sync
The ability to use a file format (rpm spec files) that I'm more familiar with
Easy access to tests that are embedded in the source tree
minimizing the test harness setup in test.yml
For anyone interested, I've got a pull request started here: https://src.fedoraproject.org/rpms/cscope/pull-request/2
If anyone wants to take a look at the changes I had to make to do this (fair warning, its still very rough).
That all said, I was wondering if perhaps there was general interest in making this kind of test model somewhat more formal (i.e. creating an rpm macro library to make test package generation a bit easier, creating a standard entry point to run tests, etc).
Thoughts welcome _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Mon, Jul 08, 2019 at 02:17:44PM +0200, Vít Ondruch wrote:
I am skeptical about this proposal. While this might work for your package, I am afraid it won't work generally and trying to do something like this is wasted energy. Let me explain.
well, that may be fair, but when you way it "won't work generally", I could paraphrase that to say "it may not be sufficient for all packages". I would make the argument that the implication is that, for some subset of packages, this approach works quite well. That is in fact, the case for all the packages I maintain.
When RubyGems were designed in 2004, TDD, BDD and testing in general was becoming good practice. Therefore they decided that it is good idea to ship code together with its test suite and also execute the tests during installation. So you were supposed to be able to run something like "gem install -t foo". But it proven to be problematic. Generally this "-t" option never worked and could be used just for the most naive test suites. So the option was removed [1]. Nowadays, more and more gem packages does not ship their tests suites and even the support for shipping the test suites is deprecated [2].
Ok, that seems like a fair conclusion to draw from your experience, but it is by no means the only experience. I currently maintain: cscope irqbalance rng-tools dropwatch rstp
and a few others, all of which have very mature test suites embedded into their soruce code. For these pacakages, being able to run their test code in the CI environment would be very helpful to me (and I think, by extension), others who have simmilarly constructed packages.
If you want to understand how complex it might be to run the test suite of some packages, I welcome you to explore the %check sections of rubygem- packages. Some are simple, but most are not that simple, starting with small differences as load path, ending with test suites which needs to run various servers or check against cloud services. Not mentioning test matrices.
I do run make check and part of an rpm %check script on most of these, and if the consensus is that that approach is sufficient for CI purposes, so be it (that would in fact be much easier for me). But its my understanding that there is a desire to move all our testing to a CI pipeline (or perhaps it would be beter to say that we wish to add CI to all our packages), and running the included self tests with an upstream source tree within the CI pipeline is dificult at best (owing to the lack of source availability in the CI environment), nor is running a %check script during a build an available trigger to clear gating during CI, hence my desire to find a way to make my included selftests from the source tree available in CI via -test rpm subpackages.
Also, for all these tests we usually try to simulate the environment as similar as possible to upstream repository, because this is how these test suites are designed. It would be even harder to execute the test suites against installed packages.
Unless you were able to extract the test suite into a separate package, install it in the environment, and run it, while pointing to the installed binaries that you are trying to test :)
Neil
Vít
[1] https://github.com/rubygems/rubygems/commit/429f883210f8b2b38ea310f7fc6636cd...
[2] https://github.com/rubygems/rubygems/issues/735
Dne 05. 07. 19 v 19:49 Neil Horman napsal(a):
Hey all- I was starting to setup CI for one of my packages in Fedora (cscope), which requires that I have access to the sources to run my test (cscope uses its own source tree to search for various symbols to confirm that its working properly). Getting the sources in the CI environment is a bit of a pain, so I started working on trying to do this by creating a test subpackage (specifically named -citest) to package up the sources solely for the purpose of getting them installed and available during CI runs. It occured to me that this offers several advantages, among them:
- the ability to codify dependencies within the ame spec file, rather than
having to copy them to the test.yml file, and keep them in sync
The ability to use a file format (rpm spec files) that I'm more familiar with
Easy access to tests that are embedded in the source tree
minimizing the test harness setup in test.yml
For anyone interested, I've got a pull request started here: https://src.fedoraproject.org/rpms/cscope/pull-request/2
If anyone wants to take a look at the changes I had to make to do this (fair warning, its still very rough).
That all said, I was wondering if perhaps there was general interest in making this kind of test model somewhat more formal (i.e. creating an rpm macro library to make test package generation a bit easier, creating a standard entry point to run tests, etc).
Thoughts welcome _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Hi Neil,
Neil Horman nhorman@redhat.com writes:
Hey all- I was starting to setup CI for one of my packages in Fedora (cscope), which requires that I have access to the sources to run my test (cscope uses its own source tree to search for various symbols to confirm that its working properly). Getting the sources in the CI environment is a bit of a pain, so I started working on trying to do this by creating a test subpackage (specifically named -citest) to package up the sources solely for the purpose of getting them installed and available during CI runs. It occured to me that this offers several advantages, among them:
- the ability to codify dependencies within the ame spec file, rather than
having to copy them to the test.yml file, and keep them in sync
The ability to use a file format (rpm spec files) that I'm more familiar with
Easy access to tests that are embedded in the source tree
This is imho a pretty big advantage, as it ensures that the tests and the source don't diverge.
- minimizing the test harness setup in test.yml
For anyone interested, I've got a pull request started here: https://src.fedoraproject.org/rpms/cscope/pull-request/2
If anyone wants to take a look at the changes I had to make to do this (fair warning, its still very rough).
That all said, I was wondering if perhaps there was general interest in making this kind of test model somewhat more formal (i.e. creating an rpm macro library to make test package generation a bit easier, creating a standard entry point to run tests, etc).
I am not sure whether a generalization makes sense, as there are so many languages with such a wide range of test suites. What would make sense to standardize would be the generation of a -citest subpackage though, so that it is setup correctly and consistently.
Thoughts welcome
I like this idea and you're actually not the first one ;-). Something comparable is being done in openSUSE's Ruby RPM packages: if the gem ships a testsuite, then a -test subpackage is created with the tests inside it. (In practice these packages are unfortunately never used, as they often lack the necessary dependencies to be installable and even if, the testsuite usually doesn't run outside of bundler, but that's a different story).
I think this approach makes especially sense for packages which ship an extensive test suite that is not feasible to run during %check, but can be run in the gating CI.