Hello,
The purpose of this long email is to justify the need for a /usr/tests directory and to ask for approval for its creation, with the final goal of packaging ATF, its tests and the tests provided by arbitrary applications (e.g. the just-approved lutok package[1]). This email is long because I feel the need to provide tons of background to explain why this directory is required. I hope the text is clear and that it makes enough sense to convince you of its approval. Here we go!
I am the author of the Automated Testing Framework[2] (ATF for short). ATF is the testing framework currently used in the NetBSD operating system, although the ATF project itself is OS-agnostic. In other words, ATF builds and runs fine on multiple Linux distributions, BSD systems and OS X. (ATF is being replaced by a new project called Kyua[3]; this will come into play later in the text.)
I am now interested in creating an RPM package of ATF (and Kyua) for Fedora and there is a "little detail" in ATF that will surely be controversial. This is why I am reaching the FPC before sending any spec file for review, as this particular point will need discussion and approval.
ATF is made up of two parts: 1) a collection of libraries (C, C++ and shell) that enable developers to write test programs for their applications and libraries; and 2) a collection of tools (atf-run, atf-report) that allow the developers and *end users* to execute these tests.
To re-emphasize the point above, note that I said "end users". One of the major features that differentiate ATF from other testing frameworks is that *the tests written using ATF are supposed to be installed in the system* as part of the regular application or library. ATF then provides standalone tools to execute these tests and to generate reports of their results, all without the need of the original program sources nor any development tools (read: no gcc, and specially no make). These tests can be run by any user as they don't require write access to the directory in which the tests live.
To put this in the context of a Linux distribution, let's consider a fictitious example: imagine that glib and gtk came with ATF-based tests. These tests would get installed into testsdir/glib/... and testsdir/gtk/... The user would then do "cd testsdir/gtk/ && atf-run" to execute the tests on his machine, without the need for any -devel packages. The key point of this idea is that this allows end users to gain confidence in their systems, because they can validate that a particular piece of software runs fine on their OS / hardware combination. What is more, suppose that glib got upgraded to a new version but not gtk. If there were any incompatibilities between the two, the installed gtk tests would highlight it because they'd start failing. The packager may never have noticed that fact himself if he always kept his build environment up-to-date (as is commonly the case). (Also, consider creating a cron job to run all these tests nightly!)
Well, imagine no more. We already do something like this in NetBSD. NetBSD defines the testsdir above as "/usr/tests". This tree[4] contains a subdirectory for each package that provides ATF-based tests, and there is a top-level Atffile file that allows atf-run to recurse into all the package subdirectories. (I.e. executing atf-run within /usr/tests runs *all* the tests available in the system for all packages.) You can see in [5] a more detailed description as to how this comes into practice.
There are two important things to consider about /usr/tests:
1) The first is that there is a single directory acting as the "root" for the tests of arbitrary packages because this is what allows all these tests to be run at once. In other words, instead of having /usr/pkg1/tests /usr/pkg2/tests, we have /usr/tests/pkg1 and /usr/tests/pkg2.
2) The second is that ATF is not encoded anywhere in the path. I.e. the directories are /usr/tests/pkg*, not /usr/atf/pkg* nor /usr/tests/atf/pkg*. The reason for this is that the package-specific tests needn't be implemented using the ATF libraries. In fact, Kyua, the replacement for ATF, is able to execute these same unmodified tests and can also execute tests written other frameworks. In a future world, packages would install arbitrary tests into /usr/tests written using their preferred libraries (e.g. pyunit, autotest, etc.) and a single tool (Kyua) would run them all. Encoding the tool name in the directory introduces a dependency on the tool that does not necessarily exist.
So, to sum it up: the whole point of this email is to discuss what the "testsdir" above should be in Fedora. I think it makes a lot of sense to keep the default of /usr/tests.
However, of course, this does not complain with any the LSB policies. The problem is that no other already-existing directory seems to suit the goals of /usr/tests. For example: /usr/share/tests is unsuitable because the tests are obviously executables and thus not shareable. /usr/libexec/tests also appears unsuitable because these programs are not helper binaries internally used by other programs: they are supposed to be exposed to the end user, who may choose to run them by hand. /usr/lib/tests is even worse because these are not libraries and also has the same problem of libexec in the sense that the files are hidden from end users. End users need to be encouraged to "cd /usr/tests" and to run stuff from there, just as they do with /usr/bin or /usr/sbon.
Therefore, I believe that introducing a new directory is sensible enough, particularly because it is potentially optional (e.g. could be made part of a -filesystem package). Actually, the way I envision the packaging to work is the following. Let's consider lutok again, as its distribution file contains ATF-based tests. lutok would come in these packages: lutok, lutok-devel, lutok-tests. The latter package would place files in /usr/tests/lutok only and it would depend on libatf-c++ (because lutok's tests are implemented in C++) and maybe some "tests-filesystem" package. lutok-tests needn't depend on the atf tools. Then, the user could just install the 'atf' package providing atf-run and atf-report to execute these tests from within /usr/tests or /usr/tests/lutok. Similarly, other tools could place their installable tests in /usr/tests/<pkg>.
So, what packages would make use of /usr/tests if it were introduced now? As far as I can tell, these would be atf, lutok and kyua (including all of its subpackages for the libraries, etc.). Bind 9 recently adopted ATF too, although I'm not sure if they install the tests.
This should be all for now. Is it OK to introduce this optional /usr/tests directory?
Please let me know if you have any questions or concerns.
Thank you,
Julio Merino
1: https://bugzilla.redhat.com/show_bug.cgi?id=785619 2: http://www.NetBSD.org/~jmmv/atf/ 3: http://code.google.com/p/kyua/ 4: http://cvsweb.netbsd.org/bsdweb.cgi/src/tests/?only_with_tag=MAIN 5: http://blog.netbsd.org/tnf/entry/testing_netbsd_easy_does_it
On 02/06/2012 04:59 AM, Julio Merino wrote:
Hello,
The purpose of this long email is to justify the need for a /usr/tests directory and to ask for approval for its creation, with the final goal of packaging ATF, its tests and the tests provided by arbitrary applications (e.g. the just-approved lutok package[1]). This email is long because I feel the need to provide tons of background to explain why this directory is required. I hope the text is clear and that it makes enough sense to convince you of its approval. Here we go!
[LOTS OF GOOD RATIONALE SNIPPED]
So, the first thought I had was "these binaries won't be in the $PATH". You say above that "End users need to be encouraged to 'cd /usr/tests' and to run stuff from there, just as they do with /usr/bin or /usr/sbin." ... but end users do not cd into /usr/bin to run a binary from there, they simply call the binary name and expect it to work. I'm assuming that the fact that these tests are not in the $PATH is a feature, not a bug. :)
If this is the case, then it may be possible after all to use /usr/libexec/tests for this. After all, /usr/libexec is a bit of a grey area, the FHS doesn't mention it at anywhere (despite repeated pleading to them to do so), it comes from the GNU Coding Standards and Fedora has adopted it as an exception. The Fedora Packaging Guidelines say:
Libexecdir (aka, /usr/libexec on Fedora systems) should only be used as the directory for executable programs that are designed primarily to be run by other programs rather than by users.
That said, I could definitely see the argument being made that /usr/libexec/tests is a perfectly acceptable use case for binaries that are "designed primarily to be run by testing frameworks, but also available outside of $PATH for users to manually execute".
However, whether it is /usr/tests or /usr/libexec/tests, there is going to be a need to tightly constrain the acceptable files that could go into /usr/tests and document exactly how they are allowed to go into it.
For example, my experience with test suites is almost certainly far less than yours, but there seems to also be a fair amount of non-executable test data content that comes along with the executable tests. E.g. test-xpdf reads in 00-Broken.pdf with XFAIL as the result. Would non-executable test data live in /usr/[libexec/]tests/$program/ ? Or /usr/share/tests/$program/ ?
It is also possible that introducing a new home for binaries would cause multi-lib heartburn in RPM, but I suspect that is a simple enough problem to fix, since we haven't really hit it with our use of /usr/libexec to date.
You probably should open a ticket with the FHS working group: https://bugs.linuxfoundation.org/ (product: FHS) and explain all of this to them so that it has a chance of being officially added in the next major FHS revision, whenever that may be.
Above and beyond your proposal, the only other way I can think of handling this would be an elaborate namespacing of the test binaries in /usr/bin, e.g. /usr/bin/test-$pkg-$testname , and while that would be a solution that would permit a strict compliance with the FHS as it exists today, I'm not very excited about it and am not seriously recommending it as an alternative.
Thanks for your message,
~tom
== Fedora Project
On 2/6/12 7:27 AM, Tom Callaway wrote:
On 02/06/2012 04:59 AM, Julio Merino wrote:
Hello,
[...]
Hello Tom,
Thanks for the detailed reply. A longer reply comes below ;)
So, the first thought I had was "these binaries won't be in the $PATH". You say above that "End users need to be encouraged to 'cd /usr/tests' and to run stuff from there, just as they do with /usr/bin or /usr/sbin." ... but end users do not cd into /usr/bin to run a binary from there, they simply call the binary name and expect it to work. I'm assuming that the fact that these tests are not in the $PATH is a feature, not a bug. :)
Correct. They are not supposed to be in the PATH. (They could be though, but their utility would be reduced and they'd just screw up autocompletion. Also, this might not be totally doable given that testsdir is a tree with multiple nested subdirectories; see below for a detailed example showcasing corner cases.)
It'd be argued that the tools to run the tests (atf-run in this case) should automatically look for stuff in /usr/tests unless told otherwise, thus hiding the details of where the tests are stored. I'll grant you that this is a nice idea to pursue, but that's not how things work at the moment and I'm not sure about what the ramifications of such change would be.
(The idea of having to cd into the tests directory before running the tests is that you can have tests in multiple "roots". One root is the testsdir, where the tests for all installed packages live. But other valid roots are the top directory of your source trees, so that you can execute the tests while doing development before doing an actual installation!)
If this is the case, then it may be possible after all to use /usr/libexec/tests for this. After all, /usr/libexec is a bit of a grey area, the FHS doesn't mention it at anywhere (despite repeated pleading to them to do so), it comes from the GNU Coding Standards and Fedora has adopted it as an exception. The Fedora Packaging Guidelines say:
Libexecdir (aka, /usr/libexec on Fedora systems) should only be used as the directory for executable programs that are designed primarily to be run by other programs rather than by users.
That said, I could definitely see the argument being made that /usr/libexec/tests is a perfectly acceptable use case for binaries that are "designed primarily to be run by testing frameworks, but also available outside of $PATH for users to manually execute".
I see. /usr/libexec/tests doesn't sound too bad considering this description! However, the data files issue you raised could get in the way. Some more details below.
However, whether it is /usr/tests or /usr/libexec/tests, there is going to be a need to tightly constrain the acceptable files that could go into /usr/tests and document exactly how they are allowed to go into it.
For example, my experience with test suites is almost certainly far less than yours, but there seems to also be a fair amount of non-executable test data content that comes along with the executable tests. E.g. test-xpdf reads in 00-Broken.pdf with XFAIL as the result. Would non-executable test data live in /usr/[libexec/]tests/$program/ ? Or /usr/share/tests/$program/ ?
As it turns out, I forgot to mention this exact point (data files) in my original email. The way things work at the moment within ATF is to have data files and helper binaries in the same directory as the test program that needs them. The test programs can then dynamically query what their "source directory" is so that they can locate these files regardless of where the test program lives.
The rationale for this was two-folded. First, it allows the test programs to run both from the source directory and the installed tree because they don't hardcode absolute paths to their helpers. And second, the whole tests tree lives in a self-contained directory that can either be pruned from the system (which is convenient due to the way NetBSD works) or shared on the network. Looking at existing practice, a test tree is composed mostly of binaries or scripts; the percentage of data files is very low. (There was also an idea of being able to provide such programs as "bundles" that could be copied to slave machines for remote testing, but has never been implemented yet.)
In other words, the layout for your example would be:
testsdir/ testsdir/xpdf/ testsdir/xpdf/parser_test - Binary; atf-run executes it from within a temporary directory to "isolate" it from the read-only testsdir and other parts of the system. parser_test can query its "srcdir", which in this case is "testsdir/xpdf/", and thus locate where the 00-broken.pdf file is. testsdir/xpdf/00-broken.pdf - The auxiliary data file for the xpdf tests.
This may rule out the suitability of /usr/libexec/tests as the location for these files, given that your description seems to imply that libexec is for binaries only. But if it isn't, I'd be OK with using this directory.
It is also possible that introducing a new home for binaries would cause multi-lib heartburn in RPM, but I suspect that is a simple enough problem to fix, since we haven't really hit it with our use of /usr/libexec to date.
What do you mean by "multi-lib heartburn in RPM"? (Sorry, not familiar from the term and a couple of Google searches did't provide much insight.) FWIW, I have a preliminary atf.spec file here that provides subpackages for libraries, tests and binaries and things seem to be fine to my untrained eye. I can share this file if it helps at all.
You probably should open a ticket with the FHS working group: https://bugs.linuxfoundation.org/ (product: FHS) and explain all of this to them so that it has a chance of being officially added in the next major FHS revision, whenever that may be.
OK, I can do that. Wouldn't it be easier to illustrate the point / convince them if there was existing practice somewhere? (Not sure if BSD flavors count in this case!)
Above and beyond your proposal, the only other way I can think of handling this would be an elaborate namespacing of the test binaries in /usr/bin, e.g. /usr/bin/test-$pkg-$testname , and while that would be a solution that would permit a strict compliance with the FHS as it exists today, I'm not very excited about it and am not seriously recommending it as an alternative.
I don't think this would be an appropriate layout, and it doesn't play well with the structure of atf (let aside the clutter that would appear in /usr/bin). Let me provide some additional details of how the tests tree is structured.
Each directory in testsdir has an Atffile that indicates which test programs exist in that directory, which subdirectories to recurse into, and additional properties that indicate meta-data for the test programs. So, for example, the testsdir would look like this:
testsdir/ - The top-level directory. This directory would be owned by a filesystem package and be a dependency of anything that places files in testsdir/.
testsdir/Atffile - The main control file. This has the knowledge to recurse into the direct subdirectories of testsdir/ looking for other Atffiles to execute. E.g. this would find testsdir/a/Atffile and testsdir/b/Atffile to source tests, but it would not find testsdir/c/Notatf nor testsdir/d/e/Atffile. This file would be installed by the "atf" tools package, as it does not make any sense to have it without the corresponding tools.
testsdir/lutok/ - Owned by a lutok-tests package, including all the subfiles and subdirectories.
testsdir/lutok/Atffile - The control file for the tests provided by lutok. Contains the list of test programs to run and any optional meta-data for them. It also includes other subdirectories to "recurse" into.
testsdir/lutok/operations_test - A test program for the "operations" module of the source tree. (The name of the binary is arbitrary.)
testsdir/lutok/state_test - Another test program.
testsdir/lutok/detail/ - A subdirectory of lutok's tests containing tests for internal modules only. (This would mimic the structure of the source package.)
testsdir/lutok/detail/Atffile - Same as above, but for the tests in this subdirectory.
testsdir/lutok/detail/internal_test - Yet another test program.
testsdir/lutok/detail/state_test - Another test program. Note that this can have the same name as the program in the parent directory! There is nothing to prevent this, and thus flattening the test programs into /usr/bin as you propose would be very complicated.
testsdir/lutok/foo/ - Another subdirectory for additional tests, etc.
testsdir/foobar/ - Another package, with the same structure as above.
testsdir/baz/ - Another package, but this time without any Atffile in it. This would not be executed by atf-run because the top-level Atffile would discard this subdirectory. (Kyua, the replacement of atf, could run these tests though provided that a Kyuafile was created within the subdirectory. This is out of scope now though.)
A little detail to note above is that in a project with a good amount of test coverage, there will potentially be a test program per each source file. This means that the testsdir contains lots of binaries, and you don't want that many, potentially with conflicting names, anywhere in your path! To illustrate the point, check out this output page which contains the results of the execution of the NetBSD test suite:
http://www.whooppee.com/~paul/amd64-results/last_atf.html
Anyway. This layout may remind you a lot of make(1) and a traditional source tree populated with Makefiles. And actually, this is the main idea of how the tests tree is organized and is an artifact of ATF's origins (NetBSD's source tree). However, having played with other large source code bases, I can say that this organization is not unique to NetBSD.
We'd discuss extensively whether this layout is suboptimal for Fedora's guidelines or not... But please note that all I'm trying to achieve here is to figure out how to package an existing piece of software, not how to re-architect it!
If you made it this far again, thank you.
On 02/06/2012 03:58 PM, Julio Merino wrote:
What do you mean by "multi-lib heartburn in RPM"? (Sorry, not familiar from the term and a couple of Google searches did't provide much insight.) FWIW, I have a preliminary atf.spec file here that provides subpackages for libraries, tests and binaries and things seem to be fine to my untrained eye. I can share this file if it helps at all.
No, what I meant here is that RPM, when dealing with multi-arch support (essentially the ability to simultaneously foo.i686.rpm and foo.x86_64.rpm containing the same set of files, just built for the different architecture targets), won't understand /usr/tests (or /usr/libexec/tests for that matter).
However, if you want to be able to support this use-case, you'll need to do something here to enable it, such as requiring that architecture be embedded in the naming scheme, e.g. /usr/tests/xpdf-x86_64/
I'm not sure how much value there is in running the test suites for supported architectures aside from the default though. If there isn't a desire for this, we just need to explicitly state that packages containing test-suites cannot be multi-arch.
You probably should open a ticket with the FHS working group: https://bugs.linuxfoundation.org/ (product: FHS) and explain all of this to them so that it has a chance of being officially added in the next major FHS revision, whenever that may be.
OK, I can do that. Wouldn't it be easier to illustrate the point / convince them if there was existing practice somewhere? (Not sure if BSD flavors count in this case!)
Honestly, I don't know. Common sense says yes, but common sense has yet to appear in the FHS revision process. ;)
Fedora does not have to wait for the FHS to approve it, but we do want the things that we adopt to also be driven through the FHS approval process with the hopes that we can "merge" these exceptions "upstream".
Above and beyond your proposal, the only other way I can think of handling this would be an elaborate namespacing of the test binaries in /usr/bin, e.g. /usr/bin/test-$pkg-$testname , and while that would be a solution that would permit a strict compliance with the FHS as it exists today, I'm not very excited about it and am not seriously recommending it as an alternative.
I don't think this would be an appropriate layout, and it doesn't play well with the structure of atf (let aside the clutter that would appear in /usr/bin). Let me provide some additional details of how the tests tree is structured.
[LAYOUT INFO SNIPPED]
Yes, I understand this. If we were to propose such a layout, we'd put all the other stuff in say, /usr/share/tests/$pkg/, then provide symlinks for the binaries into that hierarchy.
But like I said, I don't really like it.
A little detail to note above is that in a project with a good amount of test coverage, there will potentially be a test program per each source file. This means that the testsdir contains lots of binaries, and you don't want that many, potentially with conflicting names, anywhere in your path! To illustrate the point, check out this output page which contains the results of the execution of the NetBSD test suite:
This further illustrates why I don't really like it. I was simply stating it because inevitably, someone else would, and I wanted it on the record has having been considered and discarded. ;)
~tom
== Fedora Project
On 02/06/2012 05:30 PM, Tom Callaway wrote:
On 02/06/2012 03:58 PM, Julio Merino wrote:
What do you mean by "multi-lib heartburn in RPM"? (Sorry, not familiar from the term and a couple of Google searches did't provide much insight.) FWIW, I have a preliminary atf.spec file here that provides subpackages for libraries, tests and binaries and things seem to be fine to my untrained eye. I can share this file if it helps at all.
No, what I meant here is that RPM, when dealing with multi-arch support (essentially the ability to simultaneously foo.i686.rpm and foo.x86_64.rpm containing the same set of files, just built for the different architecture targets), won't understand /usr/tests (or /usr/libexec/tests for that matter).
However, if you want to be able to support this use-case, you'll need to do something here to enable it, such as requiring that architecture be embedded in the naming scheme, e.g. /usr/tests/xpdf-x86_64/
Um, rpm doesn't do multilib conflict resolution based on specific directories but file type, essentially 32bit vs 64bit ELF. So if foo-test.i686 and too-test.x86_64 both place an ELF binary into eg /usr/libexec/tests/foo, the file from the preferred arch (normally the 64bit one) will "win", the other file is not installed at all and no conflict is raised.
Whether you actually *want* that behavior is another question: tests and their associated data can just as well be arch-specific or arch-independent. The only safe assumption is to assume arch-specific and put all test executables and associated data into arch-specific paths (whether some variant of lib vs lib64 style differentation or by actually embedding the arch string).
- Panu -
On 02/07/2012 10:36 AM, Panu Matilainen wrote:
On 02/06/2012 05:30 PM, Tom Callaway wrote:
On 02/06/2012 03:58 PM, Julio Merino wrote:
What do you mean by "multi-lib heartburn in RPM"? (Sorry, not familiar from the term and a couple of Google searches did't provide much insight.) FWIW, I have a preliminary atf.spec file here that provides subpackages for libraries, tests and binaries and things seem to be fine to my untrained eye. I can share this file if it helps at all.
No, what I meant here is that RPM, when dealing with multi-arch support (essentially the ability to simultaneously foo.i686.rpm and foo.x86_64.rpm containing the same set of files, just built for the different architecture targets), won't understand /usr/tests (or /usr/libexec/tests for that matter).
However, if you want to be able to support this use-case, you'll need to do something here to enable it, such as requiring that architecture be embedded in the naming scheme, e.g. /usr/tests/xpdf-x86_64/
Um, rpm doesn't do multilib conflict resolution based on specific directories but file type, essentially 32bit vs 64bit ELF. So if foo-test.i686 and too-test.x86_64 both place an ELF binary into eg /usr/libexec/tests/foo, the file from the preferred arch (normally the 64bit one) will "win", the other file is not installed at all and no conflict is raised.
Whether you actually *want* that behavior is another question: tests and their associated data can just as well be arch-specific or arch-independent. The only safe assumption is to assume arch-specific and put all test executables and associated data into arch-specific paths (whether some variant of lib vs lib64 style differentation or by actually embedding the arch string).
%{_libdir}/<package>/<somewhere> avoids all these issues.
Ralf
On 02/07/2012 12:04 PM, Ralf Corsepius wrote:
On 02/07/2012 10:36 AM, Panu Matilainen wrote:
On 02/06/2012 05:30 PM, Tom Callaway wrote:
On 02/06/2012 03:58 PM, Julio Merino wrote:
What do you mean by "multi-lib heartburn in RPM"? (Sorry, not familiar from the term and a couple of Google searches did't provide much insight.) FWIW, I have a preliminary atf.spec file here that provides subpackages for libraries, tests and binaries and things seem to be fine to my untrained eye. I can share this file if it helps at all.
No, what I meant here is that RPM, when dealing with multi-arch support (essentially the ability to simultaneously foo.i686.rpm and foo.x86_64.rpm containing the same set of files, just built for the different architecture targets), won't understand /usr/tests (or /usr/libexec/tests for that matter).
However, if you want to be able to support this use-case, you'll need to do something here to enable it, such as requiring that architecture be embedded in the naming scheme, e.g. /usr/tests/xpdf-x86_64/
Um, rpm doesn't do multilib conflict resolution based on specific directories but file type, essentially 32bit vs 64bit ELF. So if foo-test.i686 and too-test.x86_64 both place an ELF binary into eg /usr/libexec/tests/foo, the file from the preferred arch (normally the 64bit one) will "win", the other file is not installed at all and no conflict is raised.
Whether you actually *want* that behavior is another question: tests and their associated data can just as well be arch-specific or arch-independent. The only safe assumption is to assume arch-specific and put all test executables and associated data into arch-specific paths (whether some variant of lib vs lib64 style differentation or by actually embedding the arch string).
%{_libdir}/<package>/<somewhere> avoids all these issues.
Yes, except for noarch packages where %{_libdir} is not meaningful.
- Panu -
On 02/08/2012 05:10 PM, Tom Callaway wrote:
On 02/07/2012 05:04 AM, Ralf Corsepius wrote:
%{_libdir}/<package>/<somewhere> avoids all these issues.
No it doesn't.
Surely it is. It's what autoconf calls pkglibdir: http://sourceware.org/autobook/autobook/autobook_106.html
Not to mention the fact that it isn't really appropriate to be putting binaries and architecture independent files there.
pkglibdir has been common practice for more than a decade. It's a packages private playground, a package can play almost all games it wants to.
Using it would duplicate "shareable" binaries, but it at the same time allows parallel installation of multiarched binaries. As the binaries we are talking about here are "test" binaries, this would be exactly what the OS asked for.
He could use $libdir/<package>/bin or $libdir/<package>/tests at his personal preference.
Ralf
On 02/06/2012 04:59 AM, Julio Merino wrote:
Hello,
The purpose of this long email is to justify the need for a /usr/tests directory and to ask for approval for its creation, with the final goal of packaging ATF, its tests and the tests provided by arbitrary applications (e.g. the just-approved lutok package[1]). This email is long because I feel the need to provide tons of background to explain why this directory is required. I hope the text is clear and that it makes enough sense to convince you of its approval. Here we go!
<snip>
The first thing coming in my mind was using something like /usr/pkg1/test and put a configuration file for each package (containing path and other stuff) into something like /etc/tests/conf.d/. The benefit of this would be that additional information can be stored for each test.
Just my 2 cents
regards
Markus
On 02/06/2012 01:37 PM, Markus Mayer wrote:
The first thing coming in my mind was using something like /usr/pkg1/test and put a configuration file for each package (containing path and other stuff) into something like /etc/tests/conf.d/. The benefit of this would be that additional information can be stored for each test.
The FHS explicitly forbids that:
Large software packages must not use a direct subdirectory under the /usr hierarchy.
~tom
== Fedora Project
On 02/06/2012 01:46 PM, Tom Callaway wrote:
On 02/06/2012 01:37 PM, Markus Mayer wrote:
The first thing coming in my mind was using something like /usr/pkg1/test and put a configuration file for each package (containing path and other stuff) into something like /etc/tests/conf.d/. The benefit of this would be that additional information can be stored for each test.
The FHS explicitly forbids that:
Large software packages must not use a direct subdirectory under the /usr hierarchy.
I'd use a directory under %{libdir}/<pkg> (aka. pkglibdir).
Ralf
On Sun, 05 Feb 2012 22:59:10 -0500, JM (Julio) wrote:
The purpose of this long email is to justify the need for a /usr/tests directory
To put this in the context of a Linux distribution, let's consider a fictitious example: imagine that glib and gtk came with ATF-based tests. These tests would get installed into testsdir/glib/... and testsdir/gtk/... The user would then do "cd testsdir/gtk/ && atf-run" to execute the tests on his machine,
- The second is that ATF is not encoded anywhere in the path. I.e. the
directories are /usr/tests/pkg*, not /usr/atf/pkg* nor /usr/tests/atf/pkg*. The reason for this is that the package-specific tests needn't be implemented using the ATF libraries.
Still, the tests must be compatible with the testing framework's set of tools. That's enough of a reason to add some sort of namespace to the root directory.
I also find it contradictory how you refer to "atf-run", "atf-report" and "a collection of tools", which have "atf-" in their name. Apparently, they are stored in global search path for executables (and therefore it is reasonable to prefix them with something to avoid creating too generic names, and hopefully you wouldn't propose dropping the "atf-" prefix from their names either).
Surely the testing framework would not be renamed regularly. And software authors, who would use this framework for writing own tests, would need to be able to rely on an interface for retrieving filesystem path details anyway. For example, a pkg-config file that can be queried. Or a similar atf-config script that returns various paths.
In fact, Kyua, the replacement for ATF, is able to execute these same unmodified tests and can also execute tests written other frameworks. In a future world, packages would install arbitrary tests into /usr/tests written using their preferred libraries (e.g. pyunit, autotest, etc.) and a single tool (Kyua) would run them all. Encoding the tool name in the directory introduces a dependency on the tool that does not necessarily exist.
Implementation details here would need to be discussed. The names of the tools are irrelevant so far. The various testing frameworks would need to be compatible somehow and meet the requirements of the controlling tools. One couldn't dump arbitrary tests into a shared directory and e.g. mix automated, semi-automated and non-automated tests.
Leaving FHS issues aside, in general it is not nice to let any package "occupy" directories, which have very generic names, without very good reason. That's not limited to /usr, and it's not a "first come, first served" thing either. Imagine, a second group of testing framework authors would also want to use a global /usr/tests in order to store incompatible tests in there.
On 2/7/12 6:26 AM, Michael Schwendt wrote:
On Sun, 05 Feb 2012 22:59:10 -0500, JM (Julio) wrote:
The purpose of this long email is to justify the need for a /usr/tests directory
To put this in the context of a Linux distribution, let's consider a fictitious example: imagine that glib and gtk came with ATF-based tests. These tests would get installed into testsdir/glib/... and testsdir/gtk/... The user would then do "cd testsdir/gtk/&& atf-run" to execute the tests on his machine,
- The second is that ATF is not encoded anywhere in the path. I.e. the
directories are /usr/tests/pkg*, not /usr/atf/pkg* nor /usr/tests/atf/pkg*. The reason for this is that the package-specific tests needn't be implemented using the ATF libraries.
Still, the tests must be compatible with the testing framework's set of tools. That's enough of a reason to add some sort of namespace to the root directory.
If you created "unknown" subdirectories under /usr/tests/, they would be skipped by the tools. (Actually, the goal would be for the tools to eventually run arbitrary test programs regardless of whichever libraries they use.)
I also find it contradictory how you refer to "atf-run", "atf-report" and "a collection of tools", which have "atf-" in their name.
Contradictory with what? You certainly don't want a tool named "run" in your path. There, of course, would be an "atf" dispatcher that accepted subcommands and executed atf-run and atf-report on your back (much like git does). But there isn't.
Surely the testing framework would not be renamed regularly. And software authors, who would use this framework for writing own tests, would need to be able to rely on an interface for retrieving filesystem path details anyway. For example, a pkg-config file that can be queried. Or a similar atf-config script that returns various paths.
Such a thing (atf-config) already exists, although it cannot tell what the testsdir is. It actually should not, because the location of the testsdir is a packaging issue and not something the tools care about at all. atf-run cares about what's in the current directory and won't try to load stuff from /usr/tests by itself, for example.
In fact, Kyua, the replacement for ATF, is able to execute these same unmodified tests and can also execute tests written other frameworks. In a future world, packages would install arbitrary tests into /usr/tests written using their preferred libraries (e.g. pyunit, autotest, etc.) and a single tool (Kyua) would run them all. Encoding the tool name in the directory introduces a dependency on the tool that does not necessarily exist.
Implementation details here would need to be discussed. The names of the tools are irrelevant so far.
Exactly; the names are irrelevant.
Let's say you start by putting all the tests into /usr/tests/atf/<package>, which would be the case if the name mattered. This directory contains the top-level Atffile, and each subdirectory contains an Atffile of its own. atf-run can only be executed from within /usr/tests/atf.
Now, a new tool comes along (let's call it "magic") that is able to interpret atf-based test cases but can also run pyunit tests. This tool needs its own Magicfiles. How do you make the previous /usr/tests/atf/ directory tree work? Surely putting Magicfiles within /usr/tests/atf/ would not be appropriate, because this is in the atf namespace. But you actually need to have those files in there to let the magic tool know what to do.
Before you say "well, the tools should auto-discover the tests instead of reading Magicfiles et. al.": no, they should not. These files include much more than just a list of test programs. They provide metadata, such as what "framework" the test programs are written with, or arbitrary configuration properties for the programs.
The various testing frameworks would need to be compatible somehow and meet the requirements of the controlling tools. One couldn't dump arbitrary tests into a shared directory and e.g. mix automated, semi-automated and non-automated tests.
Why not? As I said above, what the "controlling tool" runs is defined by a file in the tests directory (Atffile). Any of these incompatible files would not have a matching Atffile, and thus atf-run would not see them.
Leaving FHS issues aside, in general it is not nice to let any package "occupy" directories, which have very generic names, without very good reason. That's not limited to /usr, and it's not a "first come, first served" thing either. Imagine, a second group of testing framework authors would also want to use a global /usr/tests in order to store incompatible tests in there.
Such other systems do not exist, and I seriously doubt any will. If they ever do, they'd have to play well with existing practice, just as you ask any package to respect the subpackage-directories layout in /usr/share. And asking such imaginary systems to respect the proposed layout would be trivial for them, so it doesn't sound too unreasonable.
Lastly: please keep in mind that all I want to do is package an existing piece of software, not redesign it. Comments of how things should be or should not be done within the tool are not relevant in this discussion. (But I'll gladly take suggestions if they are raised where they belong: in the upstream mailing lists.)
packaging@lists.fedoraproject.org
-
Julio Merino -
Markus Mayer -
Michael Schwendt -
Panu Matilainen -
Ralf Corsepius -
Tom Callaway