Hi Dominik,

Thank you for your questions. See my comments inline.

On Thu, 23 Jul 2020 at 14:39, Dominik 'Rathann' Mierzejewski dominik@greysector.net wrote:

On Wednesday, 22 July 2020 at 14:49, Petr Šplíchal wrote:

...

Hi!

I am writing today to share a proposal for how we can make it easier to enable tests in Fedora CI.

We've heard the user story mentioned in the subject many times. Fedora developers have frequently mentioned the difficulties which they face with enabling tests in Fedora CI. Some people don't understand the complex yaml syntax. Other people find that they can't easily reproduce issues revealed by Fedora CI on their laptops. People have asked us to provide a tool for local testing:

https://pagure.io/fedora-ci/general/issue/4

A few of us have been working on a tool which we hope will improve the situation considerably. The proof of concept below shows how everyday developer tasks (related to testing) might look in the near future:

Enable basic smoke test for my component:

dnf install -y tmt
tmt init --template mini
vim plans/example.fmf
git add . && git commit -m tests && git push

Create a new test based on a shell/beakerlib template:

tmt test create --template shell tests/basic
tmt test create --template beakerlib tests/advanced

Run all/selected tests safely in a virtual machine:

tmt run
tmt run test --name smoke

I'd say it's good to have such option, but for quick testing a full VM is often too heavy and slow to boot. Even a container takes time to set up and run.

Yes, there are different needs and preferences related to the environment where the tests should be executed. We want to provide enough flexibility here so the 'provision' step allows you to choose your preferred way.

My primary environment would be local.

So for your case that would be 'provision --how local'. If you have a test box prepared for testing you could also connect to it using 'provision --how connect'.

...

Execute tests in my preferred environment:

tmt run --all provision --how container
tmt run --all provision --how virtual
tmt run --all provision --how local
tmt run --all provision --how ...

I find it difficult to guess what the above commands actually do. Do they run the tests or do they provision environments to run the tests? Or both?

The 'tmt run' command is used to run tests. By default it runs all steps (discover, provision, prepare, execute, report, finish) so the examples above would run tests in the requested environment.

It is also possible to run only selected steps: For example, one of the use cases shared with us was 'just tell me what tests would be run, do not run them', this can be achieved with the following:

tmt run discover

For detailed description of individual steps have a look here: https://tmt.readthedocs.io/en/latest/spec/steps.html

...

I would like to ask for your feedback about this. What do you think? Is this something that would be useful for you? What do you think of the syntax? We would love to hear back from you concerning what essential features you might expect from such a tool. It would also be helpful to know if there is anything that should be changed early in the tool design.

For more examples and frequently asked questions have a look here:

https://tmt.readthedocs.io/en/latest/examples.html
https://docs.fedoraproject.org/en-US/ci/tmt/

Looking forward to your feedback and ideas! :)

tmt examples are overcomplicated and not explained in enough detail. What is "metadata" and why do you need 4 "levels" of it?

Metadata levels allow to store all needed test execution data at appropriate location, the levels are described here: https://tmt.readthedocs.io/en/latest/spec.html

We want the config for simple/common use cases (such as running a shell script as a test) as simple as possible so that users do not have to study all the details but just create something like this:

execute: script: ./run-my-test.sh

If one needs to maintain larger set of tests the L1 metadata can come handy: https://tmt.readthedocs.io/en/latest/spec/tests.html

What are "tiers" for?

They are used to organize tests into groups by priority. https://tmt.readthedocs.io/en/latest/spec/tests.html#tier

Which items are mandatory and which are optional?

In order to enable tests you just have to define the 'execute' step. Use 'tmt init --template mini' to get a quick start.

What's "relevancy"?

It's a test (L1) attribute allowing to filter relevant tests. https://tmt.readthedocs.io/en/latest/spec/tests.html#relevancy Currently we're working on open-sourcing the implementation.

Why is there "extra-summary" in addition to "summary"? What's "extra-task"?

These are because of backward compatibility, for more see: https://tmt.readthedocs.io/en/latest/stories/cli.html?highlight=extra-task#n...

What are "plans" for?

Plans define which tests should be executed, how the environment should be provisioned and prepared, how the results should be reported. See the following page for more details about plans. https://tmt.readthedocs.io/en/latest/spec/plans.html

What are "stories" for?

Stories are an advanced, optional feature which can help you to track which features/requirements have been already implemented, tested and documented. https://tmt.readthedocs.io/en/latest/spec/stories.html https://github.com/psss/tmt/tree/master/stories

You're using a lot of terms as if everyone knows what they mean in your context.

Sorry for that. We're trying to document the concept as best as we can. Hopefully, the links above have helped. Perhaps we could add some more references to the examples page.

Thank you for all your feedback. Please, let us know what's still missing and how we can further improve the docs. Thanks.

psss...

Regards, Dominik -- Fedora https://getfedora.org | RPM Fusion http://rpmfusion.org There should be a science of discontent. People need hard times and oppression to develop psychic muscles. -- from "Collected Sayings of Muad'Dib" by the Princess Irulan _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org