Hi folks! You may have seen my recent post to test-announce and devel- announce about automated update testing using openQA. If you didn't, here's the executive summary:

* Since a few days ago, every time a critical path update is submitted or edited, a subset of the openQA tests is run on it

* The results of the tests are submitted to ResultsDB

* Bodhi will show these results in the Automated Tests tab

As of today, we've now extended this system to also run the tests on updates containing certain *non*-critical path packages as well. Specifically - since I've been planning this all along, and also because sgallagh asked for it - updates containing FreeIPA-adjacent packages (also postgresql) will have the 'server' update tests (which includes the FreeIPA tests) run on them. Here's an example of how this looks in openQA:

https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=26...

And how it looks in Bodhi:

https://bodhi.fedoraproject.org/updates/FEDORA-2017-15e2a038b2 (click on the 'Automated Tests' tab, it doesn't seem like Bodhi is capable of forming a hyperlink to it...)

The key tests for FreeIPA are role_deploy_domain_controller (the test that deploys a FreeIPA server and runs some basic checks on it), realmd_join_sssd (enrols into the domain using realmd and runs some checks that things seem to be working correctly - does a 'getent' for a domain user, logs in as a domain user, that sort of thing) and realmd_join_cockpit (enrols to the domain using Cockpit and does the same tests as realmd_join_sssd; also tests the FreeIPA webUI and tests changing a user's password both from the console and the webUI). If you see any of those failing, there might be a problem with the update.

The package whitelist at present looks like this:

UPDATEWL = {     # FreeIPA-related bits     '389-ds': ('server',),     '389-ds-base': ('server',),     'bind': ('server',),     'bind-dyndb-ldap': ('server',),     'certmonger': ('server',),     'ding-libs': ('server',),     'freeipa': ('server',),     'krb5-server': ('server',),     'pki-core': ('server',),     'sssd': ('server',),     'tomcat': ('server',),     # Cockpit-related bits     'cockpit': ('server',),     # PostgreSQL is a release-blocking server role     'postgresql': ('server',), }

It can be adjusted, but we can't just throw everything and the kitchen sink on there, because openQA only has so much capacity (we can run 14 tests concurrently at any given time, and that's for both compose and update testing), so we can't just go all the way down the dependency stack from freeipa to kernel. Still, if you think something non- critpath that's about as crucial to FreeIPA functioning as those packages are is missing from the list, do let me know.

Hopefully this will help us make sure we don't break FreeIPA with updates. Please do keep an eye on the results, if you're a packager or a tester!

If you need help interpreting any of the results, or would like a test re-run (they *do* occasionally fail for bogus reasons, though it's fairly rare for the server tests), please just poke me or Jan Sedlak (garretraziel on IRC).

These tests are run for relevant updates on all release branches, both stable and Branched (so right now, 24, 25 and 26).

I hope folks find this useful!