Hi all,
BuildBot instance is up and running for some time now [1].
There is a couple of issues I would like to discuss here:
1) The tests are fragile
When some test fails to clean after itself it might affect other test or the next run of the test. We want to make sure that we make the tests as solid as possible but there will always be troubles (like issues in provider backends - blivet, NetworkManager). I'm working on improving this - I'm going to remove all openlmi-* packages, CIMOM and any other leftovers (like CIMOM repository in /var) before (or rather after?) the test.
2) Results are not visible enough
There is IRC bot on #openlmi channel of freenode notifying when test status changes (it used to fail and now it succeeds and vice versa). But it not visible enough. Also we don't want to send email to this list because it will be quite a lot of traffic. So, should I create another mailing list? openlmi- build? openlmi-ci?
3) Building machines updates
Now the test machines are updated manually but this should be probably automated too. But we need to know whether the build/test fails because of change in providers or system update. So the workflow should probably be like:
* Build our packages and run the test suite * Update the system * Build and test our packages again to see if the result is the same * Report it
Does it make sense? Any better ideas?
Tips and suggestions how to deal with these issues are welcomed.
Radek Novacek