Hi folks!
So, since we've been getting Workstation ostree installer images in the last few Rawhide composes, I've been setting up openQA to test it. This is not at present deployed to production openQA, but it *is* deployed to staging openQA. Unfortunately the last few Rawhide composes have contained a broken anaconda so the image has not been built, but the 20170615.n.0 compose *did* feature a working image, and you can see the test results there:
https://openqa.stg.fedoraproject.org/tests/overview?distri=fedora&versio...
(note that *right now* there's an infra outage going on so as I write this the page is inaccessible, but all should be back to normal shortly).
Look for the Workstation dvd-ostree 'Flavor'. As you can see, for now I just have it running a subset of the regular Workstation tests, with the update-related tests - which assume an RPM-based system - left out, for now. Everything mostly appears to work well, except for the known bug https://bugzilla.redhat.com/show_bug.cgi?id=1193590 (that's the cause of most of the tests showing up as 'soft failures', the 'orange' state, rather than 'passes' / green).
So I have two purposes here: firstly just to let you know that this testing is now happening (they will be run for any Rawhide or Branched compose which contains a Workstation ostree installer image), and secondly to ask about any other testing that would be useful. Is there any practical way we can actually test the ostree update process, for instance? For testing RPM-based updates, what we do is install a 'dummy' python3-kickstart package, with a NEVR known to be lower than any 'real' version of the package, so we know for sure that an update will be available from the official repositories, and then we go ahead and run an update in the usual fashion. Would there be any similar kind of possibility for testing ostree updates? The goals of openQA testing are to test as 'realistically' as possible, so we only test unmodified images, and want to have the test behave as similarly as possible to how the 'real' operation would behave on an end user's system.
And besides updates, can anyone think of any other test that would be particularly useful to run?
Of course, to be clear, this is mostly 'advisory' testing for now: this image is still not a release-blocking image, so failures of the tests will not automatically trigger any kind of special attention or even bugs being filed, it will require people to be interested in looking at them and filing bugs. I will try and do so, of course.
On Wed, Jun 21, 2017 at 05:07:12PM -0700, Adam Williamson wrote:
'dummy' python3-kickstart package, with a NEVR known to be lower than any 'real' version of the package, so we know for sure that an update will be available from the official repositories, and then we go ahead and run an update in the usual fashion. Would there be any similar kind of possibility for testing ostree updates? The goals of openQA testing are to test as 'realistically' as possible, so we only test unmodified images, and want to have the test behave as similarly as possible to how the 'real' operation would behave on an end user's system.
You should be able to go back and forth between various ostree commits, right? Possibly in addition to a "do full install, test" thing, there could just be a "update from previous to new, test" image. Or, am I misunderstanding?
On Wed, Jun 21, 2017, at 08:07 PM, Adam Williamson wrote:
Look for the Workstation dvd-ostree 'Flavor'. As you can see, for now I just have it running a subset of the regular Workstation tests, with the update-related tests - which assume an RPM-based system -
rpm-ostree is also an RPM-based system; it's even literally in the name of the project! It's *also* an ostree-based system. Just like hybrid cars are both gasoline and electric.
That said of course, things are different since again it's a hybrid and one needs to consider the base tree vs layered packages.
The most realistic test is to test updates from "previous stable image/installer". to "latest tree".
(As well of course to test layered package updates, but that runs into https://github.com/projectatomic/rpm-ostree/issues/391 )
There's a variety of rpm-ostree tests that are used for Atomic Host here: https://github.com/projectatomic/atomic-host-tests/tree/master/roles
That all said I'm still not sure about doing these types of tests in AutoQA versus something more like Ansible as we already use for a-h-t.
Though testing gnome-software's forthcoming rpm-ostree backend would be back in the realm of GUI testing.
On Fri, 2017-06-23 at 09:54 -0400, Colin Walters wrote:
On Wed, Jun 21, 2017, at 08:07 PM, Adam Williamson wrote:
Look for the Workstation dvd-ostree 'Flavor'. As you can see, for now I just have it running a subset of the regular Workstation tests, with the update-related tests - which assume an RPM-based system -
rpm-ostree is also an RPM-based system; it's even literally in the name of the project! It's *also* an ostree-based system. Just like hybrid cars are both gasoline and electric.
Well, yes, but I'm usually talking from the *user* perspective. So far as the user is concerned, it's not an RPM-based system: we can't test updates by fiddling around with dnf, is the pertinent point here.
That said of course, things are different since again it's a hybrid and one needs to consider the base tree vs layered packages.
The most realistic test is to test updates from "previous stable image/installer". to "latest tree".
(As well of course to test layered package updates, but that runs into https://github.com/projectatomic/rpm-ostree/issues/391 )
There's a variety of rpm-ostree tests that are used for Atomic Host here: https://github.com/projectatomic/atomic-host-tests/tree/master/roles
That all said I'm still not sure about doing these types of tests in AutoQA versus something more like Ansible as we already use for a-h-t.
openQA, not AutoQA.
If we have another system where we can test the update experience exactly as it works for a real end user - starting from deploying ostree Workstation in the way we actually expect a user to do so, starting from our actual nightly / candidate images - then that's great, let's do that. But if we don't, openQA is probably a practical way to do it. What I'm really kinda looking for is more specific details on the level of "here is approximately what you could do to reliably put a Workstation ostree system into a state where an update for it would be available through the usual mechanism".
Though testing gnome-software's forthcoming rpm-ostree backend would be back in the realm of GUI testing.
So, updating via GNOME Software is not yet possible? That's useful information, thanks.
Still, thinking about it, for ostree Workstation we really need to test two *separate* things, yes? Updating the base system, and then deploying and updating software on top of the base system. Again, more details on how that process is expected to work would be useful; are Flatpaks the only expected deployment method for apps on ostree Workstation, for instance?
On Fri, Jun 23, 2017, at 03:38 PM, Adam Williamson wrote:
Well, yes, but I'm usually talking from the *user* perspective. So far as the user is concerned, it's not an RPM-based system: we can't test updates by fiddling around with dnf, is the pertinent point here.
But one can (and is definitely expected to in many general cases) to use layered packages. A lot (but not all) of the functionality of dnf is also possible with rpm-ostree - increasingly so, for example we're working on supporting "removing" things from the base tree: https://github.com/projectatomic/rpm-ostree/pull/797
If we have another system where we can test the update experience exactly as it works for a real end user - starting from deploying ostree Workstation in the way we actually expect a user to do so, starting from our actual nightly / candidate images - then that's great, let's do that. But if we don't, openQA is probably a practical way to do it. What I'm really kinda looking for is more specific details on the level of "here is approximately what you could do to reliably put a Workstation ostree system into a state where an update for it would be available through the usual mechanism".
We already indeed do these types of things in CentOS CI.
Still, thinking about it, for ostree Workstation we really need to test two *separate* things, yes? Updating the base system, and then deploying and updating software on top of the base system.
Right!
Again, more details on how that process is expected to work would be useful; are Flatpaks the only expected deployment method for apps on ostree Workstation, for instance?
No. There's lots of stuff that isn't yet in flatpak form. And further, not everything is a desktop app.
I have `emacs flatpak-builder opensc pcsc-lite-ccid keepassx powerline tmux vagrant-libvirt virt-manager ykclient` layered here for example; about half of those are desktop apps I haven't bothered to try to flatpak, and most of them are privileged anyways. Half are utilities or userspace driver-type things like ykclient.
Now things like `fedpkg` live in my tools container.
I also have `oc cluster up` going for server side development.
There's a bit more here: https://fedoraproject.org/wiki/Workstation/AtomicWorkstation
(Though origin-clients isn't yet in "workstation-ostree" partly due to this ongoing slow moving debate about whether it's "the same workstation, just with rpm-ostree" or whether it's something more different than that where e.g. devel tools are always in a container, one uses openshift for local server development etc)
On Fri, 2017-06-23 at 15:56 -0400, Colin Walters wrote:
On Fri, Jun 23, 2017, at 03:38 PM, Adam Williamson wrote:
Well, yes, but I'm usually talking from the *user* perspective. So far as the user is concerned, it's not an RPM-based system: we can't test updates by fiddling around with dnf, is the pertinent point here.
But one can (and is definitely expected to in many general cases) to use layered packages. A lot (but not all) of the functionality of dnf is also possible with rpm-ostree - increasingly so, for example we're working on supporting "removing" things from the base tree: https://github.com/projectatomic/rpm-ostree/pull/797
Ah, I see, thanks for the clarification; so, long story short, we *could* actually test using dnf for install / remove / update of packages on Workstation-ostree? But is this also something that's already tested in CentOS CI? If so, probably not worth duplicating it.
If we have another system where we can test the update experience exactly as it works for a real end user - starting from deploying ostree Workstation in the way we actually expect a user to do so, starting from our actual nightly / candidate images - then that's great, let's do that. But if we don't, openQA is probably a practical way to do it. What I'm really kinda looking for is more specific details on the level of "here is approximately what you could do to reliably put a Workstation ostree system into a state where an update for it would be available through the usual mechanism".
We already indeed do these types of things in CentOS CI.
Great, one less thing for me to worry about, then. Well. I suppose there's one more thing to ask: do you test install from the *installer image* in CentOS CI? If not, how significant are the chances (do you think) that a system installed from the installer image might differ in its behaviour from one deployed via a filesystem image or however you get to the starting state of the CentOS CI tests?
Still, thinking about it, for ostree Workstation we really need to test two *separate* things, yes? Updating the base system, and then deploying and updating software on top of the base system.
Right!
Again, more details on how that process is expected to work would be useful; are Flatpaks the only expected deployment method for apps on ostree Workstation, for instance?
No. There's lots of stuff that isn't yet in flatpak form. And further, not everything is a desktop app.
OK, so, to summarize, you expect people to deploy stuff on top of ostree Workstation:
* Via layered packages (which is basically just 'dnf install foo' so far as the end user is concerned, right?)
* As Flatpaks, installed via GNOME Software
* As containers
Does that sound about right? Thanks!
On Fri, 2017-06-23 at 13:25 -0700, Adam Williamson wrote:
OK, so, to summarize, you expect people to deploy stuff on top of ostree Workstation:
- Via layered packages (which is basically just 'dnf install foo' so
far as the end user is concerned, right?)
As Flatpaks, installed via GNOME Software
As containers
Does that sound about right? Thanks!
If you want a full test matrix, things that need to work are:
* Updating or downgrading the base image, using commandline tools or gnome-software
* The same, in the presence of layered packages
* Installing, updating and removing flatpaks, using commandline tools of gnome-softare
* Installing, updating and removing layered packages, using commandline tools
* Installing, updating and removing containers, using commandline tools
On Fri, Jun 23, 2017 at 03:56:16PM -0400, Colin Walters wrote:
But one can (and is definitely expected to in many general cases) to use layered packages. A lot (but not all) of the functionality of dnf is also possible with rpm-ostree - increasingly so, for example we're working on supporting "removing" things from the base tree: https://github.com/projectatomic/rpm-ostree/pull/797
I think we'll want to test _both_ this and ostree updates (and rollbacks), right?
way to do it. What I'm really kinda looking for is more specific details on the level of "here is approximately what you could do to reliably put a Workstation ostree system into a state where an update for it would be available through the usual mechanism".
We already indeed do these types of things in CentOS CI.
Do the results of that get put out to fedmsg?
(Though origin-clients isn't yet in "workstation-ostree" partly due to this ongoing slow moving debate about whether it's "the same workstation, just with rpm-ostree" or whether it's something more different than that where e.g. devel tools are always in a container, one uses openshift for local server development etc)
FWIW, I'm in favor of "more different". Let's make it interesting and stand out.
On Fri, 2017-06-23 at 17:07 -0400, Matthew Miller wrote:
On Fri, Jun 23, 2017 at 03:56:16PM -0400, Colin Walters wrote:
But one can (and is definitely expected to in many general cases) to use layered packages. A lot (but not all) of the functionality of dnf is also possible with rpm-ostree - increasingly so, for example we're working on supporting "removing" things from the base tree: https://github.com/projectatomic/rpm-ostree/pull/797
I think we'll want to test _both_ this and ostree updates (and rollbacks), right?
way to do it. What I'm really kinda looking for is more specific details on the level of "here is approximately what you could do to reliably put a Workstation ostree system into a state where an update for it would be available through the usual mechanism".
We already indeed do these types of things in CentOS CI.
Do the results of that get put out to fedmsg?
It'd be great if they could be forwarded to ResultsDB; that would give us fedmsgs for free, I believe.
desktop@lists.fedoraproject.org