I realized I had only informally gone over some of the items which will be changing in December when hardware is moved. However the date is finally arriving so it needs to be laid out more formally :).
In December, Infrastructure will be moving to new racks with better power monitoring, networking, and other tools. One of the side goals of this is that any old hardware which has been meant to be recycled over the years is to be done so. Which brings us to a set of qa systems which were listed to be replaced a couple of years ago.
qa01, qa02, qa03, qa04, qa05, qa07, qa08 are all IBM systems which have reached the end of the road and are planned not to be moved over to the new racks. [They have been scheduled to be removed for 2-3 years now but it has finally come to that date.] What processes are currently running on these systems and what is needed to replace them?
On Thu, 2017-11-02 at 12:41 -0400, Stephen John Smoogen wrote:
I realized I had only informally gone over some of the items which will be changing in December when hardware is moved. However the date is finally arriving so it needs to be laid out more formally :).
In December, Infrastructure will be moving to new racks with better power monitoring, networking, and other tools. One of the side goals of this is that any old hardware which has been meant to be recycled over the years is to be done so. Which brings us to a set of qa systems which were listed to be replaced a couple of years ago.
qa01, qa02, qa03, qa04, qa05, qa07, qa08 are all IBM systems which have reached the end of the road and are planned not to be moved over to the new racks. [They have been scheduled to be removed for 2-3 years now but it has finally come to that date.] What processes are currently running on these systems and what is needed to replace them?
qa05 and qa07 are the 'small' worker hosts for openQA (prod and stg, respectively). Each hosts 4 of the total 14 x86_64 worker instances. So losing them will cut the capacity of each openQA instance by about 30%.
What I'd need to replace them *most easily* is...another couple of hosts in infra (technically, 'able to mount the NFS shares from the openQA servers and connect back to the openQA servers via http and possibly websockets', I think) capable of running four or more x86_64 VMs simultaneously, basically. Up till now openQA worker hosts have always been dedicated to that task; there's no fundamental theoretical reason why openQA workers couldn't run on a box which also does some other stuff, though we'd have to check if there were any conflicts in requirements as to how the box has to be set up (check the ansible plays).
openQA has some newer capabilities that are basically about making 'more remote' workers possible, but I have not investigated those yet. It *may* be possible to use those to run workers in some kind of cloud, but that'd involve much more research and configuration and probably debugging than just doing straight-up replacements in infra.
Another possibility I guess is having a single more powerful box hosting workers for both prod and staging. Again I believe it may be possible to get openQA to do this but it's different from what we do now and I'd need to look into it.
infrastructure@lists.fedoraproject.org
-
Adam Williamson -
Stephen John Smoogen