I talked about this at todays meeting, but I thought I would note it here for further input and also see if there was interest in doing a Virtual Fedora Activity Day next week to finish out hosted plans.
background:
We have 5 servers at serverbeach (Thanks for donating them SB Folks!)
They were setup long ago and are getting a bit old in the tooth, so we are getting 5 new machines to migrate our stuff to. This will allow us to be on hardware thats newer/better supported and also move to new OS and re-arrange things. We must migrate to those new machines by 2011-12-11. (around a month from now).
current:
serverbeach01 -> a mirror
serverbeach02 -> collab01 (all our lists.fedoraproject.org lists). ns01
serverbeach03 -> collab02 (a warm spare of collab01, it rsyncs from there every hour).
serverbeach04 -> hosted01 (all fedorahosted.org)
serverbeach05 -> hosted02 (a warm spare, but currently down and not working).
Plan:
serverbeach06 -> collab03 ns01 serverbeach07 -> collab04
(move these to rhel6 for both host/guest. Also, look at setting up drbd for /srv on them, so we can easily switch over from 03 to 04 in case of failure. If we do do drbd, I'd like to schedule at least one failover to make sure it really is all working as expected. ns01 can go on either. ) We will also need to add drbd rpms to our repo.
That leaves us with hosted to deal with, and serverbeach08/09/10 to do it in (currently). We have discussed lots of plans with hosted, I'd like to propose meeting up on IRC next Wed to do a virtual Fedora Activity Day around planning that. Or we could just try and work out details here on list.
Things we have discussed:
* Moving lists.fedorahosted.org to collab* * Moving projects to a CNAME or aliased setup so we can then move them to whatever backend host we like. (what exact changes?) * Making whatever changes we need now so we could move to spinning out some projects into "the cloud" instances. What are those specific changes?
Issues:
* Backups of hosted01 take ~6 hours, which is not very good. ;( (thats incrementals, full backups are much slower... days). * Hosted01 is pretty heavily loaded. Some of which may just disappear when we move to rhel6 (I know some of it will). * Ideally we could bring up projects on a new instance in the event of a failure of a node.
I'd like to have a SOP with steps for migrating hosted instances soon. So, we can say, 'here's what we are going to change and why'. Then we can get setup and start moving projects that opt-in, then have a flag day before our migration window to just move everyone else.
Thoughts?
kevin
As someone who has intimate knowledge of some special setups on hosted, I'd be happy to participate. Just let me know when and where to show up :)
On Thu, Nov 10, 2011 at 4:45 PM, Kevin Fenzi kevin@scrye.com wrote:
I talked about this at todays meeting, but I thought I would note it here for further input and also see if there was interest in doing a Virtual Fedora Activity Day next week to finish out hosted plans.
background:
We have 5 servers at serverbeach (Thanks for donating them SB Folks!)
They were setup long ago and are getting a bit old in the tooth, so we are getting 5 new machines to migrate our stuff to. This will allow us to be on hardware thats newer/better supported and also move to new OS and re-arrange things. We must migrate to those new machines by 2011-12-11. (around a month from now).
current:
serverbeach01 -> a mirror
serverbeach02 -> collab01 (all our lists.fedoraproject.org lists). ns01
serverbeach03 -> collab02 (a warm spare of collab01, it rsyncs from there every hour).
serverbeach04 -> hosted01 (all fedorahosted.org)
serverbeach05 -> hosted02 (a warm spare, but currently down and not working).
Plan:
serverbeach06 -> collab03 ns01 serverbeach07 -> collab04
(move these to rhel6 for both host/guest. Also, look at setting up drbd for /srv on them, so we can easily switch over from 03 to 04 in case of failure. If we do do drbd, I'd like to schedule at least one failover to make sure it really is all working as expected. ns01 can go on either. ) We will also need to add drbd rpms to our repo.
That leaves us with hosted to deal with, and serverbeach08/09/10 to do it in (currently). We have discussed lots of plans with hosted, I'd like to propose meeting up on IRC next Wed to do a virtual Fedora Activity Day around planning that. Or we could just try and work out details here on list.
Things we have discussed:
- Moving lists.fedorahosted.org to collab*
- Moving projects to a CNAME or aliased setup so we can then move them
to whatever backend host we like. (what exact changes?)
- Making whatever changes we need now so we could move to spinning out
some projects into "the cloud" instances. What are those specific changes?
Issues:
- Backups of hosted01 take ~6 hours, which is not very good. ;( (thats
incrementals, full backups are much slower... days).
- Hosted01 is pretty heavily loaded. Some of which may just disappear
when we move to rhel6 (I know some of it will).
- Ideally we could bring up projects on a new instance in the event of
a failure of a node.
I'd like to have a SOP with steps for migrating hosted instances soon. So, we can say, 'here's what we are going to change and why'. Then we can get setup and start moving projects that opt-in, then have a flag day before our migration window to just move everyone else.
Thoughts?
kevin
infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
On Thu, 10 Nov 2011 14:45:16 -0700 Kevin Fenzi kevin@scrye.com wrote:
I talked about this at todays meeting, but I thought I would note it here for further input and also see if there was interest in doing a Virtual Fedora Activity Day next week to finish out hosted plans.
background:
We have 5 servers at serverbeach (Thanks for donating them SB Folks!)
They were setup long ago and are getting a bit old in the tooth, so we are getting 5 new machines to migrate our stuff to. This will allow us to be on hardware thats newer/better supported and also move to new OS and re-arrange things. We must migrate to those new machines by 2011-12-11. (around a month from now).
current:
serverbeach01 -> a mirror
serverbeach02 -> collab01 (all our lists.fedoraproject.org lists). ns01
serverbeach03 -> collab02 (a warm spare of collab01, it rsyncs from there every hour).
serverbeach04 -> hosted01 (all fedorahosted.org)
serverbeach05 -> hosted02 (a warm spare, but currently down and not working).
Plan:
serverbeach06 -> collab03 ns01 serverbeach07 -> collab04
(move these to rhel6 for both host/guest. Also, look at setting up drbd for /srv on them, so we can easily switch over from 03 to 04 in case of failure. If we do do drbd, I'd like to schedule at least one failover to make sure it really is all working as expected. ns01 can go on either. ) We will also need to add drbd rpms to our repo.
That leaves us with hosted to deal with, and serverbeach08/09/10 to do it in (currently). We have discussed lots of plans with hosted, I'd like to propose meeting up on IRC next Wed to do a virtual Fedora Activity Day around planning that. Or we could just try and work out details here on list.
Things we have discussed:
- Moving lists.fedorahosted.org to collab*
- Moving projects to a CNAME or aliased setup so we can then move them to whatever backend host we like. (what exact changes?)
- Making whatever changes we need now so we could move to spinning out some projects into "the cloud" instances. What are those specific changes?
Issues:
- Backups of hosted01 take ~6 hours, which is not very good. ;( (thats incrementals, full backups are much slower... days).
- Hosted01 is pretty heavily loaded. Some of which may just disappear when we move to rhel6 (I know some of it will).
- Ideally we could bring up projects on a new instance in the event of a failure of a node.
I'd like to have a SOP with steps for migrating hosted instances soon.
1. I think we need some volunteer projects. Do we know of any enterprising folks who would like to be suckers.... err test subjects?
2. For collab2 and hosted2 - would it be possible to run both of those secondary machines on one of the new sb boxes? Disk-space wise we should be able to make it work - and it from a failover standpoint it feels unlikely we would need to failover both hosted1 and collab1 at the same time.
3. Then do backups from collab2 and hosted2 - until we have our stuff together for hosted-tng.
So, we can say, 'here's what we are going to change and why'. Then we can get setup and start moving projects that opt-in, then have a flag day before our migration window to just move everyone else.
I suspect we'll need to waddle them all through a little at a time. Remember getting rid of cvs.fedoraproject.org? like that :(
-sv
On Fri, 11 Nov 2011 09:55:58 -0500 seth vidal skvidal@fedoraproject.org wrote:
- I think we need some volunteer projects. Do we know of any
enterprising folks who would like to be suckers.... err test subjects?
I'm sure we can find a few projects that wouldn't mind...
- For collab2 and hosted2 - would it be possible to run both of
those secondary machines on one of the new sb boxes? Disk-space wise we should be able to make it work - and it from a failover standpoint it feels unlikely we would need to failover both hosted1 and collab1 at the same time.
That possible, but it might hit I/O on that machine a bit hard. Depending on how we sync to those secondaries.
- Then do backups from collab2 and hosted2 - until we have our stuff
together for hosted-tng.
Yeah, if we could get 2 to work that would work too probibly. More I/O tho.
So, we can say, 'here's what we are going to change and why'. Then we can get setup and start moving projects that opt-in, then have a flag day before our migration window to just move everyone else.
I suspect we'll need to waddle them all through a little at a time. Remember getting rid of cvs.fedoraproject.org? like that :(
Yep. Until flag day...
kevin
On Fri, 2011-11-11 at 16:23 -0700, Kevin Fenzi wrote:
- I think we need some volunteer projects. Do we know of any
enterprising folks who would like to be suckers.... err test
subjects?
I'm sure we can find a few projects that wouldn't mind...
Volunteering for the R2spec project (though it is small and no so lively).
Pierre
On Mon, 2011-11-14 at 22:38 +0100, Pierre-Yves Chibon wrote:
On Fri, 2011-11-11 at 16:23 -0700, Kevin Fenzi wrote:
- I think we need some volunteer projects. Do we know of any
enterprising folks who would like to be suckers.... err test
subjects?
I'm sure we can find a few projects that wouldn't mind...
Volunteering for the R2spec project (though it is small and no so lively).
And after confirmation from Stanislav, we're ok to test on FedoraReview (both project don't have mailing list so we won't be testing this).
Pierre
On Fri, Nov 11, 2011 at 9:55 AM, seth vidal skvidal@fedoraproject.org wrote:
- I think we need some volunteer projects. Do we know of any
enterprising folks who would like to be suckers.... err test subjects?
I recently got added as an admin for the "pym" project on Fedora Hosted, and I'd be happy to let you use pym as a guinea pig project.
-- Jared Smith Fedora Project Leader
infrastructure@lists.fedoraproject.org
-
Jared K. Smith -
Jon Stanley -
Kevin Fenzi -
Pierre-Yves Chibon -
Seth Vidal