Greetings.
Our nice timetable of events for migrating collab01/02 and hosted01 ran into a monkey wrench the other week when we had to replace our new hardware again. We've made progress on the new new hardware and here's where we stand:
* I've installed a hosted02 on serverbeach06. This is a rhel5 clone of hosted01. This is mostly just to make sure if anything happens to hosted01 we are still able to bring something up quick.
* I've installed a collab03 (sb08) and collab04 (sb09). I've set them up with a drdb device for /srv to keep them in sync. The idea is that collab03 will be primary and in case anything fails, collab04 will be there to bring up quickly.
* I plan to install a hosted03 later today. (rhel6, hosted01 replacement).
I'd like to get things moved sooner rather than later. Also, we are starting to push up against the holiday season. So, I am proposing:
2011-12-12 (1 week away): collab move. (collab01 -> collab03)
2011-12-14 (next wed): hosted move. (hosted01 -> hosted03)
Outstanding questions:
1. When do we want to move the lists.fedorahosted.org? on the 12th? the 14th? or some other time entirely? We are going to need to get collab03 able to handle the additional domain and data.
2. On hosted, do we want to still change all projects to use project.fedorahosted.org ? Or should we perhaps just wait on that change? We could do that change on a per project opt in basis as we move them to other instances, possibly with less disruption. On the other hand, doing it now means we are ready to move things down the road.
From our testing on hosted03 before we had to rebuild, the rhel5/rhel6 upgrade seemed to go pretty smoothly from all I could tell.
Any other gotchas, ideas, or things we could setup to test?
If no one screams on timing, I will probibly announce the collab move later today and the hosted move on wed (giving people 1 weeks notice).
kevin
On Mon, 5 Dec 2011 12:30:31 -0700 Kevin Fenzi kevin@scrye.com wrote:
Greetings.
Our nice timetable of events for migrating collab01/02 and hosted01 ran into a monkey wrench the other week when we had to replace our new hardware again. We've made progress on the new new hardware and here's where we stand:
I've installed a hosted02 on serverbeach06. This is a rhel5 clone of hosted01. This is mostly just to make sure if anything happens to hosted01 we are still able to bring something up quick.
I've installed a collab03 (sb08) and collab04 (sb09). I've set them up with a drdb device for /srv to keep them in sync. The idea is
that collab03 will be primary and in case anything fails, collab04 will be there to bring up quickly.
- I plan to install a hosted03 later today. (rhel6, hosted01 replacement).
I'd like to get things moved sooner rather than later. Also, we are starting to push up against the holiday season. So, I am proposing:
2011-12-12 (1 week away): collab move. (collab01 -> collab03)
2011-12-14 (next wed): hosted move. (hosted01 -> hosted03)
Outstanding questions:
- When do we want to move the lists.fedorahosted.org? on the 12th?
the 14th? or some other time entirely? We are going to need to get collab03 able to handle the additional domain and data.
- On hosted, do we want to still change all projects to use
project.fedorahosted.org ? Or should we perhaps just wait on that change? We could do that change on a per project opt in basis as we move them to other instances, possibly with less disruption. On the other hand, doing it now means we are ready to move things down the road.
From our testing on hosted03 before we had to rebuild, the rhel5/rhel6 upgrade seemed to go pretty smoothly from all I could tell.
Any other gotchas, ideas, or things we could setup to test?
I think the timing is fine - but if hosted03 and collab03 are all up and happy now I would like to suggest we do some IO/cpu tests to beat them up a bit. I'd rather like to know that they're going to survive a fair amount of load before we spring users onto them.
-sv
Outstanding questions:
- When do we want to move the lists.fedorahosted.org? on the 12th?
the 14th? or some other time entirely? We are going to need to get collab03 able to handle the additional domain and data.
- On hosted, do we want to still change all projects to use
project.fedorahosted.org ? Or should we perhaps just wait on that change? We could do that change on a per project opt in basis as we move them to other instances, possibly with less disruption. On the other hand, doing it now means we are ready to move things down the road.
From our testing on hosted03 before we had to rebuild, the rhel5/rhel6 upgrade seemed to go pretty smoothly from all I could tell.
Any other gotchas, ideas, or things we could setup to test?
I think the timing is fine - but if hosted03 and collab03 are all up and happy now I would like to suggest we do some IO/cpu tests to beat them up a bit. I'd rather like to know that they're going to survive a fair amount of load before we spring users onto them.
Sure, I think thats quite reasonable to do.
collab03 should have data on it and be mostly synced up.
hosted03 has the data, but I need to (re) write the migration script I had on the old one. Hopefully I can have that done later today.
Any thoughts on the 2 questions above? When should we move hosted lists, and should we do our great hosted cname stuff now, or save that for a later more controlled time.
At this point I am leaning toward just migrating hosted over and dealing with the cname and other changes when we have more time/energy. I'm not sure I care when we move lists, but we could save that too if we wanted. I'd like to get moved to new hardware and rhel6 as a priority. ;)
kevin
On Tue, 6 Dec 2011 15:20:09 -0700 Kevin Fenzi kevin@scrye.com wrote:
Outstanding questions:
- When do we want to move the lists.fedorahosted.org? on the
12th? the 14th? or some other time entirely? We are going to need to get collab03 able to handle the additional domain and data.
- On hosted, do we want to still change all projects to use
project.fedorahosted.org ? Or should we perhaps just wait on that change? We could do that change on a per project opt in basis as we move them to other instances, possibly with less disruption. On the other hand, doing it now means we are ready to move things down the road.
From our testing on hosted03 before we had to rebuild, the rhel5/rhel6 upgrade seemed to go pretty smoothly from all I could tell.
Any other gotchas, ideas, or things we could setup to test?
I think the timing is fine - but if hosted03 and collab03 are all up and happy now I would like to suggest we do some IO/cpu tests to beat them up a bit. I'd rather like to know that they're going to survive a fair amount of load before we spring users onto them.
Sure, I think thats quite reasonable to do.
collab03 should have data on it and be mostly synced up.
hosted03 has the data, but I need to (re) write the migration script I had on the old one. Hopefully I can have that done later today.
Any thoughts on the 2 questions above? When should we move hosted lists, and should we do our great hosted cname stuff now, or save that for a later more controlled time.
1. I think we can have the hosted cnames be an easyfix for an fi-apprentice. Generating the full list of cnames from the entries on hosted1 should be a trivial scripting operation for someone.
2. I don't have a strong feeling on the mailing list migration timing.
-sv
Any thoughts on the 2 questions above? When should we move hosted lists, and should we do our great hosted cname stuff now, or save that for a later more controlled time.
- I think we can have the hosted cnames be an easyfix for an
fi-apprentice. Generating the full list of cnames from the entries on hosted1 should be a trivial scripting operation for someone.
Yeah, that part is not hard, however:
- Making changes to our gitweb and trac to output 'project.fedorahosted.org/scm' instead of 'git.fedorahosted.org'
- educating projects to use the new hostname instead of old.
Are likely to take more. If we shift this off until after our rhel6 move, we can pick some projects and figure out all the needed changes with them before moving everyone else.
- I don't have a strong feeling on the mailing list migration timing.
ok.
I guess my inclination is to wait and do it at the same time we migrated hosted01 or even push it out a week or so (although that gets us up against the holidays).
kevin
On 5 December 2011 12:30, Kevin Fenzi kevin@scrye.com wrote:
Greetings.
Our nice timetable of events for migrating collab01/02 and hosted01 ran into a monkey wrench the other week when we had to replace our new hardware again. We've made progress on the new new hardware and here's where we stand:
- I've installed a hosted02 on serverbeach06. This is a rhel5 clone of
hosted01. This is mostly just to make sure if anything happens to hosted01 we are still able to bring something up quick.
- I've installed a collab03 (sb08) and collab04 (sb09). I've set them
up with a drdb device for /srv to keep them in sync. The idea is that collab03 will be primary and in case anything fails, collab04 will be there to bring up quickly.
- I plan to install a hosted03 later today. (rhel6, hosted01
replacement).
I'd like to get things moved sooner rather than later. Also, we are starting to push up against the holiday season. So, I am proposing:
2011-12-12 (1 week away): collab move. (collab01 -> collab03)
2011-12-14 (next wed): hosted move. (hosted01 -> hosted03)
Outstanding questions:
- When do we want to move the lists.fedorahosted.org? on the 12th? the
14th? or some other time entirely? We are going to need to get collab03 able to handle the additional domain and data.
- On hosted, do we want to still change all projects to use
project.fedorahosted.org ? Or should we perhaps just wait on that change? We could do that change on a per project opt in basis as we move them to other instances, possibly with less disruption. On the other hand, doing it now means we are ready to move things down the road.
From our testing on hosted03 before we had to rebuild, the rhel5/rhel6 upgrade seemed to go pretty smoothly from all I could tell.
Any other gotchas, ideas, or things we could setup to test?
If no one screams on timing, I will probibly announce the collab move later today and the hosted move on wed (giving people 1 weeks notice).
No screams from me. I would say we start at 10am Mountain (12 Eastern/ 1700 UTC). That should make sure that everyone from coast to coast is awake to work on it.
I think we move the fedorahosted over and then do the great die-off. Projects that aren't getting updated.. get archived and we concentrate on what we can make work.
infrastructure@lists.fedoraproject.org