Greetings.
I'd like to propose we migrate pagure01 (which is pagure.io) and pagure-stg01 (pagure-stg.io) from osuosl to the community cage in rdu2.
We have had odd networking problems between .cz and osuosl and at times between phx2 and osuosl.
Moving things would allow us to:
drop the pagure-proxy in ibiblio (which currently proxies all traffic to the osuosl instances to avoid the networking issues from cz).
move forward with adding repospanner to ansible and have it appear in pagure.io so people could do PR's and CI and other fun things against our ansible repo.
Why not move it to openshift? Well, we could, but thats going to take cycles we don't have right now, so a simple migration seems much easier and gets us a lot of good.
Why not move it to phx2? Well, we would need to get a bunch of ports opened for it's use and we may well be moving resources out of phx2 at some point, so we really don't want to add another thing to move out.
Why not make a new instance in rdu2 and migrate just data to it? Again, we could, but at this point a brute force migration seems like a better use of our cycles than figuring exactly what needs synced over. If we really don't want the downtime we can look more into this, I just worry that it will take a lot of time to make sure we have everything synced and setup right.
If folks are ok with this, I'd like to consider taking it down friday late afternoon and migrating it over this weekend. It would have downtime for however long it takes to sync the disk over. I can do pagure-stg01 as a test run later this week if we like.
Thoughts?
kevin
On Tue, 9 Jul 2019 at 15:57, Kevin Fenzi kevin@scrye.com wrote:
Greetings.
I'd like to propose we migrate pagure01 (which is pagure.io) and pagure-stg01 (pagure-stg.io) from osuosl to the community cage in rdu2.
We have had odd networking problems between .cz and osuosl and at times between phx2 and osuosl.
Moving things would allow us to:
drop the pagure-proxy in ibiblio (which currently proxies all traffic to the osuosl instances to avoid the networking issues from cz).
move forward with adding repospanner to ansible and have it appear in pagure.io so people could do PR's and CI and other fun things against our ansible repo.
Why not move it to openshift? Well, we could, but thats going to take cycles we don't have right now, so a simple migration seems much easier and gets us a lot of good.
Why not move it to phx2? Well, we would need to get a bunch of ports opened for it's use and we may well be moving resources out of phx2 at some point, so we really don't want to add another thing to move out.
Why not make a new instance in rdu2 and migrate just data to it? Again, we could, but at this point a brute force migration seems like a better use of our cycles than figuring exactly what needs synced over. If we really don't want the downtime we can look more into this, I just worry that it will take a lot of time to make sure we have everything synced and setup right.
If folks are ok with this, I'd like to consider taking it down friday late afternoon and migrating it over this weekend. It would have downtime for however long it takes to sync the disk over. I can do pagure-stg01 as a test run later this week if we like.
I would say see how long it takes to move pagure-stg over then determine how long an outage it might be for the rest.
Thoughts?
kevin
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Tue, Jul 9, 2019 at 3:57 PM Kevin Fenzi kevin@scrye.com wrote:
Greetings.
I'd like to propose we migrate pagure01 (which is pagure.io) and pagure-stg01 (pagure-stg.io) from osuosl to the community cage in rdu2.
What is "community cage in rdu2"? What is "rdu2"? Where is it?
We have had odd networking problems between .cz and osuosl and at times between phx2 and osuosl.
Moving things would allow us to:
drop the pagure-proxy in ibiblio (which currently proxies all traffic to the osuosl instances to avoid the networking issues from cz).
move forward with adding repospanner to ansible and have it appear in pagure.io so people could do PR's and CI and other fun things against our ansible repo.
Why not move it to openshift? Well, we could, but thats going to take cycles we don't have right now, so a simple migration seems much easier and gets us a lot of good.
There's a community PR for Helm charts for Pagure, but I haven't gotten around to building the underlying bits for that (container images, etc.).
And even so, that wouldn't help in our case because of repoSpanner, which isn't supported by our container stuff right now.
Why not move it to phx2? Well, we would need to get a bunch of ports opened for it's use and we may well be moving resources out of phx2 at some point, so we really don't want to add another thing to move out.
Why would we be moving things out of phx2? We're not losing resources in the dc, are we?
Why not make a new instance in rdu2 and migrate just data to it? Again, we could, but at this point a brute force migration seems like a better use of our cycles than figuring exactly what needs synced over. If we really don't want the downtime we can look more into this, I just worry that it will take a lot of time to make sure we have everything synced and setup right.
This makes sense to me.
If folks are ok with this, I'd like to consider taking it down friday late afternoon and migrating it over this weekend. It would have downtime for however long it takes to sync the disk over. I can do pagure-stg01 as a test run later this week if we like.
Thoughts?
If we want to do this, we should make that announcement ASAP.
Is there a way we could do this without taking down the whole system for a long period of time? I expect it'd take a while to migrate...
-- 真実はいつも一つ！/ Always, there's only one truth!
On 7/9/19 1:52 PM, Neal Gompa wrote:
On Tue, Jul 9, 2019 at 3:57 PM Kevin Fenzi kevin@scrye.com wrote:
Greetings.
I'd like to propose we migrate pagure01 (which is pagure.io) and pagure-stg01 (pagure-stg.io) from osuosl to the community cage in rdu2.
What is "community cage in rdu2"? What is "rdu2"? Where is it?
It's at a facility in Raleigh North carolina.
We have some stuff in a small networking space (rdu1) and rdu2 (which is also the space CentOS has some gear at and ceph and other community projects. It's sometimes known as the 'community cage'.
Why would we be moving things out of phx2? We're not losing resources in the dc, are we?
There is no concrete plan I can speak of, but there have been ongoing talks about moving out of there for the last 5 years. I think RDU is a better place for pagure at this point.
If we want to do this, we should make that announcement ASAP.
Yeah, we could also push it off to next weekend.
Is there a way we could do this without taking down the whole system for a long period of time? I expect it'd take a while to migrate...
Not without redeploying and syncing just part of the data we need.
I can test and get better numbers with pagure-stg01.
kevin
On Tue, Jul 09, 2019 at 12:09:40PM -0700, Kevin Fenzi wrote:
Greetings.
I'd like to propose we migrate pagure01 (which is pagure.io) and pagure-stg01 (pagure-stg.io) from osuosl to the community cage in rdu2.
We have had odd networking problems between .cz and osuosl and at times between phx2 and osuosl.
Moving things would allow us to:
drop the pagure-proxy in ibiblio (which currently proxies all traffic to the osuosl instances to avoid the networking issues from cz).
move forward with adding repospanner to ansible and have it appear in pagure.io so people could do PR's and CI and other fun things against our ansible repo.
Why not move it to openshift? Well, we could, but thats going to take cycles we don't have right now, so a simple migration seems much easier and gets us a lot of good.
Why not move it to phx2? Well, we would need to get a bunch of ports opened for it's use and we may well be moving resources out of phx2 at some point, so we really don't want to add another thing to move out.
Why not make a new instance in rdu2 and migrate just data to it? Again, we could, but at this point a brute force migration seems like a better use of our cycles than figuring exactly what needs synced over. If we really don't want the downtime we can look more into this, I just worry that it will take a lot of time to make sure we have everything synced and setup right.
If folks are ok with this, I'd like to consider taking it down friday late afternoon and migrating it over this weekend. It would have downtime for however long it takes to sync the disk over. I can do pagure-stg01 as a test run later this week if we like.
Big +1 on the general idea, being able to get ride of these odd networking issues would be very nice!
I agree that we may want to start with the staging instance to see how long it takes. The number of git repos containing a lot of small files will slow things down a bit, but at least staging should give us some clues as to how smooth such move will be.
If we want to do the move of production this week-end, we likely need to announce this sooner rather than later.
Finally, just a note that staging pagure is already running 5.7 while production is still on 5.5. This shouldn't impact the move itself, but might be worth keeping on the back of our minds if we end up upgrading somehow (5.6 has a DB schema change, so it will blow up if you upgrade from 5.5).
Thanks for looking into this :) Pierre
infrastructure@lists.fedoraproject.org
-
Kevin Fenzi -
Neal Gompa -
Pierre-Yves Chibon -
Stephen John Smoogen