We need to do the bit flip sooner. The announcement is out, I've sent over 600 requests and not a single one has come back a success. I know work is being done to change this whole mechanism now, but we still should be flipping the bit sooner.
-Mike
On Tue, 28 Apr 2009, Mike McGrath wrote:
We need to do the bit flip sooner. The announcement is out, I've sent over 600 requests and not a single one has come back a success. I know work is being done to change this whole mechanism now, but we still should be flipping the bit sooner.
I flipped the bits at: 8:30am EDT. That was a bit earlier than before.
-sv
On Tue, 28 Apr 2009, Seth Vidal wrote:
On Tue, 28 Apr 2009, Mike McGrath wrote:
We need to do the bit flip sooner. The announcement is out, I've sent over 600 requests and not a single one has come back a success. I know work is being done to change this whole mechanism now, but we still should be flipping the bit sooner.
I flipped the bits at: 8:30am EDT. That was a bit earlier than before.
I'm thinking more like hours. From the last release I monitored:
http://mmcgrath.fedorapeople.org/alphaMirrorRediness.html
We didn't hit 80% success rate until 3 and a half hours after the launch. I think mdomsch made some good changes since this graph that will make the outcome better, but I still think that initial launch isn't right.
Perhaps we should wait to send the announcement out until a certain % of mirrors are ready? We currently don't check that prior to announce.
-Mike
On Tue, 28 Apr 2009, Mike McGrath wrote:
I'm thinking more like hours. From the last release I monitored:
http://mmcgrath.fedorapeople.org/alphaMirrorRediness.html
We didn't hit 80% success rate until 3 and a half hours after the launch. I think mdomsch made some good changes since this graph that will make the outcome better, but I still think that initial launch isn't right.
Perhaps we should wait to send the announcement out until a certain % of mirrors are ready? We currently don't check that prior to announce.
I went by Jesse's wording of "make sure at least a few mirrors are synced" and you said on irc that 3 mirrors in the US were synced.
That's why I sent out the announcement then.
That may be where the mixup was.
-sv
On Tue, 28 Apr 2009, Seth Vidal wrote:
On Tue, 28 Apr 2009, Mike McGrath wrote:
I'm thinking more like hours. From the last release I monitored:
http://mmcgrath.fedorapeople.org/alphaMirrorRediness.html
We didn't hit 80% success rate until 3 and a half hours after the launch. I think mdomsch made some good changes since this graph that will make the outcome better, but I still think that initial launch isn't right.
Perhaps we should wait to send the announcement out until a certain % of mirrors are ready? We currently don't check that prior to announce.
I went by Jesse's wording of "make sure at least a few mirrors are synced" and you said on irc that 3 mirrors in the US were synced.
That's why I sent out the announcement then.
That may be where the mixup was.
Synced but if you clicked on those mirrors they threw a forbidden message.
-Mike
On Tue, Apr 28, 2009 at 10:18:47AM -0500, Mike McGrath wrote:
On Tue, 28 Apr 2009, Seth Vidal wrote:
On Tue, 28 Apr 2009, Mike McGrath wrote:
I'm thinking more like hours. From the last release I monitored:
http://mmcgrath.fedorapeople.org/alphaMirrorRediness.html
We didn't hit 80% success rate until 3 and a half hours after the launch. I think mdomsch made some good changes since this graph that will make the outcome better, but I still think that initial launch isn't right.
Perhaps we should wait to send the announcement out until a certain % of mirrors are ready? We currently don't check that prior to announce.
I went by Jesse's wording of "make sure at least a few mirrors are synced" and you said on irc that 3 mirrors in the US were synced.
That's why I sent out the announcement then.
That may be where the mixup was.
Synced but if you clicked on those mirrors they threw a forbidden message.
Backwards math:
Time to generate and publish updated mirrorlist: 20 minutes Time to crawl all mirrors: ~2 hours Time for most mirrors to rsync and catch the bitflip: 6 hours Time to sync bitflip to all three netapps: < 30 minutes (maybe <<). Time to do bitflip: 1 minute
So Mike is correct, that if we bitflip 2.5+ hours ahead of 1400 UTC, that by 1400 UTC we'll know which mirrors have actually picked up the bitflip and are live. If we bitflip a lot earlier, say 6.5 hours ahead, we can have nearly all the mirrors with the right permissions, and showing right on the mirrorlist.
MM is also "bad" in that it doesn't know the directory permissions on every directory that a report_mirror-using mirror has. So we're showing mirrors in the publiclist, and returning them to clients on with mirrorlist/d.fp.o when we don't know that their permissions are correct. Adding that knowledge to report_mirror and MM would prevent us from returning pre-bitflip mirrors to clients. That's a good bit of work I haven't scoped.
On Tue, Apr 28, 2009 at 10:34:51AM -0500, Matt Domsch wrote:
Backwards math:
Time to generate and publish updated mirrorlist: 20 minutes Time to crawl all mirrors: ~2 hours Time for most mirrors to rsync and catch the bitflip: 6 hours
Insert step: Time for update-master-directory-list to catch the bitflip: 30 minutes
Time to sync bitflip to all three netapps: < 30 minutes (maybe <<). Time to do bitflip: 1 minute
Shortest time through the path is: 1+30+30+120+20, or nearly 3.5 hours, which lines up well with Mike's observation that 3.5 hours was needed before we stopped sending people to closed mirrors.
So, I'd like to propose we back up the bitflip by at least 4 hours from release time, perhaps as much as 6.
I'd also like to move the update-mirrorlist cronjob back to start at :40 past the hour, so content is in place by :00.
On Tue, 2009-04-28 at 13:16 -0500, Matt Domsch wrote:
So, I'd like to propose we back up the bitflip by at least 4 hours from release time, perhaps as much as 6.
Hrm. This means we have unlocked mirrors for up to 6 hours before we make the announcement. This could lead to a lot of confusion and uncertainty about the isos and their validity, something we see whenever a mirror leaks early.
Honestly I think we need a vastly different way of getting mirrors to bit flip aside from waiting on random cron jobs to pick up the change.
On Wed, Apr 29, 2009 at 2:18 PM, Jesse Keating jkeating@redhat.com wrote:
On Tue, 2009-04-28 at 13:16 -0500, Matt Domsch wrote:
So, I'd like to propose we back up the bitflip by at least 4 hours from release time, perhaps as much as 6.
Hrm. This means we have unlocked mirrors for up to 6 hours before we make the announcement. This could lead to a lot of confusion and uncertainty about the isos and their validity, something we see whenever a mirror leaks early.
Honestly I think we need a vastly different way of getting mirrors to bit flip aside from waiting on random cron jobs to pick up the change.
-- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating
Is there any way to make the bit flip a push operation instead of a pull?
---Brett.
On Wed, 29 Apr 2009, brett lentz wrote:
On Wed, Apr 29, 2009 at 2:18 PM, Jesse Keating jkeating@redhat.com wrote:
On Tue, 2009-04-28 at 13:16 -0500, Matt Domsch wrote:
So, I'd like to propose we back up the bitflip by at least 4 hours from release time, perhaps as much as 6.
Hrm. This means we have unlocked mirrors for up to 6 hours before we make the announcement. This could lead to a lot of confusion and uncertainty about the isos and their validity, something we see whenever a mirror leaks early.
Honestly I think we need a vastly different way of getting mirrors to bit flip aside from waiting on random cron jobs to pick up the change.
-- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating
Is there any way to make the bit flip a push operation instead of a pull?
We've been discussing options on this, everwhere from push, to trigger a pull to pull less but do it more often.
-Mike
On Wed, 29 Apr 2009, Jesse Keating wrote:
On Tue, 2009-04-28 at 13:16 -0500, Matt Domsch wrote:
So, I'd like to propose we back up the bitflip by at least 4 hours from release time, perhaps as much as 6.
Hrm. This means we have unlocked mirrors for up to 6 hours before we make the announcement. This could lead to a lot of confusion and uncertainty about the isos and their validity, something we see whenever a mirror leaks early.
That's certainly a downside. But I'd generally consider that far less bad then an announcement going out and having so few mirrors actually be ready for it.
Honestly I think we need a vastly different way of getting mirrors to bit flip aside from waiting on random cron jobs to pick up the change.
We do and are, but bit flipping sooner is an easy thing we can do now to make a more smooth transition. I think this is one of those don't let the perfect get in the way of the good situations.
-Mike
On Tue, Apr 28, 2009 at 10:53:23AM -0400, Seth Vidal wrote:
On Tue, 28 Apr 2009, Mike McGrath wrote:
I'm thinking more like hours. From the last release I monitored:
http://mmcgrath.fedorapeople.org/alphaMirrorRediness.html
We didn't hit 80% success rate until 3 and a half hours after the launch. I think mdomsch made some good changes since this graph that will make the outcome better, but I still think that initial launch isn't right.
Perhaps we should wait to send the announcement out until a certain % of mirrors are ready? We currently don't check that prior to announce.
I went by Jesse's wording of "make sure at least a few mirrors are synced" and you said on irc that 3 mirrors in the US were synced.
That's why I sent out the announcement then.
That may be where the mixup was.
It didn't help that I had to stick my big fat nose in and "help" by approving the message in the moderation queue. Sorry I contributed to the angst. :-(
infrastructure@lists.fedoraproject.org