Outage: Unplanned outage with netapp filers - 2012-01-25 10:00 UTC

There was an unplanned outage in natapp filers in the phx2 location, which lasted approximately 5 hours. Due to failover, service impacting part of outage was very small.

To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run:

date -d '2012-01-25 10:00 UTC'

Reason for outage:

One of our pair of netapp filers failed at approx 10:00UTC. This caused a failover of resources to the other netapp and no service impact was seen. When the primary node was brought back on line it attempted to take resources back, but failed to do so correctly. This lead to a outage in nfs mounts on many affected Fedora servers. The issue was corrected and node brought back on line. A ticket is open with the vendor to provide a perm fix.

Affected Services:

Bodhi - https://admin.fedoraproject.org/updates/ Docs - http://docs.fedoraproject.org/ Fedora Account System - https://admin.fedoraproject.org/accounts/ Fedora Community - https://admin.fedoraproject.org/community/ Fedora Insight - https://insight.fedoraproject.org/ Mirror List - https://mirrors.fedoraproject.org/ Mirror Manager - https://admin.fedoraproject.org/mirrormanager/ Package Database - https://admin.fedoraproject.org/pkgdb/ Smolt - http://smolts.org/ Start - http://start.fedoraproject.org/ Wiki - http://fedoraproject.org/wiki/

Unaffected Services:

Ask Fedora - http://ask.fedoraproject.org/ BFO - http://boot.fedoraproject.org/ Buildsystem - http://koji.fedoraproject.org/ GIT / Source Control DNS - ns1.fedoraproject.org, ns2.fedoraproject.org Email system Fedora Hosted - https://fedorahosted.org/ Fedora People - http://fedorapeople.org/ Main Website - http://fedoraproject.org/ QA Services Secondary Architectures Torrent - http://torrent.fedoraproject.org/

Ticket Link:

Contact Information:

Please join #fedora-admin in irc.freenode.net or add comments to the ticket for this outage above.