Hi all,
Last night we have tried to update the firmware of the SAN backing the
bigger persistent volumes in the Fedora Infra Cloud.
While the update itself went without a hitch, there have been some
issues with the failover to the backup controller, and for
this reason we forced failover back to the original controller, which
restored communication.
Unfortunately, we found out that in some cases OpenStack decided to
mark the volumes as readonly once they came back
up. If this is still affecting your instance, please let us know, but
we have fixed all the instances that we could determine had
this issue.
We will be looking into the secondary controller status to find why
the failover didn't work, and we will be retrying the failover
again sometime soon, for which an outage notification will be sent to this list.
Regards,
Patrick Uiterwijk