There is currently a problem with the networking of various systems in our Phoenix data-center which houses most of our build systems. The problem has knocked off most of our x86_64 build systems and several of our aarch64/arm systems. Builds will take longer to process during this time and may queue up for the remaining builders.
There is currently no ETA for when recovery will happen but due to end of the year schedules, it will not happen until early January. If things worsen we will see what escalation paths or changes can be done.
If there's anything i can do remotely count with me.
Ivan
On Fri, Dec 21, 2018 at 9:04 AM Stephen John Smoogen smooge@gmail.com wrote:
There is currently a problem with the networking of various systems in our Phoenix data-center which houses most of our build systems. The problem has knocked off most of our x86_64 build systems and several of our aarch64/arm systems. Builds will take longer to process during this time and may queue up for the remaining builders.
There is currently no ETA for when recovery will happen but due to end of the year schedules, it will not happen until early January. If things worsen we will see what escalation paths or changes can be done.
-- Stephen J Smoogen. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On 12/21/18 7:03 AM, Stephen John Smoogen wrote:
There is currently a problem with the networking of various systems in our Phoenix data-center which houses most of our build systems. The problem has knocked off most of our x86_64 build systems and several of our aarch64/arm systems. Builds will take longer to process during this time and may queue up for the remaining builders.
There is currently no ETA for when recovery will happen but due to end of the year schedules, it will not happen until early January. If things worsen we will see what escalation paths or changes can be done.
With the help of Networking folks we have managed to recover network access to all the builders. While the underlying problem still exists and will require an outage to track down and fix, we are for now at least all back to normal on builders.
Happy holidays to everyone.
kevin
EPEL7 builds for x86_64 and ppc64le appear to be failing still. Example:
https://koji.fedoraproject.org/koji/taskinfo?taskID=31577177
Just FYI.
On 12/22/18 1:58 PM, Bojan Smojver wrote:
EPEL7 builds for x86_64 and ppc64le appear to be failing still. Example:
https://koji.fedoraproject.org/koji/taskinfo?taskID=31577177
Just FYI.
Thanks for reporting it (although a ticket might do better next time).
I think I have this fixed, can you resubmit your build(s) and let me know if it worked or not?
kevin
Thanks Kevin. The build worked!
Sorry about not opening a ticket. That's done here, right? https://pagure.io/fedora-infrastructure
On Sun, Dec 23, 2018 at 02:28:38PM +1100, Bojan Smojver wrote:
Thanks Kevin. The build worked! Sorry about not opening a ticket. That's done here, right? https://pagure.io/fedora-infrastructure
Correct :)
Pierre
On 12/21/18 4:03 PM, Stephen John Smoogen wrote:
There is currently a problem with the networking of various systems in our Phoenix data-center which houses most of our build systems. The problem has knocked off most of our x86_64 build systems and several of our aarch64/arm systems. Builds will take longer to process during this time and may queue up for the remaining builders.
There is currently no ETA for when recovery will happen but due to end of the year schedules, it will not happen until early January. If things worsen we will see what escalation paths or changes can be done.
Not sure if this is related, but COPR repositories seem to be inaccessible. I filed https://pagure.io/fedora-infrastructure/issue/7460 (I hope this is the right place for the issue)
Kind regards & happy holidays Till
On 12/23/18 3:37 AM, Till Hofmann wrote:
On 12/21/18 4:03 PM, Stephen John Smoogen wrote:
There is currently a problem with the networking of various systems in our Phoenix data-center which houses most of our build systems. The problem has knocked off most of our x86_64 build systems and several of our aarch64/arm systems. Builds will take longer to process during this time and may queue up for the remaining builders.
There is currently no ETA for when recovery will happen but due to end of the year schedules, it will not happen until early January. If things worsen we will see what escalation paths or changes can be done.
Not sure if this is related, but COPR repositories seem to be inaccessible. I filed https://pagure.io/fedora-infrastructure/issue/7460 (I hope this is the right place for the issue)
Nope, unrelated completely... you can file copr tickets there, but we don't maintain copr. Since it was just a unresponsive node this time I was able to bring it back up, but if it was anything more complex you would need to let the copr folks know, probibly at:
https://bugzilla.redhat.com/enter_bug.cgi?product=Copr
I realize this is all confusing and hopefully we can come up with a better way for people to know where to report things for which service and what to expect. It's not an easy problem however. ;(
kevin
On 12/23/18 7:20 PM, Kevin Fenzi wrote:
On 12/23/18 3:37 AM, Till Hofmann wrote:
On 12/21/18 4:03 PM, Stephen John Smoogen wrote:
There is currently a problem with the networking of various systems in our Phoenix data-center which houses most of our build systems. The problem has knocked off most of our x86_64 build systems and several of our aarch64/arm systems. Builds will take longer to process during this time and may queue up for the remaining builders.
There is currently no ETA for when recovery will happen but due to end of the year schedules, it will not happen until early January. If things worsen we will see what escalation paths or changes can be done.
Not sure if this is related, but COPR repositories seem to be inaccessible. I filed https://pagure.io/fedora-infrastructure/issue/7460 (I hope this is the right place for the issue)
Nope, unrelated completely... you can file copr tickets there, but we don't maintain copr. Since it was just a unresponsive node this time I was able to bring it back up
Thanks for fixing it so quickly!
but if it was anything more complex you would need to let the copr folks know, probibly at:
https://bugzilla.redhat.com/enter_bug.cgi?product=Copr
I realize this is all confusing and hopefully we can come up with a better way for people to know where to report things for which service and what to expect. It's not an easy problem however. ;(
OK, I was actually wondering exactly that, I remembered that COPR is maintained/run by someone else. But on the Fedora Infrastructure Status page [1], there is also a COPR status, and the link for outages directly leads to the fedora-infrastructure tracker, that's why I thought that's the right place. I guess it may already help if the status page linked to the right trackers for each component?
Is status.fp.o still actually used? There is at least one commit from April 2017 that still hasn't been deployed.
Kind regards, Till
On 12/24/18 4:24 AM, Till Hofmann wrote:
OK, I was actually wondering exactly that, I remembered that COPR is maintained/run by someone else. But on the Fedora Infrastructure Status page [1], there is also a COPR status, and the link for outages directly leads to the fedora-infrastructure tracker, that's why I thought that's the right place. I guess it may already help if the status page linked to the right trackers for each component?
Yes, or (as we have planned but not yet done, reorganize status to note different sections of things based on their SLE).
Is status.fp.o still actually used? There is at least one commit from April 2017 that still hasn't been deployed.
It is. Where are you seeing that?
The repo is:
https://github.com/fedora-infra/statusfpo/commits/master
last commit dec 18th, 2018.
kevin
On 1/4/19 6:38 PM, Kevin Fenzi wrote:
On 12/24/18 4:24 AM, Till Hofmann wrote:
OK, I was actually wondering exactly that, I remembered that COPR is maintained/run by someone else. But on the Fedora Infrastructure Status page [1], there is also a COPR status, and the link for outages directly leads to the fedora-infrastructure tracker, that's why I thought that's the right place. I guess it may already help if the status page linked to the right trackers for each component?
Yes, or (as we have planned but not yet done, reorganize status to note different sections of things based on their SLE).
Is status.fp.o still actually used? There is at least one commit from April 2017 that still hasn't been deployed.
It is. Where are you seeing that?
The repo is:
https://github.com/fedora-infra/statusfpo/commits/master
last commit dec 18th, 2018.
kevin
Ah, I was looking at the pagure repo. status.fp.o still points to fedorahosted.org in the "Open Source" link at the bottom. It seems like this commit didn't make it into GitHub: https://pagure.io/fedora-status/c/e4009d65586f9ebe8573a6148acd7f4f1816959e?b...
Kind regards Till
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel@lists.stg.fedoraproject.org
-
Bojan Smojver -
Ivan Chavero -
Kevin Fenzi -
Pierre-Yves Chibon -
Stephen John Smoogen -
Till Hofmann