On Wed, Nov 27, 2019 at 08:44:15AM -0500, Neal Gompa wrote:

On Wed, Nov 27, 2019 at 3:06 AM Miroslav Suchý msuchy@redhat.com wrote:

...

Hi. For long time, I plan to do something about https://status.fedoraproject.org/

I think that manual updates of this page does not reflect the capabilities of these century.

Well, I think then we are doing a poor job of explaining what it is and how it works, or I am missing what capabilities you are wanting it to have?

We deliberately wanted it to be a manual update process. It's for showing when we are aware of some problem and are already working on it. This allows people who didn't notice the issue to know it's happening and avoid lots of people reporting the same issue. It is also deliberately not in our infra (it runs in aws).

...

My question is: do we want to go SasS way? I found https://www.statusdashboard.com/pricing which would cost us $99 per month.

Or we want to maintain it in-house? The code of this specific service is open-source https://github.com/tomalessi/statusdashboard thou there are no commits for past few years. And there are some alternatives like: https://github.com/statusdashboard/statusdashboard

Two actively maintained open source options are:

• Staytus: https://staytus.co/

• Cachet: https://cachethq.io/

Either could be customized and ran somewhere fairly easily...

I'm open to moving to some other application, just would like to know the requirements or advantages to doing so would be...

The current "app" (really just some html and js) could be prettier. It could also organize things better (by SLE perhaps). What other features would folks like?

kevin

On Thu, Nov 28, 2019 at 03:57:11PM +0100, Miroslav Suchý wrote:

Dne 28. 11. 19 v 1:29 Kevin Fenzi napsal(a):

...

I am missing what capabilities you are wanting it to have?

• current page shows "Everything is OK" even when some service does not work. I would at least expect some orange text

"There seems to be some problems, not acknowledged by admins yet".

But thats not it's purpose. If people want to know about problems as soon as monitoring sees them, can't we just point them to the nagios page?

Also, not everything monitoring notes as a problem is actually something end users would see as a problem.

• It takes pretty high rights to change something there (e.g. I cannot change status for ABRT and Copr). I understand

why it is. But I would expect to flip it from green to orange relatively easy.

Sure, we could work out some way to get you access for those...

• get rid of manual interaction as much as possible. If nagios report that the service is not available. Then take it as

granted. When nagios report the service as running, then make it green. Manually interaction should be needed only when you want to put there more information, like cause of the outage or ETA.

But how do you map those? Some examples:

* If a mirrorlist alerts as down that does not mean the service is down to end users. There's another container on each proxy. If a proxy is out of dns the alerts for services on it don't impact users. * Tons of services are setup in HA, so if one part of them is down in nagios the service is just fine. * Sometimes users notice something that isn't monitored by nagios, so nagios couldn't report it down. * A server alerts and is down, how do you know what services are affected?

So, it sounds to me like you want a better monitoring page to point people at? We have been talking about replacing nagios (for years now), and perhaps some of the other solutions have a better answer for that...

Our intent for status is "This is a list of issues some human knows about and is working on" not "something alerted, no idea if anyone knows about it or is working on it"

• Ability to show that there is going to be planned outage.

We have been using fedocal for that. There is a fedora-infrastructure outages calendar.

kevin

People are not checking status pages when everything is working :)

Sure...

IMO is is pretty fine to have something yellow (or even red) despite the fact, that it seems to operate just fine for end user.

well, I take an entry on status to mean 'it's known, we are working on fixing it' but that well may not be the case... I think it is bad to show something there that we aren't working on fixing or doesn't impact people. Either they think 'gee, you have tons of outages' or they wonder why some non impacting thing is showing and never being fixed.

It just looks bad (and not professional) when something is obviously down and status page show nice green flag.

It just means we haven't started working on it yet and want people to tell us they hit it.

Look at some other sites:

https://www.githubstatus.com/ - I know they don't update this until there is a known/widespread problem. I have several times started to see issues and looked and it was showing fine, then a few minutes later updated to reflect the outage. They do start incidents with "investigating" and have updates, which is nice. The page is pretty readable and has a small number of important things listed.

https://status.aws.amazon.com/ - they do look to push updates to there, but they also include a contact and say to report issues not listed. Personally, I think this page is too 'busy' and long...

https://status.redhat.com/ - pretty similar to the github one.

I wonder if perhaps we shouldn't drop all the services and have something more freeform and text like. ie, normally it has links to report things and such, but says "All our services are operating normally". If we start working some outage, we update it with "There is an outage of koji and bodhi and koschei due to a storage outage" (or whatever services), then update it with "storage issue is know, ETA: 1 hour to fix" or whatever.

I think that might be more friendly to users than a wall of services, many of which they might not know. It would save us from having to list everything and avoid missing some service which has problems.

Do any of the open source ones eariler in this thread support something like that?

kevin