infrastructure May 2018

infrastructure@lists.fedoraproject.org

36 participants
36 discussions

openshift outages this week (both stg and prod!)
by Kevin Fenzi 29 May '18

29 May '18

Greetings. I'd like to have a outage for staging openshift tomorrow starting at 21UTC and running hopefully only an hour or so. I want to reinstall it and try and enable some new features we want to work with (cri-o containers, some storage for registery). Is that ok for all staging openshift users? Sorry for the short notice. Then, I'd like to have an outage for our production openshift on thursday starting at 21UTC. I'd like to reinstall it also from the ground up. It's currently 3.6 and the vm's are not sized correctly for the newer versions. I had hoped I could take each vm out, reinstall it and then readd it, but once I got to the master nodes it started to become too much hassle. I'd like to just reinstall it with 3.9 and reinstall all apps via their playbooks. I'll file the outages in a few and try and @ as many folks as I think might be interested in this, but if you have any issues, please chime in here or there. Thanks! kevin

1 0

Fedora Infrastructure Meeting 2018-05-23 1400 UTC Agenda
by Stephen John Smoogen 23 May '18

23 May '18

This shared document is for the next fedora infrastructure meeting. = Preamble = The infrastructure team will be having its weekly meeting tomorrow, 2018-05-24 at 14:00 UTC in #fedora-meeting-1 on the freenode network. We have a gobby document (see: https://fedoraproject.org/wiki/Gobby ) fedora-infrastructure-meeting-next is the document. Please try and review and edit that document before the meeting and we will use it to have our agenda of things to discuss. A copy as of today is included in this email. If you have something to discuss, add the topic to the discussion area with your name. If you would like to teach other folks about some application or setup in our infrastructure, please add that topic and your name to the learn about section. = Introduction = We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. = Meeting start stuff = #startmeeting Infrastructure (2018-05-24) #meetingname infrastructure #topic aloha #chair smooge relrod nirik pingou puiterwijk tflink = Let new people say hello = #topic New folks introductions #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves = Status / Information / Trivia / Announcements = (We put things here we want others on the team to know, but don't need to discuss) (Please use #info <the thing> - your name) #topic announcements and information #info more hardware installed #info relrod PTO 9 Jun - 19 Jun #info smooge PTO 8 Jun #info Problem with mirrormanager and newest rsync found/fixed #info armv7 Builders downgraded to fedora 27 - kevin #info = Things we should discuss = We use this section to bring up discussion topics. Things we want to talk about as a group and come up with some consensus /suor decision or just brainstorm a problem or issue. If there are none of these we skip this section. (Use #topic your discussion topic - your username) #topic Oncall reminder and discussion - kevin #topic Tickets discussion #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=2 Go thru each ticket one by one = Apprentice office hours = #topic Apprentice Open office minutes #info A time where apprentices may ask for help or look at problems. Here we will discuss any apprentice questions, try and match up people looking for things to do with things to do, progress, testing anything like that. = Learn about some application or setup in infrastructure = (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the i/nfo in this section. In the event we don't find someone to teach about something, we skip this section and just move on to open floor.) #info none this week = Meeting end stuff = #topic Open Floor #endmeeting -- Stephen J Smoogen.

1 0

[Fedocal] Reminder meeting : Bodhi stakeholder's meeting
by bowlofeggs＠fedoraproject.org 21 May '18

21 May '18

Dear all, You are kindly invited to the meeting: Bodhi stakeholder's meeting on 2018-05-22 from 15:00:00 to 16:00:00 UTC At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Source: https://apps.fedoraproject.org/calendar/meeting/4671/

1 0

Meeting Agenda Item: Introduction afdez
by Alexis Fernández 21 May '18

21 May '18

Hello, everyone, I'm going to introduce myself according to the Fedora guide to start contributing to this awesome project: - IRC nick: afdez. - Skills: - Programming: Bash, PHP, SQL, Java, HTML, - Sysadmin: Databases, Networks, Unix Server Administration, Centos, FreeBSD, FreeNas, Fedora, Arch Linux. Certs related: MTCNA, LPIC1. - Want to learn: Python, security related topics, sysadmin related topics. - I own different systems to test, workstation, servers and core versions. (Laptops, desktops and Rpis). - I have experience in professional software development for 8 years and the last 3 as Linux system administrator. - I'm willing to help in any area you think I can be of use. Thanks in advance.

1 0

Modularity WG + f2 + Infra
by Randy Barlow 18 May '18

18 May '18

Greetings! At the hackathon we did recently, we talked about the need for one of us to attend the Modularity WG meetings so we could be more aware of what work is coming our way before it's a surprise, and I volunteered to be that person. For $reasons, today was the first of the WG meetings I was able to attend. There were two items of interest: New service =========== I learned that Factory 2 is developing a new service called Ursa Major. I gathered that its mission is to make it so that RPMs that depend on modules can get those modules pulled into their buildroot. Ralph, is the above correct? If so, what time frame do you anticipate us needing to get this service deployed to infrastructure? Meetings ======== The modularity WG suggested that it was probably not that beneficial for me to attend their meetings as an infra liason, as they are not actually particularly familiar with the infrastructure side of things. They suggested instead that we interface with Factory 2. A suggestion was made that perhaps we could have an agenda item on *our* meeting, perhaps once a month, where we invite modularity WG and factory 2 to talk with us about what is coming our way, or about issues that need dealing with. I thought this was a fine suggestion, so I wanted to ask you all what you thought about it?

5 4

Factory2/Infra sync up meeting - May 2018
by Pierre-Yves Chibon 18 May '18

18 May '18

Greeting, Last Monday Ralph and I had our monthly meeting for Factory2/Infra sync up. We went over the event that lead to disabling gating in bodhi last week (see fesco ticket: https://pagure.io/fesco/issue/1872) If this situation happens again we encourage a stronger reach out for help from the folks involved (in this case Ralph or Dan Callaghan (dcallagh) who hangs on #fedora-admin and was around while the discussion was happening), by stronger we are putting down a phone call or a text as valid means. We talked about PDC, as you may recall PDC is being decomissioned upstream leaving us with no maintainer upstream. Since the team is currently quite busy with other priorities we agreed to restart this discussion in early June with all parties who expressed interest on the thread on this list. Finally, Ralph explained the new workflow they put in place to deploy their images in our openshift. This is all documented in: https://pagure.io/infra-docs/pull-request/109 Folks are welcomed to take a look at it and comment with their thoughts on the process. In short:images are built and hosted in quay, the latest tag is automatically added if the image passes testing, openshift in staging is configured to pull from the latest tag. Then a "prod" tag can be manually added to an images that openshift will pull in prod. @patrick: with your security officer hat, does this sound good to you ^? If not, the Factory crew is willing to look at other options. Hoping this is helpful, Pierre

2 3

Bodhi 3.8.0b0
by Randy Barlow 17 May '18

17 May '18

Greetings! I have deployed bodhi-3.8.0b0[0] to staging[1]. It's mostly a feature release. I do have one critical bugfix[2] that I plan to include in 3.8.0 so this is not the final beta, but I had built this beta before I had the decision to include that bug so I figured I'd go ahead and get it deployed. Happy testing! [0] https://bodhi.stg.fedoraproject.org/docs/user/release_notes.html [1] https://bodhi.stg.fedoraproject.org/ [2] https://github.com/fedora-infra/bodhi/issues/2368

1 0

Re: IRC oncall
by Zach Villers 17 May '18

17 May '18

Well, we have talked about mentoring in the past, but we run into issues with it. It's hard for someone just learning to spend tons of time being mentored and we can't (and don't want to!) force them to either. So we don't know who is going to be around for a while or who has time. Not what folks are asking, but you can pick up a lot just by hanging out and watching the on-call person mutter to themselves in IRC ( not sure that is the right phrase umm... ) I don't know if this will translate for everyone, but there is a concept of "rubber duck" problem solving, where if you have a particularly difficult issue, and you explain it to someone, it helps you solve the problem more easily. I don't know if this is how everyone works and it really doesn't help if someone is jumping up and down and quacking while your are trying to think. I guess my point is, just hanging out unobtrusively when you can is fairly helpful all around. do some kind of intensive mentoring> I wonder what it would be like if there were an apprentice slot for each oncall shift? For example, an apprentice is scheduled to be paged with the oncall sysadmin, then at the least can shadow the syasadmin, help with communication on IRC, do initial troubleshooting & monitoring with the apprentice auth/access level, etc. If I recall, alerts are pretty easily accessible. You can poke around on Nagios if there are issues. Obviously if everything is down/red, it's not a good time to ask for help with your ssh access. Well, depends on the stuff I guess. It means that the oncall person not only has to watch for and respond to pings on irc and triage tickets, but then explain/teach a apprentice. If there is time and willing apprentice thats great! If things are really busy, it could be too much work to do at once. I'm open to ideas here... we should all try and figure better ways to get stuff done and learn and have a good time doing it. :) kevin A couple ideas; - stream your terminal session when working an outage ( could be hard to find a 100% foss version that is secure ) - plan some outages in stage for apprentices to work at some time when tickets are low and nothing urgent is planned ( I don't know that I've ever heard of such a time, but in theory it could exist ) Of course, it's late here, so this may all turn out to be nonsense, but good discussion anyway. -Zach #aikidouke

3 2

Fedora Infrastructure Meeting 2018-05-17 1400 UTC Agenda
by Stephen John Smoogen 16 May '18

16 May '18

This shared document is for the next fedora infrastructure meeting. Please update the Gobby or read it at https://gobby.fedoraproject.org/cgit/infinote/tree/fedora-infrastructure-me… = Preamble = The infrastructure team will be having its weekly meeting tomorrow, 2018-05-17 at 14:00 UTC in #fedora-meeting-1 on the freenode network. We have a gobby document (see: https://fedoraproject.org/wiki/Gobby ) fedora-infrastructure-meeting-next is the document. Please try and review and edit that document before the meeting and we will use it to have our agenda of things to discuss. A copy as of today is included in this email. If you have something to discuss, add the topic to the discussion area with your name. If you would like to teach other folks about some application or setup in our infrastructure, please add that topic and your name to the learn about section. = Introduction = We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. = Meeting start stuff = #startmeeting Infrastructure (2018-05-17) #meetingname infrastructure #topic aloha #chair smooge relrod nirik pingou puiterwijk tflink = Let new people say hello = #topic New folks introductions #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves = Status / Information / Trivia / Announcements = (We put things here we want others on the team to know, but don't need to discuss) (Please use #info <the thing> - your name) #topic announcements and information #info relrod moving hosts to new boxes #info relrod PTO 9 Jun - 19 Jun #info Pagure updated #info Aarch64 Builders downgraded to fedora 27 - kevin #info = Things we should discuss = We use this section to bring up discussion topics. Things we want to talk about as a group and come up with some consensus /suor decision or just brainstorm a problem or issue. If there are none of these we skip this section. (Use #topic your discussion topic - your username) #topic Tickets discussion #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=2 Go thru each ticket one by one = Apprentice office hours = #topic Apprentice Open office minutes #info A time where apprentices may ask for help or look at problems. Here we will discuss any apprentice questions, try and match up people looking for things to do with things to do, progress, testing anything like that. = Learn about some application or setup in infrastructure = (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the i/nfo in this section. In the event we don't find someone to teach about something, we skip this section and just move on to open floor.) #info none this week = Meeting end stuff = #topic Open Floor #endmeeting -- Stephen J Smoogen.

1 0

IRC oncall
by Kevin Fenzi 15 May '18

15 May '18

Greetings everyone. As those of you who frequent IRC probably already know, we are trying out having a Fedora Infrastructure "oncall" person for IRC reuqets/queries/pings. You can find out who this is by asking zodbot: zodbot: oncall smooge (Stephen Smoogen) is oncall. If I do not respond please file a ticket and then we would really appreciate it if you directed your IRC ping to the nick listed in oncall or (as the reply says) just file a ticket. Why are we doing this? Well, there are 4 of us who are full time paid sysadmin/ops folks for Fedora Infrastructure and we were finding that a large portion of time for everyone was being taken up by IRC interrupts. Sometimes interrupting multiple people for non urgent tasks. We could of course just ignore IRC, but that prevents us from handling urgent interrupts where we do want to look at something quickly. Additionally, when the oncall person doesn't know how to solve or handle some issue, they can talk to whoever does know and learn how to do it and document it. With this setup, the oncall person can plan on watching IRC and everyone else can focus on tasks that require more concentration. If your issue is not time sensitive or urgent, do also just consider filing a ticket and we will triage it and work it as soon as we are able. Thanks in advance for helping us out here. kevin

4 4

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure May 2018