infrastructure October 2019

infrastructure@lists.fedoraproject.org

42 participants
59 discussions

Meeting Agenda Item: Introduction Orion Poplawski
by Orion Poplawski 06 Oct '19

06 Oct '19

IRC handle - orionp I have been a professional unix/linux sysadmin for over 20 years. I use ansible extensively in my job to configure a variety of systems. I have broad experience in many scripting and programming languages, though I would not consider myself an expert except for a few. I have just a bit of web programming experience including Django, but it is not my strong point. The main motivation for my wanting to join the infra team is to help with repository branching and creating, but hopefully I can find the time to help out in other areas. -- Orion Poplawski Manager of NWRA Technical Systems 720-772-5637 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion(a)nwra.com Boulder, CO 80301 https://www.nwra.com/

2 1

Nuancier, Python Fedora
by Benson Muite 06 Oct '19

06 Oct '19

Hi, We are working on Nuancier [0]. One of the current dependencies is python-fedora [1],which should contain flask_fas_openid. It seems that flask_fas_openid will remain in Python 2 and not be ported to Python 3. Will using Flask-OIDC [2] allow for full Python 3 compatibility? Regards, Benson [0] - https://fedoraproject.org/wiki/Infrastructure_2020/Friday_with_Infra [1] - https://github.com/fedora-infra/python-fedora/ [2] - https://github.com/puiterwijk/flask-oidc

2 4

CPE Team Weekly Update: 2019-10-04
by Aoife Moloney 04 Oct '19

04 Oct '19

Hi Everyone, Welcome to the CPE team weekly project update mail! Background: The Community Platform Engineering group is the Red Hat team combining IT and release engineering from Fedora and CentOS. Our goal is to keep core servers and services running and maintained, build releases, and other strategic tasks that need more dedicated time than volunteers can give. For better communication, we are giving weekly reports to the CentOS and Fedora communities about the general tasks and work being done. Fedora: Rawhide Gating: <https://github.com/fedora-infra/bodhi/projects/3> - Moving towards testing in staging with a beta release being built to deploy in staging. - Robosignatory using fedora-messaging deployed in staging. - There is an issue on documentation here that is also being addressed - Rawhide compose now fixed & some broken kde deps rebuilt - Work on staging environment set to start this week - This broke when we branched F31 in production & ticket tracked in: https://pagure.io/releng/issue/8838 - Koji storage issue was fixed and closed - Overview page of the remaining blockers and dependencies organized at: - https://hackmd.io/Gbuu9JOPR--Y2yNCBEYI5A?view - Input welcome for anything that would be missing - https://github.com/fedora-infra/bodhi/projects/3 - High priority items in the “Ready” column are all hard-dependency for pushing multi-builds to production repoSpanner <https://github.com/repoSpanner/repoSpanner> - (Another!) Race condition discovered has been fixed and tested - Test suite is more stable, but is still an ongoing work for full reliability - Performance testing is underway this week Application Handover to Community - Badges: Thread has been started to get a decision with new maintainers - Packagedb-cli: Retired this week - Pastebin: Ongoing conversations with future maintainer, expecting an update in the next 2 weeks - Elections: Move to CommuniShift underway with two blockers to fix before complete. - Fedocal: If no maintainer is found by October 18th, it will be decommissioned. - Please reach out to any of the CPE team if you want to take over this project. - Nuancier: Identified a maintainer and the team are engaging in a conversation - Whatcanidoforfedora.org: Moved to Communishift - DNS needs to be directed and a new owner is needed for it so this is open for volunteers - Documentation for onboarding contributors to Community OpenShift was started with a good mail thread on Fedora Devel Misc highlights from various parts of the ecosystem: - FPDC: No Update - Fedora Container base image released. - Merged Fedora image PR on Dockerhub https://github.com/docker-library/official-images/pull/6702, - Rawhide compose failures, due to podman gating, filed https://github.com/fedora-infra/bodhi/issues/3512 are still being worked on - F31 Final freeze will start next Tuesday, 8th October - Fedora 26 & 27 packages were archived in koji. - A discussion to define what Minimum Viable Fedora <https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapr…> looks like was started. - Please get involved! :) - Bug in Koji plugin was resolved so sending signing messages has resumed. https://pagure.io/fedora-infrastructure/issue/8158 - cpe/infra-docs repo is being removed from pagure and we are now going to use infra-docs-pagure <https://pagure.io/infra-docs> instead. Working with an active contributor in the repo to make sure there are no problems. CentOS: - The team is working with CERN about koji upgrade process for cbs.centos.org - CentOS CI - SSL Authentication issue with Fedora Messaging pluginI now solved. - Shared library being developed - 8.0.1905 docs now published on docs.centos.org. Comments & feedback are always welcome & have a great weekend! Aoife -- Aoife Moloney Feature Driver Community Platform Engineering Team Red Hat EMEA <https://www.redhat.com> Communications House Cork Road Waterford <https://www.redhat.com>

1 0

[Fedocal] Reminder meeting : Fedora Infrastructure
by smooge＠gmail.com 02 Oct '19

02 Oct '19

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2019-10-03 from 15:00:00 to 16:00:00 UTC At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/22/

1 0

Anitya (relese-monitoring.org) 0.17.2 deployed on production
by Michal Konecny 02 Oct '19

02 Oct '19

Hi everybody, Anitya (release-monitoring.org) 0.17.2 was deployed on production [0]. This release contains plenty of new features and bug fixes. See changelog for more information [1][2][3]. Few highlighted changes: - Support for multiple version prefixes - Add calendar versioning - Add semantic versioning - Allow fetching releases on GitHub backend - Hide ecosystem field for non admin users - Handle status code 403 as rate limit exception on GitHub backend - Fix for "Cannot add distro" issue On behalf of mages from release-monitoring.org, Michal [0] - release-monitoring.org [1] - https://github.com/release-monitoring/anitya/releases/tag/0.17.0 [2] - https://github.com/release-monitoring/anitya/releases/tag/0.17.1 [3] - https://github.com/release-monitoring/anitya/releases/tag/0.17.2

1 0

Meeting Agenda Item: Introduction Stephen Coady
by Stephen Coady 01 Oct '19

01 Oct '19

Hi everyone. My name is Stephen Coady and I have just joined the community platform engineering team. I have been using Fedora for about 2 years now on and off. I am looking forward to contributing to the platform in any way I can. I am familiar with Javascript, GraphQL and Docker and I have done quite a bit of hybrid mobile application development. I used Python a few years ago and I am looking forward to getting back into it. My irc nick is scoady. Looking forward to chatting to you all soon. Stephen -- Stephen Coady Software Engineer Red Hat <https://www.redhat.com/> <https://red.ht/sig>

4 3

Planned Outage - koji.fedoraproject.org - 2019-10-02 21:00 UTC
by Kevin Fenzi 01 Oct '19

01 Oct '19

There will be an outage starting at 2019-10-2 21:00 UTC, which will last approximately 1 hour. To convert UTC to your local time, take a look at https://fedoraproject.org/wiki/UTCHowto or run: date -d '2019-10-02 21:00 UTC' Reason for outage: During this outage window we plan to upgrade koji to the current 1.18.0 release, and additionally add memory to the koji database server to help improve performance. During the outage window new builds may fail to initiate, but in progress builds should complete as expected. Affected Services: koji - https://koji.fedoraproject.org/koji Ticket Link: https://pagure.io/fedora-infrastructure/issue/8265 Contact Information: Please join #fedora-admin in irc.freenode.net or add comments to the ticket for this outage above.

1 0

Starting to address backlog
by Kevin Fenzi 01 Oct '19

01 Oct '19

Greetings. A few weeks ago I went and tagged a bunch of old releng and infrastructure tickets with the 'backlog' tag. Clement added some more the other day. In our last meeting we did some simple voting on the list to determine priority. The idea is that we will take the top 1-3 per week and get small groups of people to work on them (at the very least one person can do the work and explain it to another to document/provide feedback on). Then at each meeting we look at the list, confirm what we did and take some more. Of course we have other priorities to deal with too, but focusing on a few tasks and trying to spread knowledge around them will hopefully get our backlog down over time. As a side note we are trying also with this copying these to an internal jira instance to see if we can track backlog flow better and figure out workflows, but this is purely a copy and pagure instances are where all the work and comments are done, so anyone not on the CPE team can ignore this for now. :) So, for this week we decided to work on: 8178 provision new aarch64 builders xxxxxx I have this one, it takes access to get them setup, but I am happy to explain on IRC what I am doing and how the setup works. Assistance could be used to add them to ansible, as well as documentation if there's anything special about them. I intend to work on this tomorrow morning. I'll ping everyone in #fedora-admin on this who is interested. 8157 ansible: enable ansible-report as a hook xxxxxx I'd love someone else to take this one. Any takers? I can provide pointers... 8065 Move older koji builds to archive volumes xxxxxx This is already in progress and has been for some time. I'd love to talk to others and explain how it's being done and get some documentation written up on it and others to know how to do it if I am not around. We could also use some discussion about how to split the koji volume a bit more. I might be able to work on this wed morning? Anyone interested in helping? Thanks and hopefully we can start cranking out some of this backlog now... kevin

3 4

2020 Datacenter Move: Request for comments
by Kevin Fenzi 01 Oct '19

01 Oct '19

Greetings, Fedora Infrastructure currently has the majority of its hardware in a datacenter in Arizona, USA. Red Hat leases this space for use by a number of teams, including Fedora. However, they've been seeking a more modern and cost effective location for some time and have decided on one: So, we will be migrating to a new datacenter located in Ashburn, Virginia in 2020. FESCo has approved a 2 week window for the actual move to take place ( https://pagure.io/fesco/issue/2221 ): 2020-06-01 to 2020-06-15. This window is after Fedora 32 is released, but before any major Fedora 33 Milestones. At a high level, our current plan is: * Setup the new datacenter with networking/storage/management * Populate the new datacenter with new hardware to replace old hardware that either wouldn’t survive the shipping or is due to be refreshed * Ship some small shipment of hardware from the old datacenter to the new that are not easily duplicated like signing hardware, alternative arch builders, etc. * Setup and have by the early part of the outage window a Minimum Viable Fedora Infrastructure (see below) using new hardware and some old. * Function in this minimal state as all the rest of the hardware is shipped to the new datacenter. * Re-add hardware to return to normal state. We want to maintain continuity of service as best we can, so we have defined a Minimal Viable Fedora which will move in advance of the main hardware. Our intention is to reroute traffic to this setup before moving the bulk of our hardware. Our current list of what a Minimum Viable Fedora Infrastructure is: * Mirroring fully functional. Users get metalinks, mirrors are crawled, etc * The complete package lifecycle must work. From commit to update installed on users machines. We need this to push security and important bugfixes as well as to allow maintainers to work toward Fedora 33. * Our production openshift cluster must be up and running normally. (This cluster has fas, bodhi and other important items in it) * Builders will likely be constrained. Ie, less of most arches. Capacity will be re-added as soon as the hardware for it arrives. * Rawhide composes take place as normal. * Nameservers functional * rabbitmq/fedora-messaging should be up and functional. * Internal proxies must be functional (used by builders and other internal items) * Mailing lists must be functional * Backups must be functional * OpenQA must be available to test updates/rawhide composes * Wiki must be available for common bugs / qa Other services not listed may or may not be up depending on capacity and issues with more important services. And explicitly some things will NOT be available during that window: * Staging. There will be no staging, so no rolling out new services. * Full capacity/number of builders * External proxies in the new datacenter * HA for some services. We are sending this announcement not only to let you all be aware of this move, but to help us plan. If you see some service that you think is critical to Fedora and cannot be down for 2 weeks, and isn't listed above please let us know so we can adjust our plans. We want to make sure things that are critical keep running smoothly for the Fedora community. Feedback by next friday (2019-10-04) would be welcome. Thanks, Kevin for CPE and the Fedora Infrastructure team.

6 10

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure October 2019