infrastructure April 2020

infrastructure@lists.fedoraproject.org

37 participants
54 discussions

infra ticket stats
by Kevin Fenzi 15 Apr '20

15 Apr '20

Just FYI, I gathered some real simple stats on our infrastructure tickets. Just counting the number opened and the number closed per month. So this doesn't include existing tickets, just new and closed. And of course some tickets are easier than others, etc. open | closed 2018-01: 57 | 42 2018-02: 98 | 85 2018-03: 65 | 44 2018-04: 92 | 105 2018-05: 113 | 96 2018-06: 67 | 60 2018-07: 52 | 66 2018-08: 68 | 85 2018-09: 72 | 99 2018-10: 77 | 76 2018-11: 81 | 66 2018-12: 61 | 54 2019-01: 81 | 84 2019-02: 108 | 59 2019-03: 117 | 85 2019-04: 103 | 96 2019-05: 123 | 132 2019-06: 72 | 70 2019-07: 94 | 111 2019-08: 97 | 118 2019-09: 113 | 118 2019-10: 102 | 103 2019-11: 103 | 84 2019-12: 90 | 78 2020-01: 94 | 80 2020-02: 137 | 115 2020-03: 141 | 126 2020-04: 29 | 51 (so far) I think we are improving our close rate, but also incoming rate is increasing (which may be due to us asking everyone to actually file tickets instead of pinging us, etc). Also, we can see we handle usually around 100 tickets a month, but have been increasing that rate too. Just thought I would toss it out there to see if there's anything we can learn from it. :) kevin

2 1

[FBR] Add kerneltest.fp.o reverse proxy + post-fact FBR
by Pierre-Yves Chibon 14 Apr '20

14 Apr '20

Good Morning, While working on https://pagure.io/fedora-infrastructure/issue/8035 to help Kevin I ended up adding the following commit to the proxies and running the proxy playbook for it: commit d6fe2e08a6897470cdaf3ecd191c5329dca49b3c Author: Pierre-Yves Chibon <pingou(a)pingoured.fr> Date: Tue Apr 14 11:25:17 2020 +0200 proxies-website: Add kerneltest to the proxies Signed-off-by: Pierre-Yves Chibon <pingou(a)pingoured.fr> diff --git a/playbooks/include/proxies-websites.yml b/playbooks/include/proxies-websites.yml index a18f7a3a3..5159ecdff 100644 --- a/playbooks/include/proxies-websites.yml +++ b/playbooks/include/proxies-websites.yml @@ -996,6 +996,13 @@ cert_name: "{{wildcard_cert_name}}" tags: calendar + - role: httpd/website + site_name: kerneltest.fedoraproject.org + sslonly: true + server_aliases: [kerneltest.stg.fedoraproject.org] + cert_name: "{{wildcard_cert_name}}" + tags: kerneltest + # fedorahosted is retired. We have the site here so we can redirect it. - role: httpd/website I had totally forgot proxies were frozen, so my apologies for pushing this through and I'd like to ask for a post-fact FBR. With this I got kerneltest mostly working expect that it doesn't send back the expected html page. Clément took a look at it and pointed me to the reverse proxy which haven't been configured for kerneltest. So I'd like to apply the following patch and run the corresponding playbook: commit d6fe2e08a6897470cdaf3ecd191c5329dca49b3c Author: Pierre-Yves Chibon <pingou(a)pingoured.fr> Date: Tue Apr 14 11:25:17 2020 +0200 proxies-website: Add kerneltest to the proxies Signed-off-by: Pierre-Yves Chibon <pingou(a)pingoured.fr> diff --git a/playbooks/include/proxies-websites.yml b/playbooks/include/proxies-websites.yml index a18f7a3a3..5159ecdff 100644 --- a/playbooks/include/proxies-websites.yml +++ b/playbooks/include/proxies-websites.yml @@ -996,6 +996,13 @@ cert_name: "{{wildcard_cert_name}}" tags: calendar + - role: httpd/website + site_name: kerneltest.fedoraproject.org + sslonly: true + server_aliases: [kerneltest.stg.fedoraproject.org] + cert_name: "{{wildcard_cert_name}}" + tags: kerneltest + # fedorahosted is retired. We have the site here so we can redirect it. - role: httpd/website +1s? Thanks, Pierre

3 4

CPE Weekly: 2020-04-14
by Aoife Moloney 14 Apr '20

14 Apr '20

# CPE Weekly: 2020-04-14 --- title: CPE Weekly status email tags: CPE Weekly, email --- Hi All, Apologies for the delayed weekly mail, I enjoyed a lovely four-day weekend with my family over Easter which was important and didn't get around to sending this email. The upside is you get two emails from me this week instead :) Background: The Community Platform Engineering group is the Red Hat team combining IT and release engineering from Fedora and CentOS.Check out our teams info here https://docs.fedoraproject.org/en-US/cpe/ ## GitForge Updates * Initial conversations have started with GitLab to investigate options and weight them against more refined requirements to make sure we reach the best possible outcome * We are still engaging with Fedora Council https://pagure.io/Fedora-Council/tickets/issue/292 * We are also engaging with the CentOS board too * And we are looking at ways to have open conversations/Q&A sessions in public forums too as we move through this journey. ## Fedora Updates * F32 release first go/no-go meeting is scheduled for this week to review the estimated release date of 21st April * Calendar for meetings here https://apps.fedoraproject.org/calendar/Fedora%20release/ * F32 release schedule here https://fedoraproject.org/wiki/Releases/32/Schedule ### Data Centre Move * CommuniShift is now down until May 8th est. to facilitate the data centre move that has now officially begun. * Please see our list of affected https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA * Our first few sets of hardware have been deracked and are awaiting collection from PHX2 this week. * Please check our schedule here - we are on Week 06 https://hackmd.io/R3EkjzVyTG2TYwQvkfzYrA ### AAA Replacement * The team are planning phase two of the replacement which will target UX/UI improvements, ### CI/CD * rpmautospec * Call for testers has been sent to the devel-announce list: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapr… * Documentation is also being refined on: * How it works * What are its peculiarities * How to opt-in * What is working and what isn't yet * What remains to do before this gets pushed to production * Thank you for your feedback! ### Sustaining Team * Mbbox Upgrade * The team are making good progress on the koji-builder & koji-hub CRD https://github.com/fedora-infra/mbbox/pull/20 * They also have resolved the issue with PVC in the mbox-operator https://github.com/fedora-infra/mbbox/pull/23 * We are now looking into CI support and the work that this will involve * Bodhi * Investigation database lockups, seems related to celery worker not completing their tasks * Working on a 5.3 release * Releng * fedscm-admin 1.0.13 released * Future openh264 composes will use ODCS, script is added to releng repo * Infra * The daily standup the team has has helped a lot with managing infra tickets - they are down to 99 tickets! * Mass update of stg and prod * Please note you may experience some Kojira slowness * New review-stats application deployed - (https://fedoraproject.org/PackageReviewStatus/) ## CentOS Updates ### CentOS * CentOS CI is stable and all working fine * Investigation for cloud vms addition to cico is also underway * RHEL 8.1 batch update 07-Apr ### CentOS Stream * We have now ~200 packages built in CentOS Stream & we expect a few hundred more to become available to us soon! * The team are also still working on general compose config, koji tag & housekeeping :) As always, feedback is welcome, and we will continue to look at ways to improve the delivery and readability of this weekly report. Have a great weekend! Aoife Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ - Aoife Moloney Product Owner Community Platform Engineering Team Red Hat EMEA Communications House Cork Road Waterford

1 0

Packit Pagure support (was: [RFC] Optionally using git repositories instead of the lookaside cache)
by Neal Gompa 13 Apr '20

13 Apr '20

On Thu, Apr 9, 2020 at 11:41 AM Tomas Tomecek <ttomecek(a)redhat.com> wrote: > > On Thu, Apr 9, 2020 at 4:34 PM Jeremy Cline <jeremy(a)jcline.org> wrote: > > > > We actually are. I assume you're asking about why we're not using > > packit. We're not on GitHub so the service isn't (as far as I can tell) > > useful to us. There's huge piles of existing bash scripts and makefiles > > that achieve about what I think packit-the-cli would give us so it > > would be some amount of work with no obvious benefit to move at the > > moment. Generally speaking, though, I'm not against the idea. > > When we started packit, we played with the scripts and makefiles in > your kernel repo and I hope it's not too bold of me to say that it > shouldn't be that hard to integrate the two now. > > I'm assuming you're using pagure.io - at this point, we're not going > to integrate packit with pagure (for obvious reasons). What are your > future plans for the git forge? What the heck are you talking about? pagure.io is not going away anytime soon. And there will be other pagure instances that host source code that packit integration support would be useful for. Hell, I've already offered to help with the pagure.io service. -- 真実はいつも一つ！/ Always, there's only one truth!

4 6

FBR: Big patch to dhcpd to move all mgmt interfaces to dhcp for move
by Stephen John Smoogen 11 Apr '20

11 Apr '20

Kevin and I found all the mac addresses for the mgmt boxes and are putting them onto dhcp so when they show up in new location.. they should come up without trying to use 'dead' networks. -- Stephen J Smoogen.

3 2

Re: Issue with Fedora GeoIP service
by Kevin Fenzi 09 Apr '20

09 Apr '20

On Fri, Dec 06, 2019 at 04:49:15PM +0000, Tom Hughes wrote: > On 06/12/2019 16:41, Martin Kolman wrote: > > On Fri, 2019-12-06 at 08:38 -0600, Chris Adams wrote: > > > > > I also installed the Fedora 31 GeoIP packages and ran the geoipupdate, > > > and that DB has the correct info. > > > > IIRC the infra team mentioned some issues with the new geoip database > > being incompatible with how the service is currently implemented, > > resulting in being stuck with an outdated database until this is resolved. > > Sounds like it maybe doesn't have support for GeoLite2 and is using the > old MaxMind GeoLite Legacy databases which haven't been updated since > the start of this year. Yes, that is exactly the case. We recently looked at this to see if we could retire the service, but it looks like it's still needed, so we need to figure out how to get cycles to update it. If someone wants to work on this, let us know! kevin

7 13

[RFC] Optionally using git repositories instead of the lookaside cache
by Jeremy Cline 09 Apr '20

09 Apr '20

Hi folks, The Fedora kernel is moving to maintaining the package in a source (sometimes people refer to it as an "exploded") tree. Basically just a fork of upstream. This makes a lot of packager tasks easier, but has introduced a minor issue with respect to the lookaside cache. Right now, it's configured to create a tarball from the git tree and upload it to the lookaside cache for each build. We build the rawhide kernel every weekday (give or take) and the xz compressed source tarball is ~110MB. This works out to about 28GB per year for Rawhide alone (if this is a drop in the bucket and no one cares please let me know and we'll just do this). The old approach uploaded a release tarball and then incremental tarballs on top of that. If, however, Fedora allowed packagers to optionally generate tarballs from a git repository we could just push the linux git repository. The entire repository with history going back 15 years is under 4GB total, which is pretty good when compared to ~419GB which is the space required for the equivalent time using the lookaside cache. What would need to change: * Fedora offers a git repository to push source trees to. * A new file in the dist-git repository could be added if the packager wishes called "source-repos". In it, it contains a git url and commit identifier. For example, an entry might look like: " https://src.fedoraproject.org/sources/kernel.git v5.6" where v5.6 is a tag in the repository. We can restrict it so the git repository must be hosted by Fedora so we keep all the sources forever. * fedpkg and fedpkg-minimal would need to be updated to pull the source tree if the "source-repos" file is found and run "git archive". Fortunately this work is actually already done since Red Hat's version of fedpkg already supports this. I'm happy do to all the work for fedpkg/fedpkg-minimal to make this possible because the other option is to add a bunch of hacks to the kernel tooling to spit out a bunch of incremental tarballs to reduce what we have to upload. I assume this is something that will need to go through the packaging SIG, but from an infra side of things are there any thoughts/concerns? Regards, Jeremy

6 15

Hello world!
by David Kirwan 09 Apr '20

09 Apr '20

Hello all, My name is David Kirwan, I'm originally from Kilkenny, Ireland, now living in Waterford, Ireland. I've recently transferred to the CPE team at Red Hat, where I'll likely be working on CentOS and Fedora infrastructure in a devops capacity. I also hope to get involved in some of the packaging work for the Ruby/ML SIGs, although that might be in my spare time! To share a little background about me, I spent a few years as a software researcher, working on projects in the energy sector, security and software defined networking. In my previous role at Red Hat, I worked as a devops/releng engineer on an operations team responsible for maintaining a hosted SaaS product. Most recently, I've been working as a regular software engineer on a team developing and improving Red Hat's middleware products to work on Openshift 3/4 using Kubernetes "Operators" (Custom resource controllers). When I'm not working, I try to spend as much time as possible getting involved in the local Makerspace (South East Makerspace), working on personal art, astronomy, amateur radio, diy, electronics and woodworking projects. I'm delighted to be here, and I'm looking forward to working with everyone! -- David Kirwan SOFTWARE ENGINEER Red Hat Cloud Services Communications House Cork Road, Waterford dkirwan(a)redhat.com T: +(353) 86-8624108 IM: @dkirwan

4 4

[Fedocal] Reminder meeting : Fedora Infrastructure
by smooge＠gmail.com 08 Apr '20

08 Apr '20

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2020-04-09 from 15:00:00 to 16:00:00 UTC At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/9675/

1 0

Weekly Backlog Refinement - Week Apr 06 2020
by Clement Verna 08 Apr '20

08 Apr '20

Hi all, Let's look at 5 tickets and rate them using the following categories : * low-trouble, medium-trouble, high-trouble * low-gain, medium-gain, high-gain #8729 Do we want to have a copy of translate.fp.o backups? - https://pagure.io/fedora-infrastructure/issue/8729 Trouble : ? Gain : ? #8693 Please create `@copr-team` src.f.o group - https://pagure.io/fedora-infrastructure/issue/8693 Trouble: ? Gain: ? #8600 All maintainers should be removed from retired packages eventually. - https://pagure.io/fedora-infrastructure/issue/8600 Trouble: ? Gain: ? #8481 Cannot subscribe to the ocaml-devel mailing list via hyperkitty, get a webserver error - https://pagure.io/fedora-infrastructure/issue/8481 Trouble: ? Gain: ? #8370 Fedora CoreOS + multi-arch - https://pagure.io/fedora-infrastructure/issue/8370 Trouble: ? Gain: ? Thanks all

2 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure April 2020