infrastructure December 2019

infrastructure@lists.fedoraproject.org

28 participants
27 discussions

Meeting Agenda Item: Introduction Igor Vasiliev
by vaska vaska 14 Dec '19

14 Dec '19

Hello. I want to enroll in your team. I am a telecommunication engineer with management and deployment packet networks skills. Also have a bit ability to programm on Python and Bash. My own home computer has been running linux distributions for 8 years. Want to learn how to use Ansible, Openshift,Virsh and etc. Sorry, but now can't choose a problem to resolve by me. Need a more time to lean a lot about project. Senserolly yours, Igor Vasilev. Fedoraproject nickname: bacbka IRC Nickname: babcka87

1 0

epel-playground.repo installed by epel8 version of epel-release package?
by Merlin Mathesius 13 Dec '19

13 Dec '19

Greetings. This was discussed a bit in the #epel channel, but it was suggested I take it to the list... While adding the epel-modular.repo file to the epel8 branch of the epel-release package, I noticed that the package was installing /etc/yum.repos.d/epel-playground.repo (although with enabled=0)--which seemed a little out of place. The question came up whether the epel8 version of the epel-release package should be installing that repo file at all, or whether it should be completely dropped and only be installed by the epel8-playground version of the epel-release package. Are there any opinions about dropping the epel-playground.repo file from the epel8 branch of the epel-release package? Or leave it as is? I can see one potential use case where someone could run dnf --enablerepo=epel-playground update epel-release as a way to switch from the standard to playground repos. Merlin

2 3

CPE Weekly: 2019-12-13
by Aoife Moloney 13 Dec '19

13 Dec '19

Hi everyone, Welcome to the CPE team weekly project update mail! Background: The Community Platform Engineering group is the Red Hat team combining IT and release engineering from Fedora and CentOS. Our goal is to keep core servers and services running and maintained, build releases, and other strategic tasks that need more dedicated time than volunteers can give. For better communication, we will be giving weekly reports to the CentOS and Fedora communities about the general tasks and work being done. Also for better communication between our groups we have created #redhat-cpe on Freenode IRC! Please feel free to catch us there, a mail has landed on both the CentOS and Fedora devel lists with context here. Note: This document is currently built from individual reports rolled into a document which we edit and copy into a final document. We are aware that this causes problems with some email readers, and are working on a method to make this less problematic. High Level Project Updates: Fedora Production issues with Bodhi: Bodhi is under investigation for misbehaving. Creating update can take a long time or just fail. The status can be tracked here https://status.fedoraproject.org/ Rawhide Gating Fixed: https://pagure.io/fedora-ci/general/issue/70 repoSpanner This has been turned off on Fedora side except for src.stg.fedoraproject.org The team are now working on whether to convert src.stg or redeploy Pagure Updates Pagure 5.8.1 has been released Pagure 5.8.1 has been pushed to staging & production Pagure-dist-git 1.5.1 and 1.5.2 has been released Pagure-dist-git 1.5.1 has been pushed to staging Pagure and 1.5.2 has been pushed to staging & production Fedora Docs Updates Merges: https://pagure.io/fedora-docs/quick-docs/pull-request/166 https://pagure.io/fedora-docs/quick-docs/pull-request/161 https://pagure.io/fedora-docs/quick-docs/pull-request/159 Fedora docs was published to localization stats to stage Community Fires Team Made the side-tag workflow work in stable releases Fixed the distgit-bugzilla-sync script New pagure and pagure-dist-git have been released and deployed Fix showing CI results on PR on dist-git Added a new API endpoint to Pagure to install and remove git hook Tiger Team Updates: OSBS Sprint focused on getting the aarch64 VMs in staging Deployed Fedora 30 on the VMs and started deploying OpenShift Currently blocked by the authentication to quay.io to get the aarch64 images. Focus for the next sprint is to have a working OpenShift on the aarch64 hardware. Bodhi CD Ability to build a docker image in quay.io on a push to github branch Ability to trigger a redeploy in openshift automatically after image has been rebuilt in quay Upgrade the database schema automatically with openshift deployment, instead of manually running ansible playbooks Application Retirements Elections Everything should be ready now to move Elections to Communishift Planning will start soon for a move date! Fedocal No progress on kanban board last seven weeks - looks abandoned https://teams.fedoraproject.org/project/fedora-calendar/kanban Jlanda’s permission error in communishift should be fixed now https://pagure.io/fedora-infrastructure/issue/8274 Another permission issue occurred during test Nuancier Benson Muite is now working on OIDC authentication The team have recommended to create a PR on Github to be able to see the progress PR from sebwoj - Porting to Fedora messaging Sebwoj is now working on adding scheme as part of the PR EPEL 8 Modularity EPEL 8 Modularity work tested in stage and was deployed to production on Tuesday 10th December EPEL 8 Playground Modularity work is on hold. Blocker: https://pagure.io/fm-orchestrator/issue/1541 CentOS Wiki.centos.org migration (last monday) Finishing templates for mailman ansible role (new look with community members) Preparing the next migration (sponsor relocates DC and new machines): www.centos.org (on track, to be announced) Forums (scheduled for next monday) Starting new ansible roles to automate deploy + upgrade of those migrated services Working on koji (cbs.dev) to see when we can import 8.1 content to let SIGs build against/for .el8 and .el8s (Stream) Collaborating with Artwork SIG for new design for http exposed services (new centos design) QA for 8-Stream and 8.1.1911 continues Misc Updates Tests to jms-messaging plugin are now successfully running & waiting for review Buildvmhost-01/02/03/04 has been reinstalled with rhel8 Buildvm-01 to 32 has been reinstalled Autocloud has been decommissioned ppc9-02 has been reinstalled with f31, and buildvm-ppc64le-01 to 09 ppc8-01 has been reinstalled with rhel8 F29 signed builds, updates, updates-testing, container, cloud composes in /mnt/koji/composes were cleaned up this week Holiday Season Shutdown Notices: Red Hat does a 2 week end of year shutdown where nearly everyone is encouraged to spend time away from computers and family. During that time, some operators will be monitoring tickets and services for large scale outages, but will not be active on IRC or email. The ability of the operators to fix problems is also limited because some operations require staff who will be away. People on duty will do what they can to repair and maintain services but do not give any ETA on how long it will take for it to be done. If you are making changes to infrastructure during this time, please treat it like a ‘frozen’ period. Unless it is an emergency repair, do not push changes without a review from a main system administrator. We would encourage you to bump your ticket or request early in the New Year (full service resumes January 6th) for our attention, otherwise we will work through the backlog of requests. We wish you and yours a happy holiday season and look forward to working with you in collaboration throughout 2020 and beyond. Comments? Suggestions? Feedback? Let Us Know! Kindest regards, Aoife -- Aoife Moloney Feature Driver Community Platform Engineering Team Red Hat EMEA Communications House Cork Road Waterford

1 0

koji builder status
by Kevin Fenzi 13 Dec '19

13 Dec '19

Greetings. I just thought I would share a short status update on koji builders with everyone. As many of you know, we like to keep the Fedora koji builders on the current most recent stable Fedora release. This is to make sure we have compatible rpm, etc to build rawhide and all our stable updates. TLDR summary: All Fedora koji builders are now on Fedora 31 and are using only python3 (the hub is still python2 however). Due to a bug in the linux kernel on armv7, we never upgraded to Fedora 30 when it was released, so builders stayed on Fedora 29 for the last cycle. Happily that bug was tracked down and fixed and I have been moving them to Fedora 31 for the past month or so. Normally it doesn't take to long to reinstall everything (Thanks ansible!) but this time it's been a longer road due to: New aarch64 hardware that needed racking/cabling/setup (many thanks to Smooge for getting this done), firmware updates for most all hardware, RHEL8 reinstalls for virthosts that run buildvm's, and many other items. We still have pending: * Some more aarch64 hardware to bring online early in the new year. * A bug around armv7 buildvm's with more than 24GB memory. While thats being investigated all the buildvm-armv7 instances have just 24GB memory. * A bug around power9 hosts, causing our cloud/container image builds to fail. This was thought fixed, but doesn't seem to be yet. * Daily db dumps still cause slowness, likely in the new year we will schedule an outage and patition the problem tables. A few stats: 155 total enabled builders 47 x86_64 builders 32 ppc64le builders 32 aarch64 builders 27 armhfp builders 24 s390x builders Around 10,000 builds a month. Close to 1.1 million builds since fc6 days Happy building.

1 0

Self introduction
by Mattia Verga 13 Dec '19

13 Dec '19

Hi all, my name's Mattia Verga (IRC: mattiaV // fas: mattia) and I work as a field technician for an Italian telecommunication company. I've been a Fedora user since the start and actually a packager of some astronomy related and KDE packages. I do not use any programming skills at work, but I've acquired some knowledge in Python, also by contributing to Bodhi (it's a mutually beneficial exchange, I try to help there and I acquire more Python knowledge). I've also a little experience in html / JavaScript / SQL. I'd like to help keeping Fedora running and to get a better knowledge of what it keeps running (servers - services - etc.)... so I'd be happy to continue contributing to Bodhi development and/or help maintaining the networking infrastructure. Unfortunately, it's unlikely that I'll be able to participate in IRC meetings, since I'm usually on duty shift at that time. Mattia

1 0

Task List
by Álvaro Castillo 11 Dec '19

11 Dec '19

Good morning, I joined into Fedora Project since little bit of time. Where I can found a Task List where the project identify what needs and goals? According to Fedora Project Infra info It likes does not exist. Is It right? https://fedoraproject.org/wiki/Infrastructure#Contribute_to_Fedora_Infrastr… If It does not exist, will be a nice choice to improve our productivity as team, I think so. I would like to be start contribute to this project someway. Greetings

3 2

[Fedocal] Reminder meeting : Fedora Infrastructure
by smooge＠gmail.com 11 Dec '19

11 Dec '19

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2019-12-12 from 15:00:00 to 16:00:00 UTC At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/22/

1 0

[fwd] [ANNOUNCE] Git v2.24.1 and others
by Todd Zullinger 10 Dec '19

10 Dec '19

Hi, A number of security flaws in git were announced today. It looks like most of them affect clients rather than servers, and several only apply to Windows. But I wanted to give a heads-up about it in case there are areas where infrastructure tools might perform some of the vulnerable actions. For the most part, I suspect we don't need to worry too much here. In any case, I'm preparing updates for Fedora now. Fedora 31 will be bumped from 2.23.0 to 2.24.1 (I had planed to push an update to 2.24.x due to some other bugs in 2.23.0 anway). Fedora 30 will go from 2.21.0 to 2.21.1. I have forwarded the release announcement to the Red Hat security team so they can file tracking tickets as well. ----- Forwarded message from Junio C Hamano <gitster(a)pobox.com> ----- Date: Tue, 10 Dec 2019 10:05:46 -0800 To: git(a)vger.kernel.org Cc: Linux Kernel <linux-kernel(a)vger.kernel.org>, git-packagers(a)googlegroups.com From: Junio C Hamano <gitster(a)pobox.com> Subject: [ANNOUNCE] Git v2.24.1 and others Message-ID: <xmqqr21cqcn9.fsf(a)gitster-ct.c.googlers.com> Today, the Git project is releasing the following Git versions: v2.24.1, v2.23.1, v2.22.2, v2.21.1, v2.20.2, v2.19.3, v2.18.2, v2.17.3, v2.16.6, v2.15.4, and v2.14.6 These releases fix various security flaws, which allowed an attacker to overwrite arbitrary paths, remotely execute code, and/or overwrite files in the .git/ directory etc. See the release notes attached for the list for their descriptions and CVE identifiers. Users of the affected maintenance tracks are urged to upgrade. These flaws were discovered and reported by Joern Schneeweisz of GitLab and by Microsoft Security Response Center (and in particular Nicolas Joly), and were fixed by Johannes Schindelin, Jeff King, Garima Singh and Jonathan Nieder on the git-security mailing list. The release engineering and coordination was led by Johannes Schindelin. The tarballs are found at: https://www.kernel.org/pub/software/scm/git/ The following public repositories all have a copy of the 'v2.24.1' and other tags: url = https://kernel.googlesource.com/pub/scm/git/git url = git://repo.or.cz/alt-git.git url = https://github.com/gitster/git ---------------------------------------------------------------- Git v2.14.6 Release Notes ========================= This release addresses the security issues CVE-2019-1348, CVE-2019-1349, CVE-2019-1350, CVE-2019-1351, CVE-2019-1352, CVE-2019-1353, CVE-2019-1354, and CVE-2019-1387. Fixes since v2.14.5 ------------------- * CVE-2019-1348: The --export-marks option of git fast-import is exposed also via the in-stream command feature export-marks=... and it allows overwriting arbitrary paths. * CVE-2019-1349: When submodules are cloned recursively, under certain circumstances Git could be fooled into using the same Git directory twice. We now require the directory to be empty. * CVE-2019-1350: Incorrect quoting of command-line arguments allowed remote code execution during a recursive clone in conjunction with SSH URLs. * CVE-2019-1351: While the only permitted drive letters for physical drives on Windows are letters of the US-English alphabet, this restriction does not apply to virtual drives assigned via subst <letter>: <path>. Git mistook such paths for relative paths, allowing writing outside of the worktree while cloning. * CVE-2019-1352: Git was unaware of NTFS Alternate Data Streams, allowing files inside the .git/ directory to be overwritten during a clone. * CVE-2019-1353: When running Git in the Windows Subsystem for Linux (also known as "WSL") while accessing a working directory on a regular Windows drive, none of the NTFS protections were active. * CVE-2019-1354: Filenames on Linux/Unix can contain backslashes. On Windows, backslashes are directory separators. Git did not use to refuse to write out tracked files with such filenames. * CVE-2019-1387: Recursive clones are currently affected by a vulnerability that is caused by too-lax validation of submodule names, allowing very targeted attacks via remote code execution in recursive clones. Credit for finding these vulnerabilities goes to Microsoft Security Response Center, in particular to Nicolas Joly. The `fast-import` fixes were provided by Jeff King, the other fixes by Johannes Schindelin with help from Garima Singh. Git v2.15.4 Release Notes ========================= This release merges up the fixes that appear in v2.14.6 to address the security issues CVE-2019-1348, CVE-2019-1349, CVE-2019-1350, CVE-2019-1351, CVE-2019-1352, CVE-2019-1353, CVE-2019-1354, and CVE-2019-1387; see the release notes for that version for details. In conjunction with a vulnerability that was fixed in v2.20.2, `.gitmodules` is no longer allowed to contain entries of the form `submodule.<name>.update=!command`. Git v2.16.6 Release Notes ========================= This release merges up the fixes that appear in v2.14.6 and in v2.15.4 addressing the security issues CVE-2019-1348, CVE-2019-1349, CVE-2019-1350, CVE-2019-1351, CVE-2019-1352, CVE-2019-1353, CVE-2019-1354, and CVE-2019-1387; see the release notes for those versions for details. Git v2.17.3 Release Notes ========================= This release merges up the fixes that appear in v2.14.6 and in v2.15.4 addressing the security issues CVE-2019-1348, CVE-2019-1349, CVE-2019-1350, CVE-2019-1351, CVE-2019-1352, CVE-2019-1353, CVE-2019-1354, and CVE-2019-1387; see the release notes for those versions for details. In addition, `git fsck` was taught to identify `.gitmodules` entries of the form `submodule.<name>.update=!command`, which have been disallowed in v2.15.4. Git v2.18.2 Release Notes ========================= This release merges up the fixes that appear in v2.14.6, v2.15.4 and in v2.17.3, addressing the security issues CVE-2019-1348, CVE-2019-1349, CVE-2019-1350, CVE-2019-1351, CVE-2019-1352, CVE-2019-1353, CVE-2019-1354, and CVE-2019-1387; see the release notes for those versions for details. Git v2.19.3 Release Notes ========================= This release merges up the fixes that appear in v2.14.6, v2.15.4 and in v2.17.3, addressing the security issues CVE-2019-1348, CVE-2019-1349, CVE-2019-1350, CVE-2019-1351, CVE-2019-1352, CVE-2019-1353, CVE-2019-1354, and CVE-2019-1387; see the release notes for those versions for details. Git v2.20.2 Release Notes ========================= This release merges up the fixes that appear in v2.14.6, v2.15.4 and in v2.17.3, addressing the security issues CVE-2019-1348, CVE-2019-1349, CVE-2019-1350, CVE-2019-1351, CVE-2019-1352, CVE-2019-1353, CVE-2019-1354, and CVE-2019-1387; see the release notes for those versions for details. The change to disallow `submodule.<name>.update=!command` entries in `.gitmodules` which was introduced v2.15.4 (and for which v2.17.3 added explicit fsck checks) fixes the vulnerability in v2.20.x where a recursive clone followed by a submodule update could execute code contained within the repository without the user explicitly having asked for that (CVE-2019-19604). Credit for finding this vulnerability goes to Joern Schneeweisz, credit for the fixes goes to Jonathan Nieder. Git v2.21.1 Release Notes ========================= This release merges up the fixes that appear in v2.14.6, v2.15.4, v2.17.3 and in v2.20.2, addressing the security issues CVE-2019-1348, CVE-2019-1349, CVE-2019-1350, CVE-2019-1351, CVE-2019-1352, CVE-2019-1353, CVE-2019-1354, CVE-2019-1387, and CVE-2019-19604; see the release notes for those versions for details. Additionally, this version also includes a couple of fixes for the Windows-specific quoting of command-line arguments when Git executes a Unix shell on Windows. Git v2.22.2 Release Notes ========================= This release merges up the fixes that appear in v2.14.6, v2.15.4, v2.17.3, v2.20.2 and in v2.21.1, addressing the security issues CVE-2019-1348, CVE-2019-1349, CVE-2019-1350, CVE-2019-1351, CVE-2019-1352, CVE-2019-1353, CVE-2019-1354, CVE-2019-1387, and CVE-2019-19604; see the release notes for those versions for details. Git v2.23.1 Release Notes ========================= This release merges up the fixes that appear in v2.14.6, v2.15.4, v2.17.3, v2.20.2 and in v2.21.1, addressing the security issues CVE-2019-1348, CVE-2019-1349, CVE-2019-1350, CVE-2019-1351, CVE-2019-1352, CVE-2019-1353, CVE-2019-1354, CVE-2019-1387, and CVE-2019-19604; see the release notes for those versions for details. Git v2.24.1 Release Notes ========================= This release merges up the fixes that appear in v2.14.6, v2.15.4, v2.17.3, v2.20.2 and in v2.21.1, addressing the security issues CVE-2019-1348, CVE-2019-1349, CVE-2019-1350, CVE-2019-1351, CVE-2019-1352, CVE-2019-1353, CVE-2019-1354, CVE-2019-1387, and CVE-2019-19604; see the release notes for those versions for details. ----- End forwarded message ----- -- Todd ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so. -- Douglas Adams

1 0

moving bugzilla overrides to dist-git
by Karsten Hopp 10 Dec '19

10 Dec '19

Hi, We are currently working on getting rid of the git repo at fedora-scm-requests [1] which is nowadays only used to store the overrides of the default assignee in bugzilla (for example to allow different default assignee for Fedora and EPEL). I am working on porting this mechanism to dist-git itself (much like we did for the anitya integration a few weeks ago). We are thinking on providing a simple text field to submit FAS username or email to override the default assignee, the big question is then, who should be allowed to update this field ? Should the main admin be able to set someone else as assignee ? If there is already an override assignee, who should be allowed to change that ? If there's no override assignee set, can everyone become it or is that up to the main admin of the component to decide and set ? I'd appreciate your input Thanks, Karsten [1] https://pagure.io/releng/fedora-scm-requests/

7 9

F29 and older instances
by Kevin Fenzi 10 Dec '19

10 Dec '19

Greetings everyone. On 2019-11-26 fedora 29 will go end of life and no longer supported. We still have a number of things that are f29 (or older for various reasons). I'd like everyone to look this over and see if you can't move things to f31 (or at least f30) before the f29 eol. builders: I am working on this, builders right now are mostly f29, but should be all moved before the eol date I hope. openshift apps: modernpaste: fedora-27. :) But I don't think it was ever moved to openshift, so we should delete this. joystick: we should fix this up, I think jdoss was going to take it over? the-new-hotness: can we move to f31? qa-stg01.qa.fedoraproject.org: f24? can we retire? autocloud*: f27, but should die when f29 goes eol (it's only used to test atomic-host). mdadpi: f27, but it moved to openshift, can we clean up the non openshift instances and files please? modernpaste: f27. It goes eol soon, so I guess we just wait notifs-backend: f27. I don't know what to do here. We need a replacement, which we don't have. I'm afraid that upgrading will blow it up, but I guess we could try and rollback? qa-prod01: f27. can we retire? ci-cc-rdu01: f28. Can we retire soon? copr-frontend01/02.stg: f28 and I don't think we need them anymore, can I nuke them? db-qa03: f28. Can we upgrade please? osbs: f28, but I think cverna was moving it to newer? any status? f29's: aarch64-test01/02 - can be redone as f31 anytime composer.stg - should be moved to f31 db-koji01.stg - should be moved to f31 and a prod->stg sync kojipkgs01/02 - I should do these soon to f31, adding to my list. os-proxy01 - I should do this one, on my list packages03/04 - No idea here. Can we upgrade? proxy* - we need to start working on this asap. relepel01 - do we need this one anymore? resultsdb/taskotron - can qa folks upgrade these? old cloud instances: glittergallery-dev: f23, should be nuked fedora-bootstrap: f25, should be nuked waiverdb-dev: f25, still needed? commops: f27, still needed? telegram-irc: ? still needed? copr*stg: f28, can copr folks upgrade? developer: f28, still needed? libravatar: f28, should ask them to move to communishift simple-koji-ci: f29. Can we upgrade to 31? or move it to communishift? I think thats all of them. kevin

5 7

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure December 2019