infrastructure November 2019

infrastructure@lists.fedoraproject.org

34 participants
43 discussions

Meeting agenda for 2019-11-21
by Stephen John Smoogen 20 Nov '19

20 Nov '19

The infrastructure team will be having its weekly meeting tomorrow, 2019-11-21 at 15:00 UTC in #fedora-meeting-1 on the freenode network. We have a document at https://board.net/p/fedora-infra Please try and review and edit that document before the meeting and we will use it to have our agenda of things to discuss. A copy as of today is included in this email. If you have something to discuss, add the topic to the discussion area with your name. If you would like to teach other folks about some application or setup in our infrastructure, please add that topic and your name to the learn about section. ## Introduction We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. ### Meeting start stuff ``` #startmeeting Infrastructure (2019-11-14) #meetingname infrastructure #topic aloha #chair nirik pingou relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs ``` ### Determine who the next chair is #topic Next chair #info magic eight ball says: 2019-11-21: pingou 2019-11-28 - holiday, NO MEETING! 2019-12-05 -smooge 2019-12-12 -?? 2019-12-19 -?? 2019-12-26 - NO MEETING 2020-01-02 - NO MEETING 2020-01-09 - ?? ### Let new people say hello ``` #topic New folks introductions #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted ``` ### Status / Information / Trivia / Announcements (We put things here we want others on the team to know, but don't need to discuss) (Please use ```#info (the thing - your name)``` ``` #topic announcements and information #info ``` ### Things we should discuss We use this section to bring up discussion topics. Things we want to talk about as a group and come up with some consensus /suor decision or just brainstorm a problem or issue. If there are none of these we skip this section. (Use ```#topic your discussion topic - your username)``` ``` #topic Oncall #info https://fedoraproject.org/wiki/Infrastructure/Oncall #info pingou is on call 2019-11-14->2019-11-21 #info kevin is on call 2019-11-21->2019-11-28 #info smooge is on call 2019-11-28->2019-12-05 ## .oncalltakeeu .oncalltakeus #info Summary of last week: (from current oncall ) #info #topic Monitoring discussion #info https://nagios.fedoraproject.org/nagios #info Go over existing out items and fix #topic Tickets discussion #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket #infohttps://pagure.io/fedora-infrastructure/issue/8397 #info Fedora CoreOS Team requests #link https://hackmd.io/5uB7hOJKSjGUt65iLgPnbA#Existing-requests-for-Fedora-Infra #topic backlog discussion #info go over our backlog and discuss and determine priority #link https://pagure.io/fedora-infrastructure/issue/8178 #info topic: provision new aarch64 builders #link https://pagure.io/fedora-infrastructure/issue/8157 #info topic: ansible: enable ansible-report as a hook ``` Go thru each ticket one by one #topic Community Fires Team ### Put all topics for discussion under here Here we will discuss any apprentice questions, try and match up people looking for things to do with things to do, progress, testing anything like that. ### Learn about some application or setup in infrastructure (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the i/nfo in this section. In the event we don't find someone to teach about something, we skip this section and just move on to open floor.) ``` #info ``` ### Meeting end stuff ``` #topic Open Floor #endmeeting -- Stephen J Smoogen.

1 0

[Fedocal] Reminder meeting : Fedora Infrastructure
by smooge＠gmail.com 20 Nov '19

20 Nov '19

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2019-11-21 from 15:00:00 to 16:00:00 UTC At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/22/

1 0

New mirrorlist server implementation
by Adrian Reber 19 Nov '19

19 Nov '19

Fedora's complete MirrorManager setup is still running on Python2. The code has been ported to Python3 probably over two years ago but we have not switched yet. One of the reasons is that the backend is running on RHEL7 which means we are not in a hurry to deploy the Python3 version. The mirrorlist server which is answering the actual dnf/yum queries for a mirrorlist/metalink is, however, running in a Fedora 29 container. This container also still uses Python2 and it actually cannot use the Python3 version. One of MirrorManager's design points is that the mirrorlist servers, which are answering around 27 000 000 requests per day, are not directly accessing the database. The backend creates a snapshot of the relevant data (113MB) and the mirrorlist servers are using this snapshot to answer client requests. This data exchange is based on Python's pickle format and that does not seem to work with Python3 if it is generated using Python2. Having used protobuf before, I added code to also export the data for the mirrorlist servers based on protobuf. The good news with protobuf is, that the resulting file is only 66MB instead of 113MB. The bad news is, that loading it from Python requires 3.5 times the amount of memory during runtime (3.5GB instead of 1GB). In addition to the data exchange problems between backend and mirrorlist servers the architecture of the mirrorlist server does not really make sense today. 12 years ago it made a lot of sense as it could be easily integrated into httpd and it could be easily reloaded without stopping the service. Today the mirrorlist server and httpd is all part of a container which is then behind haproxy. So there is a lot of infrastructure in the container which is not really useful. To get rid of the pickle format and to have a simpler architecture I reimplemented the mirrorlist-server in Rust. This was brought up some time ago on a ticket and with the protobuf problems I was seeing in Python it made sense to try it out. My code currently can be found at https://github.com/adrianreber/mirrorlist-server and so far the results from the new mirrorlist server are the same as from the Python based mirrorlist server. It requires less than 700MB instead of the 1GB in Python with production based data and seems really fast. I have set up a test instance with the mirror data from Sunday at: https://lisas.de/metalink?repo=updates-testing-f31&arch=x86_64 https://lisas.de/mirrorlist?repo=updates-testing-f31&arch=x86_64 The instance is based on the container I pushed to quay.io: $ podman run quay.io/adrianreber/mirrorlist-server:latest -h With this change the mirrorlist server would also finally switch to geoip2. The currently running mirrorlist server still uses the legacy geoip database. After the Fedora 31 freeze I would like to introduce this new mirrorlist server implementation on the proxies. I already verified that I can run this mirrorlist container rootless. This new container can be a drop-in replacement for the current container and no infrastructure around it needs to be changed. The main changes to get it into production is to change mirrorlist1.service and mirrorlist2.service to include a line "User=mirrormanager" and replace the current container name with new container. Adrian

4 13

Wikiedit FAS group and Fedora Join new workflow
by alciregi＠gmail.com 19 Nov '19

19 Nov '19

Hello! As you can read here[0], the Fedora Join SIG is experimenting a new way to help people become part of the community. This workflow includes also a temporary FAS group if required or asked by the newcomer. Since this overlaps with the current "wikiedit" system, we were wondering if it'd make sense to retire it and send newcomers to the Fedora Join channels instead where they can speak to community members and take their time learning/exploring the community and it's projects. It'll be one less responsibility for the infra team, and it'll help us channel folks to the new workflow. Please let us know if this sounds OK and we'll edit the wiki page here accordingly [1]. We'll send out more posts announcing the new workflow to the community this week. If you have the cycles, please hang out in the Fedora Join channels[2]. [0] : https://communityblog.fedoraproject.org/fedora-join-is-trying-a-new-people-… [1] https://fedoraproject.org/wiki/Help:Editing#Gaining_Edit_Access [2] https://fedoraproject.org/wiki/SIGs/Join?rd=Fedora_Join_SIG Thanks, A.

5 23

Problems with vcglib
by J. Scheurich 18 Nov '19

18 Nov '19

Hi, There are problems with the vcglib package vcglib review request: https://bugzilla.redhat.com/show_bug.cgi?id=1677989 $ kinit mufti11(a)FEDORAPROJECT.ORG Password for mufti11(a)FEDORAPROJECT.ORG: $ fedpkg clone vcglib Cloning into 'vcglib'... No such repository fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. Could not execute clone: Failed to execute command. $ Did i something wrong ? so long MUFTI

1 0

patches to review to prepare our ansible scripts for fedora-messaging
by karsten＠fedoraproject.org 18 Nov '19

18 Nov '19

Hello, There is work in progress to migrate our applications from the current message bus 'fedmsg' to the AMPG based 'fedora-messaging'. Attached are a couple of patches that prepare our ansible scripts for this. Please review those patches and comment, thanks ! Karsten

5 7

Greenwave release
by Giulia Naponiello 18 Nov '19

18 Nov '19

Hello, I'm going to perform a Greenwave release in a bit. I'll send an email again with the release notes when I'm done with it. Thank you Cheers Giulia

1 1

Fwd: How to approve a review request ?
by J. Scheurich 16 Nov '19

16 Nov '19

Hi, >> I want to approve the review request of vimvi-qt, but this is my first >> offical review 8-( > You need to set fedora-review+ flag. Click on "Show advanced fields" near > the top of the bug, and flags should be visible in a drop-down menu on > the right. | I still don't find it 8-( http://wdune.ourproject.org/examples/vimvi.png | Mufti, it's weird that your screenshot shows that you can set the | fedora-review flag to ? but not to +. Maybe file a ticket with | infra---possibly Bugzilla didn't sync with FAS properly or something? Can you help me ? so long MUFTI

2 1

CPE Weekly: 2019-11-16
by Aoife Moloney 16 Nov '19

16 Nov '19

Hi Everyone! Travelling yesterday saw a delay in sending this weeks CPE email blast, but nevertheless, here it is! Hope everyone has been having a great weekend and if you have any comments or feedback, please let me know- it is always welcome :) Background: The Community Platform Engineering group is the Red Hat team combining IT and release engineering from Fedora and CentOS. Our goal is to keep core servers and services running and maintained, build releases, and other strategic tasks that need more dedicated time than volunteers can give. For better communication, we will be giving weekly reports to the CentOS and Fedora communities about the general tasks and work being done. Also for better communication between our groups we have created #redhat-cpe on Freenode IRC! Please feel free to catch us there, a mail has landed on both the CentOS and Fedora devel lists with context here. Note: This document is currently built from individual reports rolled into a google document which we edit and copy into a final document. We are aware that this causes problems with some email readers, and are working on a method to make this less problematic. High Level Project Updates: Fedora: Rawhide Gating: Bodhi 5.0 has been deployed in production We received some feedback about the changes in the UI and are looking at how we can address them. We are working on documenting on how packagers will use Bodhi for multi-builds updates in rawhide: https://github.com/fedora-infra/bodhi/issues/2322 https://fedoraproject.org/w/index.php?title=User:Nphilipp/Package_update_HO… A 5.1 release is scheduled, likely some time next week, to address some of the issues encountered and fixed in the 5.0 release. repoSpanner Our team came across some test reliability issues last week & these are impeding progress on the 83x patch. Some more debugging/discussions around RCMs pushes that seem to be either very slow or hanging but the team is looking into some solutions Some progress was also made on the 83x patch however and we managed to fix one unreliable test. Application Retirements Elections Blocking issue was fixed (https://pagure.io/fedora-infrastructure/issue/8253) Fedocal jlanda hitting permission error in communishift https://pagure.io/fedora-infrastructure/issue/8274 Nuancier Benson Muite is now working on OIDC authentication Pastebin An article will be posted to Magazine instead - keep an eye out! GDPR policies are still being looked into so stay tuned for updates soon. This unfortunately takes some time given the legal perspective needed! CentOS: The team is busy working on 8.1 Comps were produced and in git repo for next compose (pungi) Transmodrify tool rewritten to match the mbs/rpm needs Analyzing what RH hasn’t pushed and so that is needed Builds started and are being working on for x86_64, ppc64le, aarch64 and armhfp architectures https://koji.mbox.centos.org/koji/index https://koji.armhfp-mbox.centos.org/koji/index Infrastructure : Dealing with hardware issues on storage box , so opening tickets with vendor and internal IT to play MITM and try to get this solved Dealing with RCM having still issues to push to git.centos.org for some identified packages (reproducible) so now waiting on Repospanner maintainer to come with a fix and an ETA for new release that would fix it Trying to provision some EC2 instances and working with Fedora to get proper access to be able to have proper AWS access for CentOS (also for S3) Working on various PR for cbs-content-control and/or issues with mash on https://cbs.centos.org (now resolved) Misc EPEL 8 modularity EPEL8 module builds are working in staging & composes should be closed There are now tags added to staging in koji Bodhi configs have been updated Pungi configs for epel8-playground are updated Bug about zchunk update for EPEL-7 was fixed https://github.com/fedora-infra/bodhi/pull/3692 Fedora Docs Multiple small PRs were merged where people were being sent to Fedora docs after F31 came out & below are all fixed: https://pagure.io/fedora-docs/system-administrators-guide/issue/40 https://pagure.io/fedora-docs/system-administrators-guide/issue/39 https://pagure.io/fedora-docs/system-administrators-guide/issue/35 https://pagure.io/fedora-docs/install-guide/issue/33 https://pagure.io/fedora-docs/system-administrators-guide/issue/25 https://pagure.io/fedora-docs/system-administrators-guide/issue/23 https://pagure.io/fedora-docs/quick-docs/issue/155 Kindest regards, Aoife -- Aoife Moloney Feature Driver Community Platform Engineering Team Red Hat EMEA Communications House Cork Road Waterford

1 0

[Fedocal] Reminder meeting : Fedora Infrastructure
by smooge＠gmail.com 13 Nov '19

13 Nov '19

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2019-11-14 from 15:00:00 to 16:00:00 UTC At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/22/

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure November 2019