infrastructure June 2022

infrastructure@lists.fedoraproject.org

25 participants
24 discussions

About JS framework
by Pierre-Yves Chibon 14 Sep '22

14 Sep '22

Good Morning Everyone, Our infrastructure is mostly a python store, meaning almost all our apps are written in python and most using wsgi. However in python we are using a number of framework: * flask for most * pyramid for some of the biggest (bodhi, FAS3) * Django (askbot, Hyperkitty) * TurboGears2 (fedora-packages) * aiohttp (python3, async app: mdapi) While this makes sometime things difficult, these are fairly standard framework and most of our developers are able to help on all. However, as I see us starting to look at JS for some of our apps (fedora-hubs, wartaa...), I wonder if we could start the discussion early about the different framework and eventually see if we can unify around one. This would also allow those of us not familiar with any JS framework to look at the recommended one instead of picking one up semi-randomly. So has anyone experience with one or more JS framework? Do you have one that would you recommend? Why? Thanks for your inputs, Pierre

15 19

planet
by Kevin Fenzi 24 Aug '22

24 Aug '22

Hey folks. I thought I would open a discussion about fedoraplanet and possibly some plans for it. Right now: fedoraplanet.org runs on people02.fedoraproject.org (aka fedorapeople). To add a blog/rss feed you have to login there and edit your .planet file, then scripting pulls all those .planet files and tries to fetch all the feeds and then serves them up at http://fedoraplanet.org. It uses a app called 'venus' to do this. venus is written in very old python2 and very very dead upstream. We run into the following problems with it: * Sometimes it gets stuck and just stops processing until it's killed. * It's serving on a http site, which causes people to ask us to make it https, but that would just change the errors because many feeds it pulls are still http since they were added back before letsencrypt existed. * We have a handy 'website' field in our new account system, but aren't using it at all. * The .planet parsing is poor, any number of things can cause it to break. We have two open tickets on it: - https://pagure.io/fedora-infrastructure/issue/10383 (upgrade to pluto, a ruby based, but maintained thing) - https://pagure.io/fedora-infrastructure/issue/10490 ( planet not served via ssl) Which I am just going to close now. So, I can think of a number of options and would love everyone who has thoughts on it to chime in: 1. Do nothing. Venus "works" and .planet files are cool and retro. 2. Switch to pluto and use account system 'website' fields of contributors. We could likely shove it in openshift and serve it directly from there to avoid fedorapeople entirely. (This would likely break anyone who has multiple feeds in there) 3. Switch to something better/bigger. I would think (although I don't know) that there might be something that would not only aggregate rss feeds for contributors, but perhaps mastodon/twitter/whatever also. 4. Planets are old and tired, just drop the entire thing. People can maintain their own rss lists. 5. Planets are old and tired, just drop the entire thing. But also, get our social media people to maintain contributor / interesting lists. ie, the fedoraproject twitter account could maintain a list of 'fedora contributors' and 'fedora packagers' or whatever. 6. Switch to pluto as in 2, but also setup some curators. Have a 'firehose' of all feeds, but the main fedora planet would be just curated things that are known to be related to fedora and not off topic or unrelated. 6. Get someones (not it!) to take in all the twitter/facebook/mastodon/blog posts/rss feeds and post some kind of curated round up every week or something. 7. Your brilliant idea here! So, thoughts? this is not at all urgent, but we should end up doing something with it sometime. :) kevin

10 21

openshift application container versions
by Kevin Fenzi 29 Jun '22

29 Jun '22

I whipped up a brute force little script and ran it on our staging openshift 4 cluster. It just gets all running pods and does oc rsh into each and looks at /etc/os-release. For a few of these that aren't running yet in stg, I looked at prod (like toddlers). Here's the results and some comments in between. I can work on upgrading some of these, but would be nice if application owners update their own applications. ;) asknot "platform:f34" blockerbugs "platform:el8" bodhi "platform:f36" "platform:f36" "platform:f36" "platform:f36" bugzilla2fedmsg "platform:el8" community-metrics compose-tracker "platform:f36" coreos-cincinnati "platform:f35" datagrepper "platform:el8" datanommer "platform:el8" discourse2fedmsg "platform:el8" docsbuilding "platform:f34" ^ This should get updated. docstranslation ^ No running pods, but cronjobs use f34. Should be updated elections "platform:el8" fasjson "platform:f35" fedocal "platform:el8" fedora-packages-static ^ debian 11 in the solr pod ^ fedora 34 in the static pod, should get updated flask-oidc-dev "platform:el8" flatpak-indexer "platform:el8" "platform:el8" "platform:f36" greenwave "platform:f33" "platform:f33" "platform:f33" ^ Needs update. ipsilon-website ^ Alpine Linux v3.16 (is this current? no idea) koschei "platform:f35" "platform:f35" "platform:f35" "platform:f35" "platform:f35" "platform:f35" languages "platform:el8" mdapi "platform:f31" "platform:f31" ^ Definitely needs update. message-tagging-service "platform:f29" ^ Definitely needs update. messaging-bridges "platform:el8" "platform:el8" "platform:el8" monitor-gating "platform:f31" ^ Definitely needs update. mote "platform:f36" "platform:f36" "platform:el9" noggin "platform:el8" noggin-centos "platform:el8" oraculum "platform:el8" "platform:el8" "platform:el8" "platform:el8" "platform:el8" release-monitoring "platform:f36" "platform:f36" "platform:f36" resultsdb "platform:f35" "platform:f35" resultsdb-ci-listener "platform:f36" review-stats ^ just cronjobs, fedora:36 silverblue PRETTY_NAME="Fedora 27 (Twenty Seven)" ^ very much needs updating. testdays "platform:el8" "platform:el8" the-new-hotness "platform:f36" toddlers PRETTY_NAME="Alpine Linux v3.16" ^ is this current? transtats "platform:f34" "platform:f34" ^ needs updating. waiverdb "platform:f34" "platform:f34" ^ needs updating. websites ^ cron jobs, fedora:34. Needs updating. zezere "platform:el8"

3 4

openshift 3->4 moves status and info needed
by Kevin Fenzi 27 Jun '22

27 Jun '22

Greetings everyone. I finally had a quiet afternoon to really dig into the ocp3->4 migration. We have a total of 64 unique applications/projects over the 4 clusters (ocp3 prod/stg and ocp4 prod/stg). Of those 20 have been moved to ocp4 (either stg or prod or both). There's 4 that are sort of in the midst of moving to ocp4 prod, but need some more tweaking (flatpak-indexer, release-monitoring, discourse2fedmsg, transtats). I'd like to propose removing the following 9 entirely: REMOVE: accountsystem.yml ( an old proof of concept of noggin/caapi/etc) REMOVE: distgit_bugzilla_sync.yml ( we do this in a toddler anymore) REMOVE: fas-changes.yml ( I think this was just needed for a short time for the account system migration, please correct me if it's got some better use) REMOVE: fas-stg.yml ( Old account system staging, we can use the fas.yml playbook if we have to keep fas) REMOVE: ipsilon.yml ( we moved this to vm's because we couldn't run pam_sssd in openshift. Has that changed?) REMOVE: kerneltest.yml ( This never went into service and doesn't do exactly what kernel maintainers want ) REMOVE: sanitarium.yml ( some test project? ) REMOVE: simple-koji-ci.yml ( This was retired a long time ago) REMOVE: sso.yml ( another test?) These 3 are broken in staging and need to be fixed before we can move them (I really like to see things run in stg before we move them to prod): BROKEN: compose-tracker.yml BROKEN: monitor_gating.yml BROKEN: transtats.yml I'll try and ping owners of these to look and fix them soon. That leaves: datagrepper.yml datanommer.yml docsbuilding.yml docstranslation.yml elections.yml (will wait for current elections to finish) fas.yml fedocal.yml fedora-packages-static.yml greenwave.yml koschei.yml languages.yml mdapi.yml messaging-bridges.yml nuancier.yml (I think this was a staging only proof of concept) oraculum.yml review-stats.yml testdays.yml toddlers.yml waiverdb.yml websites.yml zezere.yml I'll keep working on the list this week and hopefully we can get them all moved soon. :) kevin

8 17

CPE Weekly Update – Week 25 2022
by Akashdeep Dhar 24 Jun '22

24 Jun '22

Hi everyone, This is a weekly report from the CPE (Community Platform Engineering) Team. If you have any questions or feedback, please respond to this report or contact us on #redhat-cpe channel on libera.chat (https://libera.chat/). Week: 20th - 24th June 2022 If you wish to read this in form of a blog post, check the post on the Fedora community blog: https://communityblog.fedoraproject.org/cpe-weekly-update-week-25-2022/ Highlights of the week Infrastructure & Release Engineering Goal of this Initiative The purpose of this team is to take care of day-to-day business regarding CentOS and Fedora Infrastructure and Fedora release engineering work. It’s responsible for services running in Fedora and CentOS infrastructure and preparing things for the new Fedora release (mirrors, mass branching, new namespaces etc.). The ARC (which is a subset of the team) investigates possible initiatives that CPE might take on. Link to planning board: https://zlopez.fedorapeople.org/I&R-2022-06-22.pdf Link to docs: https://docs.fedoraproject.org/en-US/infra/ Update Fedora Infra - Most apps have moved over to the OpenShift4 cluster. Hopefully, the transition should be finishing up this week. - Wiki: All upgraded in production and working (thanks Ryan!) - Resultsdb: All moved over to OpenShift 4 in prod and working (thanks Leo!) - Business proceeding as usual CentOS Infra including CentOS CI - Kerberos settings switch for git.centos.org (kcm on el8 vs keyring on el7) for lookaside upload CGI <https://pagure.io/centos-infra/issue/811> [1] - Issue on iad2 hosted reference mirror <https://pagure.io/centos-infra/issue/812>[2] (epel.next and mirrormanager), all fixed now - Duffy CI ongoing tasks and deployments (all announced) - Equinix nodes migration <https://pagure.io/centos-infra/issue/816>[3] (on their request) - Business proceeding as usual Release Engineering - Compose-tracker updated to f36 in staging, production happening tomorrow - Python 3.11 merged to rawhide - MBS randomly fails to process builds - Rawhide compose failures recently (syslinux retirement, then python 3.11 merge) all fixed now - Business proceeding as usual CentOS Stream Goal of this Initiative This initiative is working on CentOS Stream/Emerging RHEL to make this new distribution a reality. The goal of this initiative is to prepare the ecosystem for the new CentOS Stream. Updates - CentOS Stream 8: Manually keeping regular RPMs and module RPMs updated on the koji.stream server as current updates are composed and released. CentOS Duffy CI Goal of this Initiative Duffy is a system within CentOS CI infrastructure allowing tenants to provision and access machines (physical and/or virtual, of different architectures and configurations) for the purposes of CI testing. Development of Duffy is largely finished, we're currently planning and testing deployment scenarios. Updates - Release version 3.2.1 - Docs, docs, docs and a Dojo Package Automation (Packit Service) Goal of this initiative Automate RPM packaging of infra apps/packages Updates - Mostly business as usual - Thanks again to all who are reviewing our PRs - Most of our GitHub critical apps are enabled now or close to being enabled Flask-oidc: oauth2client replacement Goal of this initiative Flask-oidc is a library used across the Fedora infrastructure and is the client for ipsilon for its authentication. flask-oidc uses oauth2client. This library is now deprecated and no longer maintained. This will need to be replaced with authlib. Updates: - POC working using authlib, tidying up code to prepare to submit a PR back to upstream EPEL Goal of this initiative Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest Group that creates, maintains, and manages a high-quality set of additional packages for Enterprise Linux, including, but not limited to, Red Hat Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux (OL). EPEL packages are usually based on their Fedora counterparts and will never conflict with or replace packages in the base Enterprise Linux distributions. EPEL uses much of the same infrastructure as Fedora, including a build system, Bugzilla instance, updates manager, mirror manager and more. Updates - This week we have 6442 (+127) packages, from 2882 (+76) source packages - Containerd and puppet retired from EPEL7 because of upstream EOL and multiple CVEs. - Caddy was updated, fixing 4 CVEs in EPEL9 *Index* [1] https://pagure.io/centos-infra/issue/811 [2] https://pagure.io/centos-infra/issue/812 [3] https://pagure.io/centos-infra/issue/816 Thanks and Regards, Akashdeep Dhar (he/him), Objective Representative, Fedora Council Red Hat Community Platform Engineering t0xic0der(a)fedoraproject.org akashdeep(a)redhat.com

1 0

release-monitoring.org 1.4.0 is now live
by Michal Konecny 22 Jun '22

22 Jun '22

Hi everyone, today was deployed 1.4.0 version of https://release-monitoring.org. Here is a highlight of some of the changes: *Features* * Add link to AlmaLinux package to distribution mapping * Add sourceforge (git) backend to retrieve git tags * Add Python (PEP 440) versioning scheme *Bug fixes* * Only include unyanked PyPI versions * Version Filter not applied on Test Check * Intermediate versions are skipped while update checking For full changelog please visit https://github.com/fedora-infra/anitya/releases/tag/1.4.0 Regards, Michal

1 0

[Fedocal] Reminder meeting : Websites & Apps Team Meeting
by t0xic0der＠fedoraproject.org 20 Jun '22

20 Jun '22

Dear all, You are kindly invited to the meeting: Websites & Apps Team Meeting on 2022-06-21 from 15:00:00 to 16:00:00 UTC At https://meet.jit.si/fedora-websites-apps-meeting The meeting will be about: Weekly team meeting for the Websites & Apps Team. This is part of the [Websites & Apps Community Revamp Objective](https://fedoraproject.org/wiki/Objectives/Websites_%26_Apps_Comm…. See [past discussion]( https://discussion.fedoraproject.org/t/planning-meeting-for-websites-apps-t…) to learn more about the Websites & Apps Team and how it came together. More information available at: [hackmd.io/Mxm2We3yTqKybLsdohadOA](https://hackmd.io/Mxm2We3yTqKybLsdohadOA?view) Source: https://calendar.fedoraproject.org//meeting/9990/

1 0

openshift applications best practices
by Kevin Fenzi 20 Jun '22

20 Jun '22

So, when moving openshift apps this week, I really came to the conclusion we should look at writing up some guidelines for openshift apps. Perhaps it could have 'MUST' items and 'SHOULD' items and 'SUGGESTION' items. I know I am coming at this from the operations side of things, so please do share other viewpoints, but let me mention a few issues I see: It's very difficult to tell what an application is running on: - We should probibly forbid 'X:latest' because what the application is running depends on when it was last built/deployed. (unless: see below) - A number of apps use images elsewhere (quay.io, etc). We have very little way to tell whats in those or how they are made or if anyone is still making them. We have a few apps that rebuild/deploy on git changes, which is fine, but doesn't do anything to keep their container env up to date/secure. We have a number of apps that tigger on imagestream changes, but as far as I can tell thats only changes in their image ('bodhi-base') not the base image that it was built from ('fedora:36'). I'd like to see us track base images and build/deploy when one changes. If we do that then :latest might be ok to use, since we know automation took care of updating it for us. There's a number of applications using ubi (rhel's base image), which I think might be worth recommending, especially if we start build/deploy on base image changes. Likely Fedora base images will change more quickly. Currently we have some apps that were setup and probibly have not changes in years, happily running the same old container image since there was no reason to redeploy them. Similarly, python versions are all over the place... python3.6, 3.8, 3.9, etc. Of course different apps will target different versions sometimes, but it might be nice to make sure we are using a supported version by whatever container image the app is using. (ie, gets security fixes). Our naming conventions are a bit all over the place too. Some apps have $appname-build for buildconfig, some just $appname, some seperate configs for seperate containers ($app-web, $app-frontend). Same for deploymentconfigs. Anyhow, I'll stop there... but hopefully everyone would be ok with such a doc and then changing things to match it? kevin

7 9

[RFC] Build tag in RPM: from NVR to NVRB
by Aleksandra Fedorova 18 Jun '22

18 Jun '22

Hi, all, I'd like to discuss how we can add Build tag in the RPM. As one of the key points is to turn it into a common standard for rpm packages across the ecosystem, the conversation is currently opened upstream [1] and in RHEL Engineering. And I'd like to get Fedora community on board. This is not a finalized proposal and I think it is not ready yet to be a Fedora Change. But I want to start a conversation and ask for opinions. There are also some open questions which need help, especially the requirements around build reason. And alternative suggestions are welcome as well. I've posted long version at https://discussion.fedoraproject.org/t/rfc-build-tag-in-rpms/39954 You can comment there (simply login with your FAS id) or here on the mailing list. And I am going to update that post with the new feedback. Shorter version: ================================================ We'd like to introduce Build Number/Tag/Id in the rpm metadata. By this change we would like to: * Provide a possibility to change build environment and rebuild rpm packages without changing their content: neither sources nor spec files. * Set a common standard for the RPM-based ecosystem, which can be used not just within Fedora, but also by Remixes, downstreams, SIGs and other distributions. The most visible impact of the proposal would be the filename change: Current: dnf-4.9.0-12.fc36.noarch.rpm Proposed: dnf-4.9.0-12.fc36.34897715.noarch.rpm It can be implemented in two steps: 1) Now → Compatibility mode * Introduce Build tag in the rpm metadata * Introduce “build reason” to be added to rpm changelog as a top entry * Enable passing Build tag value to the build in Koji. The value of the Build tag will be set from Koji build id. * Introduce macro in Release field of the rpm spec files, which adds Build tag after the usual disttag Release: 12.%{?dist}%{?build:.%{build}} * Introduce option to pass “build reason” to a Koji build via koji cli and fedpkg/centpkg tooling. 2) Compatibility mode → Final * Implement support for the upgrade path on the rpm side in a compatible way. So that NV(R’=R+B) and NVRB are treated the same. * Remove %{build} part from Release tag and use independent Build tag for versioning. ================================================ [1] https://github.com/rpm-software-management/rpm/discussions/2031 -- Aleksandra Fedorova CI for Fedora/CentOS Stream/RHEL bookwar on Matrix/IRC

1 0

CPE Weekly Update – Week 24 2022
by Michal Konecny 17 Jun '22

17 Jun '22

Hi everyone, This is a weekly report from the CPE (Community Platform Engineering) Team. If you have any questions or feedback, please respond to this report or contact us on #redhat-cpe channel on libera.chat (https://libera.chat/). Week: 13th - 17th June 2022 If you wish to read this in form of a blog post, check the post on Fedora community blog: https://communityblog.fedoraproject.org/cpe-weekly-update--week-24-2022/ # Highlights of the week ## Infrastructure & Release Engineering Goal of this Initiative ----------------------- Purpose of this team is to take care of day to day business regarding CentOS and Fedora Infrastructure and Fedora release engineering work. It’s responsible for services running in Fedora and CentOS infrastructure and preparing things for the new Fedora release (mirrors, mass branching, new namespaces etc.). The ARC (which is a subset of the team) investigates possible initiatives that CPE might take on. Link to planning board: https://zlopez.fedorapeople.org/I&R-2022-06-15.pdf Link to docs: https://docs.fedoraproject.org/en-US/infra/ Update ------ ### Fedora Infra * Resultsdb almost moved to ocp4 in prod, just a few parts to finish (Thanks Leo!) * Ocp4 cluster now on our vpn, so all proxies can reach apps (thanks darknao!) * Wiki upgrade looking good in staging, prod to come (thanks ryan!) * Some more vm’s to f36 * About 50% done moving apps to ocp4. * Image builder prod move blocked due to firewall issues ### CentOS Infra including CentOS CI * https://git.centos.org went live last monday * Trying to resume discussion with RH IT for Stream storage migration * New CI infra deployment tasks to be ready * https://lists.centos.org/pipermail/ci-users/2022-June/004547.html ### Release Engineering * ELN composes were broken over the weekend because of ODCS backend / front end version mismatch * Nodejs-sig removed as the default assignee on a bunch of components in BZ * we have discovered workflow in bodhi that locks update in a weird state more info https://github.com/fedora-infra/bodhi/issues/4566 ## CentOS Stream Goal of this Initiative ----------------------- This initiative is working on CentOS Stream/Emerging RHEL to make this new distribution a reality. The goal of this initiative is to prepare the ecosystem for the new CentOS Stream. Updates ------- * We imported all RPMs for modules (CentOS Stream 8) to the shared buildsystem * All sources imported to GitLab (CentOS Stream 8) ## CentOS Duffy CI Goal of this Initiative ----------------------- Duffy is a system within CentOS CI infrastructure allowing tenants to provision and access machines (physical and/or virtual, of different architectures and configurations) for the purposes of CI testing. Development of Duffy is largely finished, we're currently planning and testing deployment scenarios. Updates ------- * Test and polish `duffy client …` experience * Docs and CentOS Dojo talk prep ## Package Automation (Packit Service) Goal of this initiative ----------------------- Automate RPM packaging of infra apps/packages Updates ------- * Almost finished, only mirrormanager2 remaining from our critical apps on Github * Couple of outliers (fasjson, flask-mod-auth) need downstream repos created * Datanommer.models manually packaged so datagrepper can be automated * Noggin now fully automated ## Flask-oidc: oauth2client replacement Goal of this initiative ----------------------- Flask-oidc is a library used across the Fedora infrastructure and is the client for ipsilon for its authentication. flask-oidc uses oauth2client. This library is now deprecated and no longer maintained. This will need to be replaced with authlib. Updates: -------- * Working [poc app](https://app-flask-oidc-dev.apps.ocp.stg.fedoraproject.org/oidc/) which authenticates against noggin/ipa using authlib and OIDC. * Working on an upstream PR with the working code now. ## EPEL Goal of this initiative ----------------------- Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest Group that creates, maintains, and manages a high quality set of additional packages for Enterprise Linux, including, but not limited to, Red Hat Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux (OL). EPEL packages are usually based on their Fedora counterparts and will never conflict with or replace packages in the base Enterprise Linux distributions. EPEL uses much of the same infrastructure as Fedora, including buildsystem, bugzilla instance, updates manager, mirror manager and more. Updates ------- * This week we have 6315 (+297) packages, from 2806 (+76) source packages * [XFCE now available in epel9-testing](https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-5fe… * Removed epel9 packages that were included in rhel9.0 * oniguruma: [backported fix](https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-a9236c0113) for CVE-2019-13225 (moderate) from rhel8 to epel7 * drbd: pull request to [switch pacemaker dependency from requires to recommends](https://src.fedoraproject.org/rpms/drbd/pull-request/2) to comply with policy Kindest regards, CPE Team

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure June 2022