infrastructure May 2018

infrastructure@lists.fedoraproject.org

36 participants
36 discussions

About JS framework
by Pierre-Yves Chibon 14 Sep '22

14 Sep '22

Good Morning Everyone, Our infrastructure is mostly a python store, meaning almost all our apps are written in python and most using wsgi. However in python we are using a number of framework: * flask for most * pyramid for some of the biggest (bodhi, FAS3) * Django (askbot, Hyperkitty) * TurboGears2 (fedora-packages) * aiohttp (python3, async app: mdapi) While this makes sometime things difficult, these are fairly standard framework and most of our developers are able to help on all. However, as I see us starting to look at JS for some of our apps (fedora-hubs, wartaa...), I wonder if we could start the discussion early about the different framework and eventually see if we can unify around one. This would also allow those of us not familiar with any JS framework to look at the recommended one instead of picking one up semi-randomly. So has anyone experience with one or more JS framework? Do you have one that would you recommend? Why? Thanks for your inputs, Pierre

15 19

Fedora Infrastructure Authentication Roadmap
by Kevin Fenzi 29 Jun '18

29 Jun '18

Greetings. Sorry it took so long to write this up and send it out, but here's our proposed plan for authentication moving forward. Please do feel free to comment or suggest changes/improvements here. Any mistakes are mine alone. :) Fedora Project Auth roadmap Background/History The Fedora project created its own authentication/user/group management system at nearly the beginning of its existance. FAS (Fedora Account System) (version 1) and then a rewrite (FAS2). At each of these points other solutions were investigated and found unacceptable for various reasons. Over the last few years, several additional applications have been added next to FAS2 to provide additional functionality: ipisilon has been added as a identity provider, and FreeIPA has been added for kerberos authentication. FAS2 is still the authoritative source of authentication data. FAS2 is currently deployed on RHEL6 servers and won't run on RHEL7. Also during the last few years, a new FAS re-write has been slowly in the works. FAS3 is written in a modern framework and has a number of functionality and interface improvements over FAS2. Additionally it can run on RHEL7. Goals and Critera Maintaining authentication applications is difficult and time consuming work, and it has always been a goal to try and move to more industry standard applications as much as possible given our goals and critera. The last time we looked, Some of those goals/critera include: * User self service registration * User self service password reset * FPCA acceptance requirement * Basset integration (may not be needed anymore) * Allow Self Service groups with their own sponsors/admins * Allow group requirements (other group first, etc) Plan: On discussion with FreeIPA developers and looking at how things are setup now, we came up with a plan to get what we need, but reduce the footprint and maintance we need to do. Many of the features we were hoping to use in FAS3 have now been implemented upstream in FreeIPA (2fa, fasClient syncing better, etc). Basically we will: * A new small wrapper type project is written (Community Account Information API) or CAIAPI. This small app provides the Critera listed above, talking at first to FAS2 on the backend then, later switching to talking to FreeIPA on the backend and providing a json API to consumers. * Switch anything we have using the direct FAS api to use CAIAPI instead. * Move to FreeIPA being the canonical source for authentication data. This should just be a switch to CAIAPI, and no consumers should even notice. * FreeIPA still provides kerberos auth. Note that kerberos will remain limited to use at ipsilon and koji. * Ipsilon provides identity auth for applications, preferably with OIDC (still provides others) * A new small website that uses the CAIAPI json to provide end user access / management. This thing would be in flask and needs a name still. Since https://fedoraproject.org/wiki/Infrastructure/fas_freeipa FreeIPA has matured and our understanding of the work required to make CAIAPI and a small web consumer has clarified. Pros: * IPA handles all the storing of credentials, replication and such and we just use it. * Our maint goes from needing to maintain FAS2 or FAS3 to just CAIAPI (a much smaller api) and a very small web application. * Easier to audit 2 small apps. * We can try and share the CAIAPI with other open source communities (Gnome? CentOS? others?) Open Source Communities already using FreeIPA would be easy to add this to. * We can stop using fasClient in favor of ipa-client setup (no more heavy syncing) * The heavy security aspects will be handled by upstreams we don't need to fully maintain (FreeIPA, sssd, ipsilon, etc). Cons: * We still need to write the CAIAPI/webapp, although Patrick may have CAIAPI already somewhat implemented. * It feels very sad to have spent so long on FAS3 and never deploy it, but sometimes thats just the way things go. ;(

6 7

Infrastructure office hours
by Clement Verna 12 Jun '18

12 Jun '18

Dear all, I would like to get feedback about starting a weekly infrastructure office hours which would be used to help anyone with contributing to Fedora's infrastructure. This would be a dedicated IRC meeting where people would be available to answer questions, help setting up development environment or even maybe work together on an issue. It could also be used to add more details to the issues we have in our backlog to help new contributor to get started. Note that we already have a general "ask whatever, whenever" on #fedora-apps and #fedora-admin where anybody can "ask whatever, whenever" :-), but this does not seems to work since we have very few requests. So do you think that having a dedicated meeting where you know that people are available would improve the situation ? Thanks Clément

7 9

Fedora and PDC
by Pierre-Yves Chibon 11 Jun '18

11 Jun '18

Good Morning Everyone, We have been informed thst week at the upstream devs working on the product-definition-center (PDC) are moving away from the project and are going to leave it without a maintainer. Since we adopted PDC for a variety of data flows, this puts us in an awkward position. :( Ralph and I met up on Tuesday to brainstorm the list of things we actively use PDC for today and to come up with a contingency plan for how to handle them. One overarching option is for us (fedora-infra) to take ownership of the PDC codebase as a whole. We didn't fully explore this option, figuring that the codebase is large and contains lots of tables, endpoints, and codepaths that we didn't use nor which we plan to use. Instead, below we've got the four things we use PDC for and some options for what to do with each. With the exception of /modules/, one common pattern that we like is to investigate splitting out the "django apps" that make up PDC into their own projects. We're calling these "pdc-lite", for fun. See more below. * Modules The data in the /modules/ PDC endpoint is *also* in the MBS db. Ralph's team is going to just use that and stop using pdc anything for modules. We're going to need to patch pungi, mbs for local builds, and a few other places. This should be a relatively low-pain transition. * Stream branches, branch ownership, retirement dates? - SLA/EOL are currently stored in PDC. - Queried by releng scripts for retirement, fedrepo-req for new branches, etc.. option #1 git repo full of yaml file similar to the override repo compiled into a single JSON blob Single place for all retired packages This feels like the lowest tech option. git gives us change control for free... but people easily get lost in the "UX" of navigating a gigantic git repo full of plaintext files. option #2 pagure's DB/API pagure knows nothing about branches currently, so that would be bigger lift option #3 PDC internally is composed of ~20 "django apps" https://github.com/product-definition-center/product-definition-center/tree… We could pick the 2 or 3 that comprise the branches feature, copy them out, and turn them into their own service: the "branch definition center": BDC. That would be the "pdc-lite" approach mentionned above, ie: PDC with only the "app" of interest * release/life-circle tracking? option #1: PDC lite with just that app of interest option #2: JSON/yaml file on the proxies option #3: pkgdb-lite option #4: ??? compose tracking? impacted: fedfind option #1: PDC-lite with just that app of interest option #2: Drop this entirely? Adam probably really wants to keep the record of composes. option #3: ??? The "pdc-lite" options are attractive, across the board. One thing we get from this is greater clarity when discussing things formerly in PDC. If something is in the branch-definition-center, the compose-definition-center, or the release-definition-center.. you know what you're talking about. Today, when talking about whether or not something should be or is in "PDC", it is easy to get confused. I propose we start the discussion on the list and plan for a meeting sometime late next week to discuss it further with the interested parties (please signal yourself) What do you think? Pierre and Ralph

10 23

Migrating fedmsg to AMQP: a proposal
by Aurelien Bompard 05 Jun '18

05 Jun '18

Hey folks! Jeremy and I have been working on a proposal to migrate fedmsg from our current brokerless architecture to a broker-based architecture. The overview and reasons for the migration are described on this page: https://fedmsg-migration-tools.readthedocs.io/en/latest/migration/overview.… Head there if you want the details. The plan has the following requirements: * No flag day. * Don't disrupt any services or applications. * Don't break any services outside of Fedora's infrastructure relying on these messages. The first step is to deploy a broker in Fedora to use. In order to avoid a flag day, bridges from AMQP to ZeroMQ and ZeroMQ to AMQP have been implemented and will be deployed. In order to validate that the bridges are functioning, a small service will be run during the transition period that connects to fedmsg and to the AMQP queues to compare messages. After the bridges are running, applications are free to migrate. There are several options when migrating and each has advantages and disadvantages. We have written a new library called fedora-messaging that has the following features: * A method to define message schemas and offer automatic validation of messages using those schemas. * Boilerplate for typical publishers and consumers. Head over to the document for a demo! What do you think of this proposal? Any blind spots? Thanks! Aurélien

5 9

Meeting Agenda Item: Introduction Chandrakant Kumar
by chandrakant kumar 01 Jun '18

01 Jun '18

Hello, I have gone through Fedora Infrastructure Getting Started wiki [1] and Fedora Infra docs [2] and wanted to introduce myself to this group. Your IRC handle chankuma What skills you have to offer Proficient in Python and Bash, I have also built web applications with django and flask. What you want to learn Systems administration skills, Infrastructure automation. [1] https://fedoraproject.org/wiki/Infrastructure/GettingStarted [2] https://pagure.io/infra-docs

2 1

FAS: SSH key type: Is it still required to use a RSA key?
by Christian Stadelmann 31 May '18

31 May '18

Today I added a ssh key to my account on FAS [1]. It says I should upload a RSA key, this is part of the help text: "Warning: In case of having ECDSA key please upload the two types of keys because some of our servers may not accept ECDSA keys." Is this warning still true? If yes, does it apply to the pagure instance on src.fedoraproject.org (ssh: pkgs.fedoraproject.org) too? I'm asking because I thought I'll stop using old pre quantum computer aera crypto and try ed25519 instead. [1] https://admin.fedoraproject.org/accounts/user/edit/genodeftest

2 2

Looking up module information via Koji
by Owen Taylor 31 May '18

31 May '18

My understanding is that with the planned retirement of the PDC: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapr… Querying for module information should be done using the MBS and/or Koji APIs. Various code that I maintain (in OSBS, fedmod, and random tooling) wants to do module build lookups - different variations of "look up the modulemd for latest build of a NAME:STREAM[:VERSION]". Variations generally being exactly what "latest" means here. The code generally already is using Koji and the MBS api is quite limited, so I've chosen to do the lookups via Koji. https://fishsoup.net/misc/get-module-builds Is a test tool that incorporates most of the capability that I needed across my uses. It's distinctly more than a couple of lines of code - I can cut-and-paste it for now, but what's the right long-term home? Is there a simpler way? My best idea right now is that if the 'base_version' and'status' part of my code was simplified to simply be "tag" and avoid reliance on the tag structure of Fedora, then this might make a reasonable addition to the Koji CLI and API - there are some things that using raw tags for the query makes trickier, but it's probably workable. Thanks for any input! Owen

1 1

Librepo/dnf zchunk integration question
by Jonathan Dieter 31 May '18

31 May '18

Zchunk works by comparing an old version of the file with the one you want to download, but when dnf refreshes a repository, it downloads the new file into a temporary directory with no information passed to the handle about where the old files are. I've been trying to keep my code changes in libsolv and librepo to make zchunk integration as universal as possible. Up until now, I have managed to do so without changing librepo's API, but I don't see any way to fix this except to have dnf pass information about the old directory (or, even better, the cache directory) to the handle, which will mean an API change. It would also mean that other utilities would probably need to do the same. Is there something I'm missing in dnf's interaction with librepo that would allow me to work around this, or do I just need to bite the bullet and propose a librepo API change? Jonathan

1 0

Infrastructure Meeting 2018-05-31 14:00 UTC
by Stephen John Smoogen 30 May '18

30 May '18

= Preamble = The infrastructure team will be having its weekly meeting tomorrow, 2018-05-31 at 14:00 UTC in #fedora-meeting-1 on the freenode network. We have a gobby document (see: https://fedoraproject.org/wiki/Gobby ) fedora-infrastructure-meeting-next is the document. Please try and review and edit that document before the meeting and we will use it to have our agenda of things to discuss. A copy as of today is included in this email. If you have something to discuss, add the topic to the discussion area with your name. If you would like to teach other folks about some application or setup in our infrastructure, please add that topic and your name to the learn about section. = Introduction = We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. = Meeting start stuff = #startmeeting Infrastructure (2018-05-31) #meetingname infrastructure #topic aloha #chair smooge relrod nirik pingou puiterwijk tflink = Let new people say hello = #topic New folks introductions #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves = Status / Information / Trivia / Announcements = (We put things here we want others on the team to know, but don't need to discuss) (Please use #info <the thing> - your name) #topic announcements and information #info relrod PTO 9 Jun - 19 Jun #info smooge PTO 8 Jun, 15 Jun -> 18 Jun #info bodhi-3.8.0 deployed = Things we should discuss = We use this section to bring up discussion topics. Things we want to talk about as a group and come up with some consensus /suor decision or just brainstorm a problem or issue. If there are none of these we skip this section. (Use #topic your discussion topic - your username) #topic Oncall #info Smooge is on call from 2018-05-29->2018-06-04 #info Nirik is on call from 2018-06-05->2018-06-11 #info Smooge is on call from 2018-06-11->2018-06-15 #info (!Smooge|!Relrod) is on call from 2018-06-15->2018-06-18 #topic Fedora web dev clinic resurrection (https://communityblog.fedoraproject.org/friday-fedora-web-dev-clinic/) - cverna #topic Tickets discussion #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=2 Go thru each ticket one by one = Apprentice office hours = #topic Apprentice Open office minutes #info A time where apprentices may ask for help or look at problems. Here we will discuss any apprentice questions, try and match up people looking for things to do with things to do, progress, testing anything like that. = Learn about some application or setup in infrastructure = (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the i/nfo in this section. In the event we don't find someone to teach about something, we skip this section and just move on to open floor.) #info none this week = Meeting end stuff = #topic Open Floor #endmeeting -- Stephen J Smoogen.

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure May 2018