infrastructure April 2018

infrastructure@lists.fedoraproject.org

43 participants
69 discussions

About JS framework
by Pierre-Yves Chibon 14 Sep '22

14 Sep '22

Good Morning Everyone, Our infrastructure is mostly a python store, meaning almost all our apps are written in python and most using wsgi. However in python we are using a number of framework: * flask for most * pyramid for some of the biggest (bodhi, FAS3) * Django (askbot, Hyperkitty) * TurboGears2 (fedora-packages) * aiohttp (python3, async app: mdapi) While this makes sometime things difficult, these are fairly standard framework and most of our developers are able to help on all. However, as I see us starting to look at JS for some of our apps (fedora-hubs, wartaa...), I wonder if we could start the discussion early about the different framework and eventually see if we can unify around one. This would also allow those of us not familiar with any JS framework to look at the recommended one instead of picking one up semi-randomly. So has anyone experience with one or more JS framework? Do you have one that would you recommend? Why? Thanks for your inputs, Pierre

15 19

Fedora Infrastructure Authentication Roadmap
by Kevin Fenzi 29 Jun '18

29 Jun '18

Greetings. Sorry it took so long to write this up and send it out, but here's our proposed plan for authentication moving forward. Please do feel free to comment or suggest changes/improvements here. Any mistakes are mine alone. :) Fedora Project Auth roadmap Background/History The Fedora project created its own authentication/user/group management system at nearly the beginning of its existance. FAS (Fedora Account System) (version 1) and then a rewrite (FAS2). At each of these points other solutions were investigated and found unacceptable for various reasons. Over the last few years, several additional applications have been added next to FAS2 to provide additional functionality: ipisilon has been added as a identity provider, and FreeIPA has been added for kerberos authentication. FAS2 is still the authoritative source of authentication data. FAS2 is currently deployed on RHEL6 servers and won't run on RHEL7. Also during the last few years, a new FAS re-write has been slowly in the works. FAS3 is written in a modern framework and has a number of functionality and interface improvements over FAS2. Additionally it can run on RHEL7. Goals and Critera Maintaining authentication applications is difficult and time consuming work, and it has always been a goal to try and move to more industry standard applications as much as possible given our goals and critera. The last time we looked, Some of those goals/critera include: * User self service registration * User self service password reset * FPCA acceptance requirement * Basset integration (may not be needed anymore) * Allow Self Service groups with their own sponsors/admins * Allow group requirements (other group first, etc) Plan: On discussion with FreeIPA developers and looking at how things are setup now, we came up with a plan to get what we need, but reduce the footprint and maintance we need to do. Many of the features we were hoping to use in FAS3 have now been implemented upstream in FreeIPA (2fa, fasClient syncing better, etc). Basically we will: * A new small wrapper type project is written (Community Account Information API) or CAIAPI. This small app provides the Critera listed above, talking at first to FAS2 on the backend then, later switching to talking to FreeIPA on the backend and providing a json API to consumers. * Switch anything we have using the direct FAS api to use CAIAPI instead. * Move to FreeIPA being the canonical source for authentication data. This should just be a switch to CAIAPI, and no consumers should even notice. * FreeIPA still provides kerberos auth. Note that kerberos will remain limited to use at ipsilon and koji. * Ipsilon provides identity auth for applications, preferably with OIDC (still provides others) * A new small website that uses the CAIAPI json to provide end user access / management. This thing would be in flask and needs a name still. Since https://fedoraproject.org/wiki/Infrastructure/fas_freeipa FreeIPA has matured and our understanding of the work required to make CAIAPI and a small web consumer has clarified. Pros: * IPA handles all the storing of credentials, replication and such and we just use it. * Our maint goes from needing to maintain FAS2 or FAS3 to just CAIAPI (a much smaller api) and a very small web application. * Easier to audit 2 small apps. * We can try and share the CAIAPI with other open source communities (Gnome? CentOS? others?) Open Source Communities already using FreeIPA would be easy to add this to. * We can stop using fasClient in favor of ipa-client setup (no more heavy syncing) * The heavy security aspects will be handled by upstreams we don't need to fully maintain (FreeIPA, sssd, ipsilon, etc). Cons: * We still need to write the CAIAPI/webapp, although Patrick may have CAIAPI already somewhat implemented. * It feels very sad to have spent so long on FAS3 and never deploy it, but sometimes thats just the way things go. ;(

6 7

Fedora and PDC
by Pierre-Yves Chibon 11 Jun '18

11 Jun '18

Good Morning Everyone, We have been informed thst week at the upstream devs working on the product-definition-center (PDC) are moving away from the project and are going to leave it without a maintainer. Since we adopted PDC for a variety of data flows, this puts us in an awkward position. :( Ralph and I met up on Tuesday to brainstorm the list of things we actively use PDC for today and to come up with a contingency plan for how to handle them. One overarching option is for us (fedora-infra) to take ownership of the PDC codebase as a whole. We didn't fully explore this option, figuring that the codebase is large and contains lots of tables, endpoints, and codepaths that we didn't use nor which we plan to use. Instead, below we've got the four things we use PDC for and some options for what to do with each. With the exception of /modules/, one common pattern that we like is to investigate splitting out the "django apps" that make up PDC into their own projects. We're calling these "pdc-lite", for fun. See more below. * Modules The data in the /modules/ PDC endpoint is *also* in the MBS db. Ralph's team is going to just use that and stop using pdc anything for modules. We're going to need to patch pungi, mbs for local builds, and a few other places. This should be a relatively low-pain transition. * Stream branches, branch ownership, retirement dates? - SLA/EOL are currently stored in PDC. - Queried by releng scripts for retirement, fedrepo-req for new branches, etc.. option #1 git repo full of yaml file similar to the override repo compiled into a single JSON blob Single place for all retired packages This feels like the lowest tech option. git gives us change control for free... but people easily get lost in the "UX" of navigating a gigantic git repo full of plaintext files. option #2 pagure's DB/API pagure knows nothing about branches currently, so that would be bigger lift option #3 PDC internally is composed of ~20 "django apps" https://github.com/product-definition-center/product-definition-center/tree… We could pick the 2 or 3 that comprise the branches feature, copy them out, and turn them into their own service: the "branch definition center": BDC. That would be the "pdc-lite" approach mentionned above, ie: PDC with only the "app" of interest * release/life-circle tracking? option #1: PDC lite with just that app of interest option #2: JSON/yaml file on the proxies option #3: pkgdb-lite option #4: ??? compose tracking? impacted: fedfind option #1: PDC-lite with just that app of interest option #2: Drop this entirely? Adam probably really wants to keep the record of composes. option #3: ??? The "pdc-lite" options are attractive, across the board. One thing we get from this is greater clarity when discussing things formerly in PDC. If something is in the branch-definition-center, the compose-definition-center, or the release-definition-center.. you know what you're talking about. Today, when talking about whether or not something should be or is in "PDC", it is easy to get confused. I propose we start the discussion on the list and plan for a meeting sometime late next week to discuss it further with the interested parties (please signal yourself) What do you think? Pierre and Ralph

10 23

FBR: Change to category based crawling
by Adrian Reber 13 May '18

13 May '18

As discussed previously I would like to change the crawler to crawl each category separately. The goal is to reduce the load on the database by distributing the crawling better over the whole day and to reduce the chance of mirrors being disabled because of the high database load. This should also remove the need for mirror administrators to create multiple hosts in MirrorManager to work around the 4 hours timeout per host. Attached is my patch. Please +1. This affects mm-crawler01 and mm-crawler02. Adrian

3 4

Bodhi 3.7.0-0.0.beta deployed to stg
by Randy Barlow 10 May '18

10 May '18

Greetings! I just deployed a Bodhi 3.7.0 beta[0] build to stg[1]. This is a pretty targeted release with only 4 patches, all aimed at improving the developer experience with respect to package gating. Testing is appreciated! [0] https://bodhi.stg.fedoraproject.org/docs/user/release_notes.html#v3-7-0[1] https://bodhi.stg.fedoraproject.org/

2 5

Fedora 28 Final freeze now in effect!
by Kevin Fenzi 02 May '18

02 May '18

Greetings. we are now in the infrastructure freeze leading up to the Fedora 28 release. This is a final release freeze. We do this to ensure that our infrastructure is stable and ready to release the Fedora 28 when it's available. You can see a list of hosts that do not freeze by checking out the ansible repo and running the freezelist script: git clone https://infrastructure.fedoraproject.org/infra/ansible.git ansible/scripts/freezelist -i inventory Any hosts listed as freezes is frozen until 2018-05-01 (or later if release slips). Frozen hosts should have no changes made to them without a sign-off on the change from at least 2 sysadmin-main or rel-eng members, along with (in most cases) a patch of the exact change to be made to this list. Thanks, kevin

1 1

Freeze Break Request: update oz on image builders
by Kevin Fenzi 30 Apr '18

30 Apr '18

Greetings. Rawhide composes have been failing recently due to kernel changes that landed in the 4.17 rc2 (and thus all later) kernels. This change causes vm's made by oz (our cloud image creator) to not have enough randomness and they don't boot up before they hit the timeout and are marked failed. imcleod was kind enough to write a upstream oz patch to have vm's use the hosts passthrough random device: https://github.com/clalancette/oz/pull/260 I have a f27 rpm of an oz package with this patch applied. I would like to update all our image builders, run ansible to make sure they are setup ok and then see if tonight's rawhide finishes. If it works, great. If not, we have no rawhide and have to untag the latest kernel and re-run it. I can't see how this could possibly affect the release tomorrow. ;) +1s? kevin

3 2

FBR: Make bodhi openshift logs a bit more sane
by Patrick Uiterwijk 30 Apr '18

30 Apr '18

This should remove the filename headers in Bodhi's tail of logs. Any +1s? commit dd27d5bd55573c039f2cd809b1f0e124eb051cce (HEAD -> master) Author: Patrick Uiterwijk <puiterwijk(a)redhat.com> Date: Mon Apr 30 17:39:42 2018 +0200 Make bodhi openshift logs slightly more sane Signed-off-by: Patrick Uiterwijk <puiterwijk(a)redhat.com> diff --git a/roles/bodhi2/base/templates/configmap.yml b/roles/bodhi2/base/templates/configmap.yml index fcd95afb5..8221eb4ff 100644 --- a/roles/bodhi2/base/templates/configmap.yml +++ b/roles/bodhi2/base/templates/configmap.yml @@ -36,7 +36,7 @@ data: mkdir /httpdir/run ln -s /etc/httpd/modules /httpdir/modules truncate --size=0 /httpdir/accesslog /httpdir/errorlog - tail -f /httpdir/accesslog /httpdir/errorlog & + tail -qf /httpdir/accesslog /httpdir/errorlog & exec httpd -f /etc/bodhi/httpd.conf -DFOREGROUND -DNO_DETACH httpd.conf: |- Listen 0.0.0.0:8080

3 2

[Release] pagure 4.0 and 4.0.1
by Pierre-Yves Chibon 30 Apr '18

30 Apr '18

Good Morning Everyone, Earlier today we released a new major version of pagure: 4.0. Following it, we were informed about two small bugs, one of which did not impact any of our running pagure instances, the other would have been annoying in dist-git. So we fixed them and released 4.0.1. Here are the two changelogs (flood incoming!): 4.0.1 (2018-04-26) ------------------ - Fix browsing projects in a namespace when logged in and the instance has only one contributor for every projects - Fix commenting on a PR or an issue if the event source server is not configured at all (Slavek Kabrda) 4.0 (2018-04-26) ---------------- - Re-architecture the project to allow potentially extending pagure outside of its core - Fix running the tests on newer pygit - Add a space between the fork and the watch buttons - Add a global configuration option to turn on or off fedmsg notifications for the entire pagure instance - Set the default username to be 'Pagure' when sending git commit notifications by email - Add project setting to show roadmap by default (Vivek Anand) - Explain in the doc where the doc is coming from - Expand and document the tokenization search - Add document that multiple keys are supported - Add a way to block non fast-forwardable commits on all branches - Fix running pagure on docker for development (Clément Verna) - Make the accordeon in the settings page work correctly - Allow calling git blame on a commit instead of a branch - Exclude the .pyc files from all folders - Fix viewing file if the identifier provider is a commit hash - Make pagure-ci use python-jenkins to work with newer Jenkins - Fix the link to the pull-request shown by the default git hook - If the tag's color is the default text, convert it to the hex value - Include documentation on how to pull locally a pull-request on the PR page - Properly retrieve the number of projects and forks users have - Replace jquery.dotdotdot by jquery.expander - Update the Preview button to display 'Edit' when previewing - Fix supporting <link> in markdown as it is supposed to be - Add missing authentication provider option to documentation (Michael Watters) - Fix couple of places where fullname is required while it's not - Let users see and access private tickets they are assigned to - Fix allowing to add multiple tags with the same color - Add a new API endpoint allowing to open new pull-requests - Fix checking if the user is authenticated - Add the possibility to mark milestones as active or inactive - Fix making the milestones showing in the correct order on the issue page - Fix showing the proper URLs in the repo overview - Include the cached merge status in the JSON representation of pull-requests - Improve the fedmsg git hook documentation - Fix display of deleted parent on index page (Lubomír Sedlář) - Adjust message shown to the user deleting a tag off a project - Fix redirecting the user when they remove themselves from a project - Add an option to notify on flags being added to a pull-request - Add an option to notify on flags being added to a commit - Document project intra-pagure hyperlinks - Refresh the PR cache of the parent repo rather than always the current one - Move the webhook service to be a celery service - Fix dead-link due to documentation for python-markdown being moved - Mention #pagure IRC channel in Contributing docs (Peter Oliver) - Fix editing and deleting comments added by the EV server to PRs - Include a count of the number of tickets shown vs recorded for each milestone - Do not try to get the avatar if the author has no email - Fix HTML on settings page - Migrate the logcom service to be celery based and triggered - Link directly to API key settings in error message about expired API key (Peter Oliver) - Drop the constraint on binaryornot - Make fork page header link consistent (Lubomír Sedlář) - Fix the rtd hook and port it to the v2 API (Clément Verna, Pierre-Yves Chibon) - Deduplicate list of contributors to a project (Lubomír Sedlář) - Remove repo from gitolite cache when it gets deleted (Slavek Kabrda) - Make the hooks use the new architecture (Clément Verna) - Switch to comments on PR page when url fragment is reset (Lubomír Sedlář) - Handle implicit issue link at start of line (Adam Williamson) - Don't treat @ in the middle of words as a mention (Adam Williamson) - Improve the CI settings docs (Clément Verna) - Ensure the tasks has finished before checking its results - Fix oidc logout with admin_session_timedout (Slavek Kabrda) - Make images be lazy loaded via javascript - Adjust activity heatmap and logs for timezone (Adam Williamson) - Use timezone not offset for user activity, fix heat map (Adam Williamson) - JS clean up (Lubomír Sedlář) - Fix UnicodeEncode on entering non-ascii password (Farhaan Bukhsh) - Add Tests and exception for non-unicode password (Farhaan Bukhsh) - Forbid adding tags with a slash in their name to a project - Migrate the loadjson service to be celery-based - Specify which service is logging the action for easier debugging/reading of the logs - Merge the fedmsg notifications on commit logic into the default hook - Merge pagure-ci into the pagure's celery-based services - When creating a new PR, allow updating the branch from - Allow pull changes from a different repo than the parent one - Add a new internal endpoint to get the family of a project - Expand the API endpoint listing tags to include the hash if asked t - List the tags of the project in the list of commits - Fix sending notifications in the default hook - Make it possible to use custom PR/commit flags based on instance configuration (Slavek Kabrda) - Show summary of flags on page with commits list (Slavek Kabrda) - Improve the info message when trying to setup an user with a known email - Make badges with flag counts in commits list to links to commit details (Slavek Kabrda) - Enable sending messages to stomp-compliant brokers (Slavek Kabrda) - Update required pygit2 version (Clément Verna) - Do not crash when getting the branches ready for PR on a fork with no parent - Adjust tests for newer flask - Make trigger CI build depends on project name (Clément Verna) - Ensure the DOCS_FOLDER and TICKETS_FOLDER really are optional - Move the `Add Milestone` button near the top and fix the layout - Add a button to delete empty line when adding new tags - Change submit button labels for issues and PRs (Akshay Gaikwad) - Add changelog.rst (Akshay Gaikwad) - Overflow heatmap automatically (Paul W. Frields) - Large unit-tests improvement both in quality and speed (Aurélien Bompard) - Initial support for commit CI trigger (Clément Verna) - Added signed-off-by during web ui commit (yadneshk) - Replace py-bcrypt by python2-bcrypt (Clément Verna) - Fix the user's requests page - Establish an order for readme files (Karsten Hopp) - Include the filename when showing the diff of remote PRs - Specify the parent repo, even when creating a remote PR - Always use md5 to get ssh key information (Patrick Uiterwijk) - Support showing comment submitted by ajax when the SSE is down/not set - Add the possibility to link issues to pull-requests (in the UI) - Rely on the list of branches rather than the ``.empty`` attribute to find out if a git repo is empty or not - Add the possibility to split the tasks into multiple queues - Fix getting the patch of a PR that no longer has a project from - Do not update the CHECKSUMS file if the file was already uploaded - Show the fork button on forks - Make the web-hook field be a textarea and improve the documentation about web-hook - Fix supporting branches containing multiple dots - Do not convert to markdown commit messages in notifications - Port pagure to use the compile-1 script from upstream gitolite (if configured to do so) (Slavek Kabrda) - Add preview when editing a comment (Rahul Bajaj) and the initial comment - Ensure that deployment keys are managed correctly (Michael Watters) - Improve human-readable date/time display in web UI (Adam Williamson) - Make sure we rollback session on task failures (Slavek Kabrda) - Fix new commit notification mails with non-ASCII (#1814) (Adam Williamson) - Don't create gitolite.conf entries for docs and tickets when they're disabled (Slavek Kabrda) - Move source git urls above contibutors list (yadneshk) - Fix private repo to be accessed by ACLs other than admin (Farhaan Bukhsh) - Change the lock name based on the git repo touched (Pierre-Yves Chibon) - Adjust the spec file, remove no longer needed lines and fix requirements (Pierre-Yves Chibon) - Add example worker systemd service file (Pierre-Yves Chibon) - Adjust the wsgi file for the new arch (Pierre-Yves Chibon) - Fix turning the read-only boolean on a fork (Pierre-Yves Chibon) - Support blaming a file is the identifier is a tag (Pierre-Yves Chibon) - Ensure the git hooks are always executable in the rpm (Pierre-Yves Chibon) - Do not syntax highlight 'huge' files (Patrick Uiterwijk) - Fix exceptions caused by missing merge object (Michael Watters) - Fix linking to a PR that was opened from a main project to a fork (Pierre-Yves Chibon) - Add support for repository templates for sources and forks (Pierre-Yves Chibon) - Enable usage of flask-session extension (Slavek Kabrda) - Add a configuration key allowing to send fedmsg notifications on all commits (Pierre-Yves Chibon) - Allow deleting branch when PR is merged (Lubomír Sedlář) If you made it until here, congratulations! Oh and 4.0.1 is currently happily running in stg.pagure.io and src.stg.fp.o :) Happy hacking, Pierre

2 2

Zchunk update
by Jonathan Dieter 30 Apr '18

30 Apr '18

It's been a number of weeks since my last update, so I thought I'd let everyone know where things are at. I've spent most of these last few weeks reworking zchunk's API to make it easier to use and more in line with what other compression tools use, and I'm mostly happy with it now. Writing a simple zchunk file can be done in a few lines of code, while reading one is also simple. I've also added zchunk support to createrepo_c (see https://github.com/jdieter/createrepo_c), but I haven't yet created a pull request because I'm not sure if my current implementation is the best method. My current effort only zchunks primary.xml, filelists.xml and other.xml and doesn't change the sort order. The one area of zchunk that still needs some API work is the download and chunk merge API, and I'm planning to clean that up as I add zchunk support to librepo. Some things I'd still like to add to zchunk: * A python API * GPG signatures in addition to (possibly replacing) overall data checksum * An expiry field? (I'm obviously thinking about signed repodata here) * Tests * More tests * Other arch testing (it's currently only tested on x86_64) I'd welcome any feedback or flames. Jonathan

3 10

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure April 2018