infrastructure August 2016

infrastructure@lists.fedoraproject.org

44 participants
44 discussions

About JS framework
by Pierre-Yves Chibon 14 Sep '22

14 Sep '22

Good Morning Everyone, Our infrastructure is mostly a python store, meaning almost all our apps are written in python and most using wsgi. However in python we are using a number of framework: * flask for most * pyramid for some of the biggest (bodhi, FAS3) * Django (askbot, Hyperkitty) * TurboGears2 (fedora-packages) * aiohttp (python3, async app: mdapi) While this makes sometime things difficult, these are fairly standard framework and most of our developers are able to help on all. However, as I see us starting to look at JS for some of our apps (fedora-hubs, wartaa...), I wonder if we could start the discussion early about the different framework and eventually see if we can unify around one. This would also allow those of us not familiar with any JS framework to look at the recommended one instead of picking one up semi-randomly. So has anyone experience with one or more JS framework? Do you have one that would you recommend? Why? Thanks for your inputs, Pierre

15 19

Proposal to mirror Docker images
by Randy Barlow 27 Oct '16

27 Oct '16

During Flock 2016, I had the opportunity to talk with Adam Miller, Dennis Gilmore, and Pierre-Yves Chibon about the technical challenges with distributing Docker images with Fedora's extensive mirror network. These conversations helped me to solidify a proposal for how Fedora could solve this problem, outlined below. High level view =============== In summary, the proposal is to write a patch for the docker client that will give it the capability to accept metalink responses upon docker pull operations. We would also need to add support for Docker images to mirror list and mirror manager. Additionally, we will need a small tool to pull the content to be mirrored out of a docker registry and write them to disk in a format that can be mirrored, as well as some Ansible code to run the tool when there is new content to be mirrored. Background ========== The Fedora project wishes to begin distributing new types of content than it has in the past. One of the types that has been identified as a goal is the Docker image. Adam Miller has already done the work that will allow packagers to build Docker images, but we still need a way to distribute those builds to Fedora's users. Adam Miller's implementation helpfully drops the builds we want into a Docker registry. Proposed Changes ================ Mirror List ----------- Users will be pointing their docker clients at Mirror List when they docker pull Fedora's Docker images. In order for this to work, we will need to make two changes to Mirror List so that it can respond to the docker client properly. The first change is that Mirror List will need to respond with a special header and a body of "{}" when the docker client sends a GET request for /v2/. The second change is that it will need to return a metalink document when the client makes additional requests so that the clients can be redirected to a list of mirrors near their locations, just as it does with the dnf client today. The docker client typically connects to port 5000. We could run a second instance of Mirror List on port 5000 if we wanted to isolate it from the current instance. We can also have the docker client pull from 443 as dnf does if we want to keep the deployment simpler. Mirror Manager -------------- We will need to make a few changes to Mirror Manager as well. We will need to provide an interface to allow mirror admins to opt in/out of mirroring Docker content. We will also need to modify the curler to detect whether a given mirror is up to date or not. We will need to make sure that UMDL is updated when content changes. docker ------ The most significant work required will likely be modifying the docker client to enable it to properly handle the metalink responses it will be receiving from Mirror List. When requesting the manifest, it will receive a metalink document that will give it a priority ordered list of mirrors. It will need to work through the list in order until it reaches a mirror that has the correct checksum for the requested manifest. It will then use that same mirror for the subsequent blob requests. There is some concern that such a feature would not be accepted by the upstream docker project. If we were to proceed with this proposal, we would propose this patch to the upstream Docker project. If upstream were not willing to accept the feature, we would need to have the Fedora docker packager carry this patch as a downstream add on. New Tool -------- The last piece that is needed is a tool that can create the filesystem tree that we want to synchronize out to the mirrors. The mirrors only need to carry manifests and blobs, so the tool needs only to pull these documents out of the registry that Adam Miller has set up and write them to disk in a particular structure. For optimization, we could use hardlinks for blobs that are common across the various images (for example, the Fedora base blob will be the same in all images) to save rsync time and mirror disk space. Additionally, we will need a playbook to run this new tool in response to fedmsgs. We may be able to use Adam Miller's loopabull project to run such a playbook at the right times. Conclusion ========== Thanks for reading, and please respond with any comments or questions you have about this proposal. I'm happy to clarify any points further, and if you have any alternative proposals I'd love to hear those as well.

12 26

[release] pagure: 2.4
by Pierre-Yves Chibon 05 Sep '16

05 Sep '16

Good Morning Everyone, This morning I cut a new release of pagure: 2.4 It had been a while since the last one and I felt we were at a right time for this. There are more changes coming, some of which will be disruptive so I wanted to cut a release before them. Here is the corresponding changelog: * Wed Aug 31 2016 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 2.4-1 - Update to 2.4 - - [Security] Avoid all html related mimetypes and force the download if any (CVE-2016-1000037) -- Fixed in 2.3.4 as well - Redirect the URL to projects <foo>.git to <foo> (Abhishek Goswami) - Allow creating projects with 40 chars length name on newer pagure instances - Fix @<user> and #<id> when editing a comment (Eric Barbour) - Display properly and nicely the ACLs of the API tokens (Lubomír Sedlář) - Removing html5lib so bleach installation finds what version is best (Tiago M. Vieira) - Remove the branchchooser from the repoheader (again) (Ryan Lerch) - Fix hard-coded urls in the master template - Made the interaction with the watch button clearer (Ryan Lerch) - Introduce pagure-ci, a service allowing to integrate pagure with a jenkins instance (Farhaan Bukhsh and I) - Accept Close{,s,d} in the same way as Merges and Fixes (Patrick Uiterwijk) - Avoid showing the 'New PR' button on the overview page is a PR already exists for this branch, in the main project or a fork (Vivek Anand) - Fix presenting the readme file and display the readme in the tree page if there is one in the folder displayed (Ryan Lerch) - Move the new issue button to be available on every page (AnjaliPardeshi) - Fix pagure for when an user enters a comment containing #<id> where the id isn't found in the db - Make the bootstrap URLs configurable (so that they don't necessarily point to the Fedora infra) (Farhaan Bukhsh) - Fix how the web-hook server determine the project and its username - Replace the login icon with plain text (Ryan Lerch) - Fix layout in the doc (Farhaan Bukhsh) - Improve the load_from_disk utility script - Fix our mardown processor to avoid crashing on #<text> (where we expect #<id>) - Fix the search for projects with a / in their names - Fix adding a file to a ticket when running pagure with `local` auth - Improve the grammar around the allowed prefix in our fake-namespaces (Jason Tibbitts) - Implement scanning of attached files for viruses (Patrick Uiterwijk) - Document how to set-up multiple ssh keys per user (William Moreno Reyes) - Add display_name and description to groups, and allow editing them - Add the ability to run the post-receive hook after merging a PR in the UI - Fix showing the group page even when user management is turned off (Vivek Anand) - Make explicit what the separators for tags is (Farhaan Bukhsh) - Include the word setting with icon (tenstormavi) - Fix the requirements.txt file (Vivek Anand) - Cleaned up the topbar a bit (Ryan Lerch) - Fix location of bottom pagination links on user page (Ryan Lerch) - Add user's project watch list in index page of the user (Vivek Anand) - Fix showing the reporter when listing the closed issues (Vivek Anand) - Fix accessing forks once the main repo has been deleted (Farhaan Bukhsh) As you can see many changes from many people, thanks to everyone who helped and worked hard to make this release. This release has been pushed to staging and will not be pushed to prod before next week. Happy hacking, Pierre

1 1

Fedora RelEng Automation
by Adam Miller 01 Sep '16

01 Sep '16

Hello all, I've been doing some work on the Fedora RelEng Automation[0][1] lately and I've gotten to a point that I have some Ansible playbooks that I'd like to test but I need to be able to run them somewhere in the Fedora Infrastructure as they will need to be able to touch various Release Engineering related systems (at least compose machines to begin with). I'd like advisement from the Infrastructure Team on how best to move forward with this. I'm happy to do the leg work, I just need some guidance to make sure I'm complying with Infra standards. Thank you, -AdamM [0] - https://pagure.io/releng-automation [1] - https://fedoraproject.org/wiki/Changes/ReleaseEngineeringAutomationWorkflow…

2 2

Mirrormanager doesn't provide F25 x86_64 debuginfo mirror for updates repository
by Christian Stadelmann 01 Sep '16

01 Sep '16

I'm trying to report a bug in a package (gnome-settings-daemon) and it would be useful to have debuginfo for this package. When running `dnf debuginfo-install gnome-settings-daemon`, I'm getting an error that the repomd.xml was not found in metalink. Therefore I manually downloaded the metalink file from https://mirrors.fedoraproject.org/metalink?repo=updates-released-debug-f25&… (URL from /etc/yum.repos.d/fedora-updates.repo file) and had a look at the file. The file itself just contains this content: <metalink version="3.0" xmlns="http://www.metalinker.org/" type="dynamic" pubdate="Wed, 31 Aug 2016 21:05:11 GMT" generator="mirrormanager" xmlns:mm0="http://fedorahosted.org/mirrormanager">  </metalink> but the file does not contain a line like # repo=updates-released-debug-f25&arch=x86_64 but it contains everything else: different architectures: # repo=updates-released-debug-f25&arch=aarch64 # repo=updates-released-debug-f25&arch=ppc64 # repo=updates-released-debug-f25&arch=ppc64le # repo=updates-released-debug-f25&arch=s390x different repos: … # repo=fedora-debug-25&arch=x86_64 … # repo=updates-released-f25&arch=x86_64 … # repo=updates-released-source-f25&arch=source … # repo=updates-testing-debug-f25&arch=x86_64 … the repo I'd need is just missing. I know of https://apps.fedoraproject.org/packages/mirrormanager/ and bugzilla and http://fedorahosted.org/mirrormanager, but those are for the package respectively its distribution through fedora, not for the infrastructure on mirrors.fedoraproject.org. I hope I'm right here.

2 2

Plan for tomorrow's Fedora Infrastructure meeting - 2016-09-01
by Kevin Fenzi 31 Aug '16

31 Aug '16

The infrastructure team will be having it's weekly meeting tomorrow, 2016-09-01 at 18:00 UTC in #fedora-meeting on the freenode network. We have a gobby document (see: https://fedoraproject.org/wiki/Gobby ) fedora-infrastructure-meeting-next is the document. Please try and review and edit that document before the meeting and we will use it to have our agenda of things to discuss. A copy as of today is included in this email. If you have something to discuss, add the topic to the discussion area with your name. If you would like to teach other folks about some application or setup in our infrastructure, please add that topic and your name to the learn about section. kevin -- = Introduction = This shared document is for the next fedora infrastructure meeting. We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. = Meeting start stuff = #startmeeting Infrastructure (2016-09-01) #meetingname infrastructure #topic aloha #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson #topic New folks introductions = Status / information / Trivia / Announcements = (We put things here we want others on the team to know, but don't need to discuss) (Please use #info <the thing> - your name) #topic announcements and information #info Fedora 25 alpha is out the door! - everyone #info We are now out of freeze until Beta - everyone #info new koschei in production on Fedora 24 - mizdebsk #info openvpn links moved to higher/better encryption - kevin #info ssl config consolidated, everything using high grade - patrick #info various small changes made post freeze - kevin #info = Things we should discuss = We use this section to bring up discussion topics. Things we want to talk about as a group and come up with some consensus or decision or just brainstorm a problem or issue. If there are none of these we skip this section. (Use #topic your discussion topic - your username) #topic fedorahosted retirement - kevin * what to do about lists.fedorahosted.org * what to do about freemedia * Other "special" projects? * When to send announcements #topic = Apprentice office hours = #topic Apprentice Open office hours Here we will discuss any apprentice questions, try and match up people looking for things to do with things to do, progress, testing anything like that. = Learn about some application or setup in infrastructure = (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the info in this section. In the event we don't find someone to teach about something, we skip this section and just move on to open floor.) #topic Learn about: = Meeting end stuff = #topic Open Floor #endmeeting

1 0

Fedora 25 Alpha Freeze now in effect
by Kevin Fenzi 31 Aug '16

31 Aug '16

Greetings. we are now in the infrastructure freeze leading up to the Fedora 25 Alpha release. This is a pre-release freeze. We do this to ensure that our infrastructure is stable and ready to release the Fedora 25 Alpha when it's available. You can see a list of hosts that do not freeze by checking out the ansible repo and running the freezelist script: git clone https://infrastructure.fedoraproject.org/infra/ansible.git scripts/freezelist -i inventory Any hosts listed as freezes is frozen until 2016-08-23 (or later if Alpha slips). Frozen hosts should have no changes made to them without a sign-off on the change from at least 2 sysadmin-main or rel-eng members, along with (in most cases) a patch of the exact change to be made to this list. Thanks, kevin

1 1

[release] fedimg: 0.7
by Sayan Chowdhury 29 Aug '16

29 Aug '16

0.7 --- Pull Requests - (@ralphbean) #41, Setup logging for cronjob https://github.com/fedora-infra/fedimg/pull/41 - (@coolsvap) #44, Update typos https://github.com/fedora-infra/fedimg/pull/44 - (@nishant-mor) #46, Dropped 'os' and 'ver' from the AWS_AMIS config https://github.com/fedora-infra/fedimg/pull/46 - (@ralphbean) #47, Pungi4 fixes. https://github.com/fedora-infra/fedimg/pull/47 - (@ralphbean) #49, Add a nice log statement at the beginning stating what we're going to upload. https://github.com/fedora-infra/fedimg/pull/49 - (@sayanchowdhury) #50, Fix to include nightly atomic uploads https://github.com/fedora-infra/fedimg/pull/50 - (@sayanchowdhury) #51, Migrate fedimg to compose based https://github.com/fedora-infra/fedimg/pull/51 - (@sayanchowdhury) #52, Send image raw_url to fedmsg instead of the build_name https://github.com/fedora-infra/fedimg/pull/52 Commits - 60aa36b2a Setup logging for cronjob https://github.com/fedora-infra/fedimg/commit/60aa36b2a - 511497384 Update typo in GCE service https://github.com/fedora-infra/fedimg/commit/511497384 - 6b9c3210d Update typo in rackspace service https://github.com/fedora-infra/fedimg/commit/6b9c3210d - f470cebef Update typo in hp service https://github.com/fedora-infra/fedimg/commit/f470cebef - 5a1c7ab51 Dropped 'os' and 'ver' from the AWS_AMIS config https://github.com/fedora-infra/fedimg/commit/5a1c7ab51 - 05452ed71 ex2.py : Added new format of AWS_AMIS config https://github.com/fedora-infra/fedimg/commit/05452ed71 - 20805fdd9 s/yum/dnf/ https://github.com/fedora-infra/fedimg/commit/20805fdd9 - aec998075 Pungi4 fixes. https://github.com/fedora-infra/fedimg/commit/aec998075 - 9d4873858 Add a nice log statement at the beginning stating what we're going to upload. https://github.com/fedora-infra/fedimg/commit/9d4873858 - 156190880 Fix to include F24 nightly atomic uploads https://github.com/fedora-infra/fedimg/commit/156190880 - 335d2236a Migrate fedimg from koji-based to compose-based https://github.com/fedora-infra/fedimg/commit/335d2236a - 7ae44d715 Minor fixes in the fedmsg consumer https://github.com/fedora-infra/fedimg/commit/7ae44d715 - a3a2300ab Change KojiConsumer to FedimgConsumer https://github.com/fedora-infra/fedimg/commit/a3a2300ab - 1d0af12c1 Update the documenation to install fedfind while setting up https://github.com/fedora-infra/fedimg/commit/1d0af12c1 - 0e199c95d An small indentation typo resulting into major issue https://github.com/fedora-infra/fedimg/commit/0e199c95d - 4f9e932f3 Send image raw_url to fedimg instead of the build_name https://github.com/fedora-infra/fedimg/commit/4f9e932f3 -- Sayan Chowdhury <https://sayanchowdhury.dgplug.org/> Senior Software Engineer, Fedora Engineering - Emerging Platform GPG Fingerprint : 0F16 E841 E517 225C 7D13 AB3C B023 9931 9CD0 5C8B Proud to work at The Open Organization!

1 0

Freeze break request: Update Autocloud tests
by Kushal Das 25 Aug '16

25 Aug '16

Hi, We want to introduce new Autocloud test cases. This will not require change in any configure or package version. But it will create a new tarball of testcases for Autocloud which is accessible at [1]. I would also request to decide if we can keep the tests outside of the release freeze so that we can add/update the tests as required. [1] http://infrastructure.fedoraproject.org/infra/autocloud/tunirtests.tar.gz Kushal -- Fedora Cloud Engineer CPython Core Developer https://kushaldas.in https://dgplug.org

4 7

Plan for tomorrow's Fedora Infrastructure meeting - 2016-08-25
by Kevin Fenzi 24 Aug '16

24 Aug '16

The infrastructure team will be having it's weekly meeting tomorrow, 2016-08-25 at 18:00 UTC in #fedora-meeting on the freenode network. We have a gobby document (see: https://fedoraproject.org/wiki/Gobby ) fedora-infrastructure-meeting-next is the document. Please try and review and edit that document before the meeting and we will use it to have our agenda of things to discuss. A copy as of today is included in this email. If you have something to discuss, add the topic to the discussion area with your name. If you would like to teach other folks about some application or setup in our infrastructure, please add that topic and your name to the learn about section. kevin -- = Introduction = This shared document is for the next fedora infrastructure meeting. We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. = Meeting start stuff = #startmeeting Infrastructure (2016-08-25) #meetingname infrastructure #topic aloha #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson #topic New folks introductions = Status / information / Trivia / Announcements = (We put things here we want others on the team to know, but don't need to discuss) (Please use #info <the thing> - your name) #topic announcements and information #info commops cloud instance fixed up for ssh port changes - kevin #info copr backend volume fscked, hopefully stable now - kevin #info Bunch of alpha issues found and sorted, hopefully go this week - kevin #info freemedia updated for f24 - kevin #info = Things we should discuss = We use this section to bring up discussion topics. Things we want to talk about as a group and come up with some consensus or decision or just brainstorm a problem or issue. If there are none of these we skip this section. (Use #topic your discussion topic - your username) #topic flock workshop followup - kevin #topic "official" announcement on Trac sunsetting to Pagure - jflory7 #idea To help encourage migration discussion in teams, a post on Community Blog about plans to sunset Trac in favor of Pagure to raise awareness? = Apprentice office hours = #topic Apprentice Open office hours Here we will discuss any apprentice questions, try and match up people looking for things to do with things to do, progress, testing anything like that. #info jflory7 would like to pursue becoming a member of sysadmin-badges - interested to know more! = Learn about some application or setup in infrastructure = (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the info in this section. In the event we don't find someone to teach about something, we skip this section and just move on to open floor.) #topic Learn about: = Meeting end stuff = #topic Open Floor #endmeeting

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure August 2016