infrastructure May 2016

infrastructure@lists.fedoraproject.org

31 participants
37 discussions

Planned Outage: Koschei update - 2016-05-12 15:00 UTC
by Mikolaj Izdebski 12 May '16

12 May '16

Planned Outage: Koschei update - 2016-05-12 15:00 UTC There will be an outage starting at 2016-05-12 15:00 UTC, which will last approximately 4 hours. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2016-05-12 15:00 UTC' Reason for outage: Koschei will be updated to newer upstream version and reinstalled on new machines. Affected Services: Koschei - continuous integration system Services not listed are not affected by this outage. Contact Information: mizdebsk, msimacek Ticket Link: https://fedorahosted.org/fedora-infrastructure/ticket/5287 Please join #fedora-admin or #fedora-noc on irc.freenode.net or add comments to the ticket for this outage above.

1 0

Celery in Fedora Infra (was: RFC: Fedora Scale-Out Docker Registry
by Pierre-Yves Chibon 11 May '16

11 May '16

Proposal) Reply-To: On Mon, May 09, 2016 at 01:49:26PM -0600, Kevin Fenzi wrote: > We have tried to avoid celery serveral times in the past and always > managed to, but perhaps we can't this time. Is there any alternative to > the celery use? Outside of the topic at hand, do you remember the reasons why we tried to avoid celery until now? Was this something specific to celery? Or AMQP in general? Or? Since I'm investigating changing the architecture of FMN's backend in such a way that it would use both redis and AMQP (rabbitmq to be precise), I'm curious if there is anything I should be aware of before going to far down this road :) Thanks, Pierre

2 1

Plan for tomorrow's Fedora Infrastructure meeting - 2016-05-12
by Kevin Fenzi 11 May '16

11 May '16

The infrastructure team will be having it's weekly meeting tomorrow, 2016-05-12 at 18:00 UTC in #fedora-meeting on the freenode network. We have a gobby document (see: https://fedoraproject.org/wiki/Gobby ) fedora-infrastructure-meeting-next is the document. Please try and review and edit that document before the meeting and we will use it to have our agenda of things to discuss. A copy as of today is included in this email. If you have something to discuss, add the topic to the discussion area with your name. If you would like to teach other folks about some application or setup in our infrastructure, please add that topic and your name to the learn about section. kevin -- = Introduction = This shared document is for the next fedora infrastructure meeting. We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. = Meeting start stuff = #startmeeting Infrastructure (2016-05-12) #meetingname infrastructure #topic aloha #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson #topic New folks introductions / Apprentice feedback = Status / information / Trivia / Announcements = (We put things here we want others on the team to know, but don't need to discuss) (Please use #info <the thing> - your name) #topic announcements and information #info Fedora 24 Beta is out the door! - everyone #info Thus we are no longer in freeze - everyone #info Added memory to mailman01 to hopefully prevent alerts - kevin #info Going to migrate the last of the lists to mailman01 this week - abompard #info most NFS netapp mounts are now moved to nfsv4 - kevin #info Planning to migrate comps and spin-kickstarts to pagure on friday - kevin #info = Things we should discuss = We use this section to bring up discussion topics. Things we want to talk about as a group and come up with some consensus or decision or just brainstorm a problem or issue. If there are none of these we skip this section. (Use #topic your discussion topic - your username) #topic Beta release day issues - kevin #topic Fedorahosted migration / EOL - kevin #topic = Apprentice office hours = Here we will discuss any apprentice questions, try and match up people looking for things to do with things to do, progress, testing anything like that. #topic Apprentice office hours = Learn about some application or setup in infrastructure = (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the info in this section. In the event we don't find someone to teach about something, we skip this section and just move on to open floor.) #topic Learn about: Our various datacenters - kevin = Meeting end stuff = #topic Open Floor #endmeeting

1 0

Freeze Break: fullefiletime list
by Kevin Fenzi 11 May '16

11 May '16

Greetings. There was some talk on the mirror lists about making a fullfilelist version that also had timestamps in it so mirrors could look for all the files that changed since the last time they synced and only sync those specific files. This would be MUCH faster for them as they don't need to pull the entire metadata over. This would be MUCH better for us as they don't have to hammer our netapps to get all the metadata. Of course this is not a 100% solution, as they won't see deleted files or hardlinks. So, they will likely still need to do a real full sync every once in a while (monthly?) to get those things synced up. I'd like to push the following change to create the file and then we can look at writing up some simple scripting for people to use and see if we can get people to use it and lower our netapp metadata hammering. +1s? kevin -- diff --git a/roles/bodhi2/backend/files/update-fullfilelist b/roles/bodhi2/backend/files/update-fullfilelist index 0302c6a..bac3f9c 100755 --- a/roles/bodhi2/backend/files/update-fullfilelist +++ b/roles/bodhi2/backend/files/update-fullfilelist @@ -1,6 +1,6 @@ #!/bin/bash -# currently runs on releng2.fedora.phx.redhat.com +# currently runs on bodhi-backend01 after updates pushes MOD=$1 [ -z "$MOD" ] && { @@ -8,6 +8,8 @@ MOD=$1 exit 1 } +# This is the old traditional fullfilelist with no timestamps + TMPFILE=$(mktemp -p /tmp/) pushd /pub/$MOD > /dev/null find * -print > $TMPFILE @@ -18,3 +20,16 @@ else fi chmod 0644 fullfilelist popd > /dev/null + +# This is the new list with timestamps + +TMPFILE=$(mktemp -p /tmp/) +pushd /pub/$MOD > /dev/null +/usr/bin/rsync --no-h --list-only -r . > $TMPFILE +if diff $TMPFILE fullfiletimelist > /dev/null; then + rm -f $TMPFILE +else + mv $TMPFILE fullfiletimelist +fi +chmod 0644 fullfiletimelist +popd > /dev/null diff --git a/roles/releng/files/update-fullfilelist b/roles/releng/files/update-fullfilelist index 0302c6a..bac3f9c 100755 --- a/roles/releng/files/update-fullfilelist +++ b/roles/releng/files/update-fullfilelist @@ -1,6 +1,6 @@ #!/bin/bash -# currently runs on releng2.fedora.phx.redhat.com +# currently runs on bodhi-backend01 after updates pushes MOD=$1 [ -z "$MOD" ] && { @@ -8,6 +8,8 @@ MOD=$1 exit 1 } +# This is the old traditional fullfilelist with no timestamps + TMPFILE=$(mktemp -p /tmp/) pushd /pub/$MOD > /dev/null find * -print > $TMPFILE @@ -18,3 +20,16 @@ else fi chmod 0644 fullfilelist popd > /dev/null + +# This is the new list with timestamps + +TMPFILE=$(mktemp -p /tmp/) +pushd /pub/$MOD > /dev/null +/usr/bin/rsync --no-h --list-only -r . > $TMPFILE +if diff $TMPFILE fullfiletimelist > /dev/null; then + rm -f $TMPFILE +else + mv $TMPFILE fullfiletimelist +fi +chmod 0644 fullfiletimelist +popd > /dev/null

5 7

Freeze break: increase memory on mailman01
by Kevin Fenzi 11 May '16

11 May '16

Greetings. Over this month so far, we have had mailman01 alert 27 times about swap space being low. When we look, it is indeed low, but there is still lots of memory available, so a swapoff -a && swapon -a "fixes" it. My theory as to what is happening is that from time to time there's a Django thread that takes up lots and lots of memory, causing the machine to swap out other things, then it completes, leaving lots of things swapped out, but memory available. I have in fact seen the web-ui take up a ton of memory. Right now we have the webui set to: maximum-requests=1000 processes=4 threads=4 We may want to adjust that? Perhaps less processes and more threads? However, one simple thing we can do right now is to just throw more memory at it. The virthost it's on has a lot of memory to spare, so it would be trivial to just double it's memory and see if the issue goes away. (from 16GB to 32GB). This would require a quick reboot, but downtime should be very short. Thoughts? +1s? kevin

4 3

Fedora 24 Beta freeze now in effect
by Kevin Fenzi 11 May '16

11 May '16

Greetings. we are now in the infrastructure freeze leading up to the Fedora 24 Beta release. This is a pre-release freeze. We do this to ensure that our infrastructure is stable and ready to release the Fedora 24 Beta when it's available. You can see a list of hosts that do not freeze by checking out the ansible repo and running the freezelist script: git clone https://infrastructure.fedoraproject.org/infra/ansible.git scripts/freezelist -i inventory Any hosts listed as freezes is frozen until 2016-05-03. (or later if Beta slips). Frozen hosts should have no changes made to them without a sign-off on the change from at least 2 sysadmin-main or rel-eng members, along with (in most cases) a patch of the exact change to be made to this list. Thanks, kevin

1 1

Backups for piwik cloud instance
by Ricky Elrod 10 May '16

10 May '16

As some of you know, some of us (mostly smooge and I) have been working on statistics gathering for Fedora web infra as of late. We are working on setting up a piwik instance to be used on our websites. However, this instance lives in the cloud and thus can't hit the VPN. A question I ran into and was told to email the list about and/or bring up at the meeting, is: What should our backups story look like for such cases? Mostly, we have a MySQL database on this node that will get really big fairly quickly (right now /var/lib/mysql is about 1.1GB and piwik only has one site added, and it's only been there for a few weeks). Are other cloud instances backed up in any way, and if so how was it done? If not, how do we go about coming up with a plan for doing this, for this instance and other instances in the future that end up having similar requirements? (Smooge can weigh in if I'm forgetting any important details here). -Ricky

3 3

Test email --please disregard
by Corey W Sheldon 09 May '16

09 May '16

Test -- Corey Sheldon PGP (64b): 8B4E89435A88E539,59276298D2264944 P: +1-310.909.7672 Ameridea , FedoraProject Full Contact Info: https://gist.github.com/linux-modder/ac5dc6fa211315c633c9 Disclaimer: All contents of this and any threaded correspondence should be deemed confidential and privileged.

1 0

OAuth2 + OpenID Connect: patches incoming
by Patrick Uiterwijk 09 May '16

09 May '16

Hi all, As some of you might know, Ipsilon, the identity provider that we use for id.fedoraproject.org, has just merged its OAuth2 + OpenID connect support. I will be rolling a new Ipsilon release in the forseeable future, and I hope to get this deployed into at least staging between Beta F24 and Final freeze. This email is mostly meant to inform people that I will be sending patches to various projects in use at Fedora Infrastructure to add support for this for authentication to the API and web interface. While I have a set of patches ready for certain applications, others might take some time to write, so if you would like to get started with the migration and don't want to wait for me to port, feel free to get in touch with me and I can help you get started. I will send more information as I get further in the process. Regards, Patrick Uiterwijk

1 0

Last of mailing lists migration
by Kevin Fenzi 06 May '16

06 May '16

Greetings. I thought I would start a thread here about the end part of our mailman3 migration plans. There's currently just 4 lists left in mailman2. One of those is package-announce that uses 'topics' pretty heavily and unfortunately, mailman3 still lacks topics support. I don't want to wait until it's implemented since we have no idea when that might be. So, I have a blog post ready that discusses topics support and alternatives to get that information, as well as a bit of stuff added to the wiki: https://fedoraproject.org/wiki/Mailman3_Migration I can post that blog post a few days before the migration? Or just publish it now? I'd like to complete the migration after Beta is out the door and we are out of freeze. Either May 11th or May 13th would be fine with me. We would migrate the last lists, then we need to change things to proxy everything to mailman01 and take collab03 out of the mix entirely. If all looks well after a week or so, we can retire collab03. Thoughts? (especially from Aurélien :) kevin

3 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure May 2016