infrastructure November 2017

infrastructure@lists.fedoraproject.org

42 participants
53 discussions

Freeze Break Request: Fix FMN notifications for package (co)maintainers
by Jeremy Cline 06 Nov '17

06 Nov '17

Hey folks, The latest version of FMN in production includes a patch[0] that breaks all the rules that query for package watchers, resulting in this[1] infrastructure issue. There's an open PR[2] on FMN that fixes the issue (reviews welcome). To fix this we have two options. The first is to backport it to the current version in production (1.5) which should be trivial since nothing in this area has been touched in 2.0. We can then update production and carry on. The second option is to update production to 2.0 now (I've included [2] as a patch in the RPM currently in stage). 2.0 includes a re-write of the back-end components of FMN to use Celery. It's running in stage now. Things to note about this: * The FMN back-end now requires F26 because of celery versions. * The FMN front-end is currently still on RHEL7, but I haven't updated it in stage yet so I don't know if there's any adjustments necessary for that (the front-end doesn't use celery so the fact that it's old _shouldn't_ be a problem). * Some care will need to be taken to switch over AMQP queue-wise, especially because the current FMN queues are jammed with unformatable messages it keeps requeuing (about 25K of them). We could also just cut our losses and drop these. * The scripts that monitor queue length will need to be adjusted since there are more queues now and existing queues have been renamed. One thing to note is that we're going to have to go through all those things above at some point anyway. FMN also doesn't really have anything to do with the release process so if it all goes south during the freeze it shouldn't matter. I don't have a preference one way or the other, really. Whatever makes the admins happy makes me happy. [0] https://github.com/fedora-infra/fmn/pull/206 [1] https://pagure.io/fedora-infrastructure/issue/6462 [2] https://github.com/fedora-infra/fmn/pull/248 -- Jeremy Cline XMPP: jeremy(a)jcline.org IRC: jcline

2 6

FBR: Sync out drpms
by Patrick Uiterwijk 04 Nov '17

04 Nov '17

Hi, Can I get +1s to update the updates sync script to sync out Pungi-generated drpms? Patrick diff --git a/roles/bodhi2/backend/files/new-updates-sync b/roles/bodhi2/backend/files/new-updates-sync index e3439a0a0..c1c4378d5 100755 --- a/roles/bodhi2/backend/files/new-updates-sync +++ b/roles/bodhi2/backend/files/new-updates-sync @@ -205,6 +205,8 @@ def sync_single_repo_arch(release, repo, arch, dest_path): results.append(rsync(os.path.join(source_path, maindir, 'Packages'), os.path.join(dest_path))) + results.append(rsync(os.path.join(source_path, maindir, 'drpms'), + os.path.join(dest_path))) if arch != 'source': results.append(rsync(os.path.join(source_path, 'debug', 'tree', 'Packages'), os.path.join(dest_path, 'debug'))) @@ -220,6 +222,9 @@ def sync_single_repo_arch(release, repo, arch, dest_path): results.append(rsync(os.path.join(source_path, maindir, 'Packages'), os.path.join(dest_path), delete=True)) + results.append(rsync(os.path.join(source_path, maindir, 'drpms'), + os.path.join(dest_path)), + delete=True) return collect_stats(results)

3 2

FBR: Fix debuginfo repodata
by Patrick Uiterwijk 04 Nov '17

04 Nov '17

Hi, Turns out my bodhi new-updates-sync script has a silly bug, ending the debug repodata in http://dl.fedoraproject.org/pub/fedora/linux/updates/26/x86_64/debug/repoda…. Can I get +1s for the following patch, which will fix that? Thanks, Patrick diff --git a/roles/bodhi2/backend/files/new-updates-sync b/roles/bodhi2/backend/files/new-updates-sync index b7ff2c8..e3439a0 100755 --- a/roles/bodhi2/backend/files/new-updates-sync +++ b/roles/bodhi2/backend/files/new-updates-sync @@ -209,7 +209,7 @@ def sync_single_repo_arch(release, repo, arch, dest_path): results.append(rsync(os.path.join(source_path, 'debug', 'tree', 'Packages'), os.path.join(dest_path, 'debug'))) results.append(rsync(os.path.join(source_path, 'debug', 'tree', 'repodata'), - os.path.join(dest_path, 'debug', 'repodata'), + os.path.join(dest_path, 'debug'), delete=True)) results.append(rsync(os.path.join(source_path, 'debug', 'tree', 'Packages'), os.path.join(dest_path, 'debug'),

3 2

FBR: Enable delta RPMs in Pungi+Bodhi
by Patrick Uiterwijk 02 Nov '17

02 Nov '17

Seems that this is disabled by default. Any +1s? diff --git a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 index 140f3438e..1c6dc02f2 100644 --- a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 +++ b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 @@ -55,6 +55,9 @@ repoclosure_backend = 'dnf' # CREATEREPO createrepo_c = True createrepo_checksum = 'sha256' +createrepo_deltas = [ + ('^Everything$', {'*': True}) +] # CHECKSUMS media_checksums = ['sha256']

4 4

Changes in hardware in December affecting QA servers
by Stephen John Smoogen 02 Nov '17

02 Nov '17

I realized I had only informally gone over some of the items which will be changing in December when hardware is moved. However the date is finally arriving so it needs to be laid out more formally :). In December, Infrastructure will be moving to new racks with better power monitoring, networking, and other tools. One of the side goals of this is that any old hardware which has been meant to be recycled over the years is to be done so. Which brings us to a set of qa systems which were listed to be replaced a couple of years ago. qa01, qa02, qa03, qa04, qa05, qa07, qa08 are all IBM systems which have reached the end of the road and are planned not to be moved over to the new racks. [They have been scheduled to be removed for 2-3 years now but it has finally come to that date.] What processes are currently running on these systems and what is needed to replace them? -- Stephen J Smoogen.

2 1

[FBR] support msi binaries released on pagure.io
by Pierre-Yves Chibon 02 Nov '17

02 Nov '17

Good Morning Everyone, There was a report that releases.pagure.org doesn't serve .msi binaries properly. I believe this diff should fix it: roles/pagure/frontend/templates/0_pagure.conf | 2 ++ 1 file changed, 2 insertions(+) diff --git a/roles/pagure/frontend/templates/0_pagure.conf b/roles/pagure/frontend/templates/0_pagure.conf index 4b76d20..8b3cda8 100644 --- a/roles/pagure/frontend/templates/0_pagure.conf +++ b/roles/pagure/frontend/templates/0_pagure.conf @@ -175,6 +175,8 @@ WSGIDaemonProcess paguredocs user=git group=git maximum-requests=1000 display-na ServerName releases.pagure.org {% endif %} + AddType application/octet-stream msi + <Directory /> Options +Indexes IndexOptions NameWidth=* Thoughts/+1s? Thanks, Pierre

4 3

Freeze Break Request: fix typo in resultsdb public db dump copy script.
by Kevin Fenzi 02 Nov '17

02 Nov '17

Greetings. Before freeze I added resultsdb database dump to one of the ones we make public. However, I had a small mistake in the script, so it wasn't making them public. This should be a very simple change and I cannot imagine how it would impact anything else. :) +1s? kevin -- diff --git a/scripts/public-db-copy b/scripts/public-db-copy index dc0300f..6bfd9a9 100755 --- a/scripts/public-db-copy +++ b/scripts/public-db-copy @@ -16,5 +16,5 @@ scp db01.phx2.fedoraproject.org:/backups/mailman-$(date +%F).dump.xz /srv/web/in scp db01.phx2.fedoraproject.org:/backups/mbs-$(date +%F).dump.xz /srv/web/infra/db-dumps/mbs.dump.xz scp db01.phx2.fedoraproject.org:/backups/odcs-$(date +%F).dump.xz /srv/web/infra/db-dumps/odcs.dump.xz scp db01.phx2.fedoraproject.org:/backups/hyperkitty-$(date +%F).dump.xz /srv/web/infra/db-dumps/hyperkitty.dump.xz -scp db-qa01.qa.fedoraproject.org:/resultsdb-$(date +%F).dump.xz /srv/web/infra/db-dumps/resultsdb.dump.xz +scp db-qa01.qa.fedoraproject.org:/backups/resultsdb-$(date +%F).dump.xz /srv/web/infra/db-dumps/resultsdb.dump.xz scp db01.phx2.fedoraproject.org:/backups/waiverdb-$(date +%F).dump.xz /srv/web/infra/db-dumps/waiverdb.dump.xz

3 2

Higher quota for Copr
by Miroslav Suchý 02 Nov '17

02 Nov '17

Hi, Copr is using up to 29 x86 VMs and 10 ppc VMs from Fedora Cloud for builders. We are hitting the limits quite often. So I would like to ask if I can assign another 10 x86 VMs to Copr? I can do it myself, I am just asking in advance if anyone is against it. I checked the hypervisors and there is still plenty of space. VCPU Usage: Used 400 of 512 Memory Usage: Used 869.0GB of 1.6TB One Copr builder has 2VCPU and 5GB RAM. Mirek

2 2

Fedora Infrastructure Meeting 2017-11-02 1800 UTC
by Stephen John Smoogen 01 Nov '17

01 Nov '17

The infrastructure team will be having it's weekly meeting tomorrow, 2017-11-02 at 18:00 UTC in #fedora-meeting on the freenode network. We have a gobby document (see: https://fedoraproject.org/wiki/Gobby ) fedora-infrastructure-meeting-next is the document. == This shared document is for the next fedora infrastructure meeting. = Introduction = We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. = Meeting start stuff = #startmeeting Infrastructure (2017-11-02) #meetingname infrastructure #topic aloha #chair smooge relrod nirik dgilmore threebean pingou puiterwijk pbrobinson maxamillion = Let new people say hello = #topic New folks introductions #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves = Status / Information / Trivia / Announcements = (We put things here we want others on the team to know, but don't need to discuss) (Please use #info <the thing> - your name) #topic announcements and information #info PHX2 Colo Trip, Dec 4th - 9th #info Infrastructure will be down during that trip. #info Final freeze has started. PLEASE TEST ANY RCs. #info EU daylight savings Oct 29/ US daylight savings Nov 5 #info new notifs in staging, please test! #info = Things we should discuss = We use this section to bring up discussion topics. Things we want to talk about as a group and come up with some consensus /suor decision or just brainstorm a problem or issue. If there are none of these we skip this section. (Use #topic your discussion topic - your username) #topic Ticket cleanup #info none this week. #topic Apprentice work day - reschedule - kevin #topic = Apprentice office hours = #topic Apprentice Open office hours Here we will discuss any apprentice questions, try and match up people looking for things to do with things to do, progress, testing anything like that. = Learn about some application or setup in infrastructure = (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the i/nfo in this section. In the event we don't find someone to teach about something, we skip this section and just move on to open floor.) #topic Learn about: #info none this week = Meeting end stuff = #topic Open Floor #endmeeting -- Stephen J Smoogen.

1 0

Freeze Break Request: fix purge-ami's cron
by Kevin Fenzi 01 Nov '17

01 Nov '17

Greetings. I happened to notice some cron output a while back and looked into it and found a slight mistake in the purge-ami's cron. 'git clone -n' was being used which tells it to not bother checking out HEAD after the clone. Just leave the checkout empty. Then it tries to run things from the checkout that just fails. :) The solution is just to drop the -n I think, but I'd love to have sayan confirm. kevin -- diff --git a/roles/releng/files/purge-amis b/roles/releng/files/purge-amis index b49942a..be677e5 100644 --- a/roles/releng/files/purge-amis +++ b/roles/releng/files/purge-amis @@ -1,5 +1,5 @@ # Remove launch permissions from the AMIs after every 5 days -0 0 */5 * * root TMPDIR=`mktemp -d /tmp/purgeamis.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/releng.git && cd releng/scripts && AWS_ACCESS_KEY={{ ec2_image_delete_access_key_id }} AWS_SECRET_ACCESS_KEY={{ ec2_image_delete_access_key }} ./clean-amis.py --change-perms && popd && rm -rf $TMPDIR +0 0 */5 * * root TMPDIR=`mktemp -d /tmp/purgeamis.XXXXXX` && pushd $TMPDIR && git clone https://pagure.io/releng.git && cd releng/scripts && AWS_ACCESS_KEY={{ ec2_image_delete_access_key_id }} AWS_SECRET_ACCESS_KEY={{ ec2_image_delete_access_key }} ./clean-amis.py --change-perms && popd && rm -rf $TMPDIR # Delete the AMIs after every 10 days. -0 0 */10 * * root TMPDIR=`mktemp -d /tmp/purgeamis.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/releng.git && cd releng/scripts && AWS_ACCESS_KEY={{ ec2_image_delete_access_key_id }} AWS_SECRET_ACCESS_KEY={{ ec2_image_delete_access_key }} ./clean-amis.py --delete && popd && rm -rf $TMPDIR +0 0 */10 * * root TMPDIR=`mktemp -d /tmp/purgeamis.XXXXXX` && pushd $TMPDIR && git clone https://pagure.io/releng.git && cd releng/scripts && AWS_ACCESS_KEY={{ ec2_image_delete_access_key_id }} AWS_SECRET_ACCESS_KEY={{ ec2_image_delete_access_key }} ./clean-amis.py --delete && popd && rm -rf $TMPDIR

4 3

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure November 2017