infrastructure April 2015

infrastructure@lists.fedoraproject.org

46 participants
79 discussions

Freeze break: exclude session data from db dumps
by Kevin Fenzi 02 Apr '15

02 Apr '15

It was noted yesterday that the koji db dump we have published contained the 'sessions' table in it. Sessions are cleared every 24 hours (a cron job on koji02) and also only contain a session key, which is tied to a specific IP address. While the exposure is small/close to 0 here, I would still like to close it out and make sure it is 0. I already added the below change to our koji backup-databases script and removed the old db dump from yesterday, so there should be no active sessions that are public from the previous dump now. This would also affect other postgresql servers db dumps. For those databases that don't have a 'sessions' table, it's a noop. If some databases do have such a table, I contend we don't want to back it up there either. ;) After this change is made, a run of the postgresql-server playbook is also needed to push the change out. +1s? kevin -- diff --git a/roles/postgresql_server/files/backup-database b/roles/postgresql_server/files/backup-database index 3f6e7d8..cd2bede 100644 --- a/roles/postgresql_server/files/backup-database +++ b/roles/postgresql_server/files/backup-database @@ -4,7 +4,7 @@ DB=$1 # Make our latest backup -/usr/bin/pg_dump -C $DB | /usr/bin/xz > /backups/$DB-$(date +%F).dump.xz +/usr/bin/pg_dump --exclude-table-data=sessions -C $DB | /usr/bin/xz > /backups/$DB-$(date +%F).dump.xz # Also, delete the backup from a few days ago. rm -f /backups/$DB-$(date --date="3 days ago" +%F).dump.xz

3 3

Freeze break: fix ssh_known_hosts
by Kevin Fenzi 02 Apr '15

02 Apr '15

When we moved our proxies from puppet to ansible, we forgot to setup something to copy the ssh_known_hosts file over to them, so anyone who goes to https://admin.fedoraproject.org/ssh_known_hosts gets a no such file. I'd like to add the following patch to the proxies playbook and run that playbook to correct this. +1s? kevin -- diff --git a/roles/httpd/fingerprints/tasks/main.yml b/roles/httpd/fingerprints/tasks/main.yml index 74dd152..00afe5c 100644 --- a/roles/httpd/fingerprints/tasks/main.yml +++ b/roles/httpd/fingerprints/tasks/main.yml @@ -13,3 +13,7 @@ - fingerprints - httpd - httpd/fingerprints + +- copy: src=/etc/ssh/ssh_known_hosts dest=/etc/ssh/ssh_known_hosts + tags: + - fingerprints

3 4

Freeze break request: Mediawiki update
by Patrick Uiterwijk 02 Apr '15

02 Apr '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi, A new mediawiki security update was released, and I'd like +1s for applying it. It's working fine in stg, and doesn't need database updates, just package update. Patrick -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCgAGBQJVHV3FAAoJEIZXmA2atR5QNi8QALQndGYL5nqVdP3JYErDuOWp qhdGzMOhUQ99fgACs5+WHt0Fk4UHdRUAhbzJbnHjKTKitf2eeS6CmZsnFV9azjUH ne1Ttz+1gIeSPCLv6MLE+L624cenTGqQL0g+NV3nFXv6TFPi0Y4BTS8IyVdjj8DP iOn1YliBy1JR97RgIufPAWdWPm48NC+mL2tsvKoYeDMrMBX7SMOxzgbXbthVQiyP SuoKLnm8N3u4h4AGdNH5CBPQTIn688BFfF/t1Ey+ARFQqH5mm7BEZONE0lOeT0eA AtNX3VFvO7JbLShcrfAhiSSTwB2/XGkhBdA4B94bMI/8WrHy33bZtJhLxEHyD3aw zfJTOvx5X+UrhTlWTZPVYLAKIlLTIII31Krq3IN6kzXIuu509u84QAIVYVFkwzGc y9ysZLGGnFL9avJ/sEwg6pMQ8YipXYxh7aLdnR+N+ewP+4yfDTPrSHJXkbg5eBQJ nCO81q3ur2R83JwfJM5LMcUyYhl98dJTsqVSlj+K3NrCoFH/VZMRwTKzT92CxW/r 13z0p3SJBT/R103uHP9DFZPL9y+K9dKaLWbdfMRCBGX6yN75rZYl0WqtV0DrDKYc i0L9w50UwDn2imilT3LKJRVd9JDOpOJ93OB9EBeEouTQpWMQyalOHiveNfEmV4nw S9oY7IV3uX9AgTrWlX3r =LCZ3 -----END PGP SIGNATURE-----

3 2

Plan for tomorrow's Fedora Infrastructure meeting (2015-04-02)
by Kevin Fenzi 01 Apr '15

01 Apr '15

The infrastructure team will be having it's weekly meeting tomorrow, 2015-04-02 at 18:00 UTC in #fedora-meeting on the freenode network. This week we are continuing to try something new. We have a gobby document (see: https://fedoraproject.org/wiki/Gobby ) fedora-infrastructure-meeting-next is the document. Please try and review and edit that document before the meeting and we will use it to have our agenda of things to discuss. A copy as of this morning is included in this email. kevin -- = Introduction = This shared document is for the next fedora infrastructure meeting. We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. = Meeting start stuff = #startmeeting Infrastructure (2015-04-02) #meetingname infrastructure #topic aloha #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk #topic New folks introductions / Apprentice feedback = Status / information / Trivia / Announcements = (We put things here we want others on the team to know, but don't need to discuss) (Please use #info <the thing> - your name) #topic announcements and information #info askbot upgraded to 0.7.51 - patrick #info All machines updated before the freeze - kevin / smooge #info New release of the releng dash out with previously missing components and smarter pkgdb integration - ralph #link https://apps.fedoraproject.org/releng-dash #info New release of the-new-hotness out with more feedback on bugzilla tickets (patches, srpm failures) and two-way syncing between anitya and pkgdb - ralph #info new pkgdb2 release (mostly bug fixes) - pingou #info new fedocal release (with some very nice new features by rtnpro) - pingou #info new anitya release (with bug fixes and new features) - pingou #info Lots of misc freeze breaks fixing minor stuff for the most part - kevin = Things we should discuss = (Use #topic your discussion topic - your username) #topic = Learn about some application or setup in infrastructure = (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the info in this section) #topic Learn about koji - kevin = Meeting end stuff = #topic Open Floor #endmeeting

1 0

Freeze break: run some playbooks
by Kevin Fenzi 01 Apr '15

01 Apr '15

I'd like to run a few playbooks to roll up changes made just before freeze. These should only change the indicated stuff thats already in git below. If it causes any issues, we should find out now, instead of later in the freeze, and also will clean up the check/diff report and make sure we are running exactly whats in git. 1. proxies I made a fix to fix mirrorlist.stg before freeze. This change also needs to be made in production configs, but actually doesn't do anything there: 07.10.44 proxy01.phx2.fedoraproject.org Apr 01 2015 07:18:42 23 CHECK_DIFF:CHANGED Copy over primary template for {{name}} task_userid:root 07.10.44 proxy02.fedoraproject.org Apr 01 2015 07:18:45 23 CHECK_DIFF:CHANGED Copy over primary template for {{name}} task_userid:root 07.10.44 proxy03.fedoraproject.org Apr 01 2015 07:18:43 23 CHECK_DIFF:CHANGED Copy over primary template for {{name}} task_userid:root 07.10.44 proxy04.fedoraproject.org Apr 01 2015 07:18:43 23 CHECK_DIFF:CHANGED Copy over primary template for {{name}} task_userid:root 07.10.44 proxy05.fedoraproject.org Apr 01 2015 07:18:45 23 CHECK_DIFF:CHANGED Copy over primary template for {{name}} task_userid:root 07.10.44 proxy06.fedoraproject.org Apr 01 2015 07:18:43 23 CHECK_DIFF:CHANGED Copy over primary template for {{name}} task_userid:root 07.10.44 proxy07.fedoraproject.org Apr 01 2015 07:18:46 23 CHECK_DIFF:CHANGED Copy over primary template for {{name}} task_userid:root 07.10.44 proxy08.fedoraproject.org Apr 01 2015 07:18:42 23 CHECK_DIFF:CHANGED Copy over primary template for {{name}} task_userid:root 07.10.44 proxy09.fedoraproject.org Apr 01 2015 07:18:43 23 CHECK_DIFF:CHANGED Copy over primary template for {{name}} task_userid:root 07.10.44 proxy10.phx2.fedoraproject.org Apr 01 2015 07:18:43 23 CHECK_DIFF:CHANGED Copy over primary template for {{name}} task_userid:root 07.10.44 proxy11.fedoraproject.org Apr 01 2015 07:18:43 23 CHECK_DIFF:CHANGED Copy over primary template for {{name}} task_userid:root 2. master playbook with -t fedmsgmonitor This should fix some permissions on fedmsg sockets so nrpe can montor: value 05.21.08 value01.phx2.fedoraproject.org Apr 01 2015 05:23:00 97 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root 05.21.08 value01.stg.phx2.fedoraproject.org Apr 01 2015 05:23:00 97 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root summershum 05.37.56 summershum01.phx2.fedoraproject.org Apr 01 2015 05:39:41 6 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root 05.37.56 summershum01.stg.phx2.fedoraproject.org Apr 01 2015 05:39:40 6 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root notifs-backend 05.39.48 notifs-backend01.phx2.fedoraproject.org Apr 01 2015 05:41:26 6 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root 05.39.48 notifs-backend01.stg.phx2.fedoraproject.org Apr 01 2015 05:41:26 6 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root hotness 06.04.30 hotness01.phx2.fedoraproject.org Apr 01 2015 06:06:28 6 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root 06.04.30 hotness01.stg.phx2.fedoraproject.org Apr 01 2015 06:06:28 6 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root fedimg 06.11.07 fedimg01.phx2.fedoraproject.org Apr 01 2015 06:13:02 6 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root 06.11.07 fedimg01.stg.phx2.fedoraproject.org Apr 01 2015 06:13:02 6 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root busgateway 06.26.45 busgateway01.phx2.fedoraproject.org Apr 01 2015 06:28:51 6 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root 06.26.45 busgateway01.phx2.fedoraproject.org Apr 01 2015 06:28:56 11 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root 06.26.45 busgateway01.phx2.fedoraproject.org Apr 01 2015 06:28:58 14 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root 06.26.45 busgateway01.stg.phx2.fedoraproject.org Apr 01 2015 06:28:51 6 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root 06.26.45 busgateway01.stg.phx2.fedoraproject.org Apr 01 2015 06:28:56 11 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root 06.26.45 busgateway01.stg.phx2.fedoraproject.org Apr 01 2015 06:28:58 14 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root badges-backend 05.13.22 badges-backend01.phx2.fedoraproject.org Apr 01 2015 05:14:48 6 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root 05.13.22 badges-backend01.stg.phx2.fedoraproject.org Apr 01 2015 05:14:48 6 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root 3. pkgs playbook. I added a file to exceptions for clamav scanning before freeze, but didn't run the playbook. It just adds it to a file. pkgs 07.42.48 pkgs01.stg.phx2.fedoraproject.org Apr 01 2015 07:44:59 130 CHECK_DIFF:CHANGED setup the periodic clam scan script task_userid:root 07.42.48 pkgs01.stg.phx2.fedoraproject.org Apr 01 2015 07:45:11 159 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root 07.42.48 pkgs01.stg.phx2.fedoraproject.org Apr 01 2015 07:45:25 179 CHECK_DIFF:CHANGED setup the periodic clam scan script task_userid:root 07.42.48 pkgs01.stg.phx2.fedoraproject.org Apr 01 2015 07:45:34 196 CHECK_DIFF:CHANGED create the gen-acls user task_userid:root 07.42.48 pkgs01.stg.phx2.fedoraproject.org Apr 01 2015 07:46:32 24 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root 07.42.48 pkgs02.phx2.fedoraproject.org Apr 01 2015 07:44:59 130 CHECK_DIFF:CHANGED setup the periodic clam scan script task_userid:root 07.42.48 pkgs02.phx2.fedoraproject.org Apr 01 2015 07:45:11 159 CHECK_DIFF:CHANGED ensure that nrpe has rights to monitor us task_userid:root 07.42.48 pkgs02.phx2.fedoraproject.org Apr 01 2015 07:45:25 179 CHECK_DIFF:CHANGED setup the periodic clam scan script task_userid:root +1s? kevin

3 3

Freeze Break Request: add db for execdb-stg to db-qa01.qa
by Tim Flink 01 Apr '15

01 Apr '15

We had intended to do this before freeze but managed to miss it. One of the upcoming features for taskotron that we've deployed to dev and want to deploy to staging is execdb - task execution tracking for Taskotron. This requires adding a database to db-qa01.qa which is the db host for the production Taskotron instance but no other changes to production or frozen systems. The only change in the frozen bits of ansible is: +++ b/inventory/host_vars/db-qa01.qa.fedoraproject.org @@ -22,6 +22,7 @@ dbs_to_backup: - fakefedorainfra - fakefedorainfra_stg - dev_fakefedorainfra +- execdb_stg - execdb_dev - resultsdb - resultsdb_stg

3 5

Freeze break: fix log sync script on log01
by Kevin Fenzi 01 Apr '15

01 Apr '15

log01 pulls httpd logs from proxys. When it was added, proxy11 wasn't right, this corrects it and saves us a cron email every night with the error: diff --git a/roles/base/files/syncHttpLogs.sh b/roles/base/files/syncHttpLogs.sh index 11a6d3a..ad25e01 100644 --- a/roles/base/files/syncHttpLogs.sh +++ b/roles/base/files/syncHttpLogs.sh @@ -38,7 +38,7 @@ syncHttpLogs proxy07.vpn.fedoraproject.org syncHttpLogs proxy08.vpn.fedoraproject.org syncHttpLogs proxy09.vpn.fedoraproject.org syncHttpLogs proxy10.phx2.fedoraproject.org -syncHttpLogs proxy11.phx2.fedoraproject.org +syncHttpLogs proxy11.vpn.fedoraproject.org syncHttpLogs proxy01.stg.phx2.fedoraproject.org syncHttpLogs app01.stg.phx2.fedoraproject.org syncHttpLogs bapp02.phx2.fedoraproject.org +1s? kevin

3 3

FYI retroactive freeze break: koji ssl connection timeout increase hotfix
by Kevin Fenzi 01 Apr '15

01 Apr '15

A few times in the last week we have hit a state where kojira on koji02 times out and doesn't run newrepo tasks for buildroots. Restarting the httpd on koji01 seems to unstick it, but this is not a good even stop gap as we would then have to manually do that and people would get small windows when koji was down. So, some investigation from koji developers at least for now the solution is to increase the ssl timeout. It's currently 60s, but I have increased it to 180 in a hotfix. http://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=ebb160d… and http://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=f99e19b… This is already applied on koji02 to get us out of an outage situation (no new buildroots means no new fedora), but with +1s, I will apply to koji01 as well and make sure the playbooks sync with the hosts kevin

3 3

Freeze break: sudoers on koji01/02
by Kevin Fenzi 01 Apr '15

01 Apr '15

I'd like to allow sysadmin-releng (which already has shell access) sudo also on the koji hubs. This will allow us to have mikem able to debug our kojiria issues. +1s? kevin

4 3

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure April 2015