infrastructure March 2019

infrastructure@lists.fedoraproject.org

40 participants
97 discussions

[FBR] Make RabbitMQ cluster publicly available via proxies
by Patrick Uiterwijk 07 Mar '19

07 Mar '19

Hi all, Could I get +1s for the below patch? Thanks, Patrick From 46dc8281edb9b874075525b1756ed0cfa0f91575 Mon Sep 17 00:00:00 2001 From: Patrick Uiterwijk <patrick(a)puiterwijk.org> Date: Wed, 6 Mar 2019 21:11:12 +0100 Subject: [PATCH] Add rabbitmq proxying to the proxies Signed-off-by: Patrick Uiterwijk <patrick(a)puiterwijk.org> --- inventory/group_vars/proxies | 5 +++++ inventory/group_vars/proxies-stg | 5 +++++ roles/haproxy/templates/haproxy.cfg | 28 ++++++++++++++++++++++++++++ 3 files changed, 38 insertions(+) diff --git a/inventory/group_vars/proxies b/inventory/group_vars/proxies index 4b97c2ae3..8901de118 100644 --- a/inventory/group_vars/proxies +++ b/inventory/group_vars/proxies @@ -17,6 +17,11 @@ tcp_ports: [ # This is for TCP krb5 1088, + # This is for RabbitMQ public access + 5671, + # This is for RabbitMQ internal-public access + 15671, + # This is for the haproxy HTML stats page # TODO -- there's no need for this to be wide open to the world. With this # in place, you can visit https://apps.fedoraproject.org:8080 and get the diff --git a/inventory/group_vars/proxies-stg b/inventory/group_vars/proxies-stg index 7aeb745e7..f5590beff 100644 --- a/inventory/group_vars/proxies-stg +++ b/inventory/group_vars/proxies-stg @@ -17,6 +17,11 @@ tcp_ports: [ # This is for TCP krb5 1088, + # This is for RabbitMQ public access + 5671, + # This is for RabbitMQ internal-public access + 15671, + # This is for the haproxy HTML stats page # TODO -- there's no need for this to be wide open to the world. With this # in place, you can visit https://apps.fedoraproject.org:8080 and get the diff --git a/roles/haproxy/templates/haproxy.cfg b/roles/haproxy/templates/haproxy.cfg index 29bb567b7..0b4835fcd 100644 --- a/roles/haproxy/templates/haproxy.cfg +++ b/roles/haproxy/templates/haproxy.cfg @@ -565,6 +565,34 @@ backend copr-backend option httpchk GET /api_3/ {% endif %} +{% if datacenter == "phx2" %} +# These ports are for proxying rabbitmq (AMQP) protocol through. +# At this moment, internal- and public-rabbitmq both point to the exact same set of +# brokers on the backend, but the internal- is intended for applications we directly control. +# This allows us to move to a separate cluster for public access if that became necessary +# on just the infra side, with no need to ask users to change anything. +frontend internal-rabbitmq + mode tcp + bind 0.0.0.0:15671 + default_backend rabbitmq + +frontend public-rabbitmq + mode tcp + bind 0.0.0.0:5671 + default_backend rabbitmq + +backend rabbitmq + mode tcp + option tcplog + balance roundrobin + maxconn 16384 + server rabbitmq01 rabbitmq01:5671 weight 1 maxconn 16384 +{% if env == "production %} + server rabbitmq02 rabbitmq02:5671 weight 1 maxconn 16384 + server rabbitmq03 rabbitmq03:5671 weight 1 maxconn 16384 +{% endif %} +{% endif %} + # Apache doesn't handle the initial connection here like the other proxy # entries. This proxy also doesn't use the http mode like the others. # stunnel should be sitting on port 9939 (public) and redirecting -- 2.20.1

4 3

[Freeze Break Request ] infrastructure.fedoraproject.org: Allow new cloud network access to repos and resources
by kevin＠scrye.com 07 Mar '19

07 Mar '19

From: Kevin Fenzi <kevin(a)scrye.com> Signed-off-by: Kevin Fenzi <kevin(a)scrye.com> --- roles/batcave/files/allows | 1 + 1 file changed, 1 insertion(+) diff --git a/roles/batcave/files/allows b/roles/batcave/files/allows index 33592e1..5439b2a 100644 --- a/roles/batcave/files/allows +++ b/roles/batcave/files/allows @@ -34,6 +34,7 @@ require ip 213.175.193.204 require ip 213.175.193.205 require ip 213.175.193.206 require ip 213.175.193.207 +require ip 38.145.48.0/23 # ibiblio require ip 152.19.134.136 -- 1.8.3.1

3 2

[FBR] To update python-productmd on rawhide-composer.phx2.fp.o
by Mohan Boddu 07 Mar '19

07 Mar '19

We need python-productmd-1.20-1.fc29 to fix an issue with image types in sync with formats. This will enable us to generate vmdk format images for cloud. Patch: https://github.com/release-engineering/productmd/pull/108/files Also, I will be using f29-infra repos as this build is still in testing. More info: https://pagure.io/pungi-fedora/pull-request/700 Thanks.

2 1

Change freeze for commops-watch IRC fedmsg filter
by richa gautam 06 Mar '19

06 Mar '19

Hi, This patch is made because the current commops-watch IRC fedmsg filter posts almost everything for the repository and makes it a bit spammy. So modifications are done in the filter so as to listen to new issues and PRs only. The concerned ticket can be found here <https://pagure.io/fedora-commops/fedora-happiness-packets/issue/136>. Regards, Richa Gautam

3 2

Tentative Agenda for 2019-03-07 Infrastructure meeting
by Stephen John Smoogen 06 Mar '19

06 Mar '19

The infrastructure team will be having its weekly meeting tomorrow, 2019-03-07 at 15:00 UTC in #fedora-meeting-1 on the freenode network. We have a gobby document at https://infinote.fedoraproject.org/cgit/infinote/tree/fedora-infrastructure… which can be edited for the agenda (see: https://fedoraproject.org/wiki/Gobby ) Please try and review and edit that document before the meeting and we will use it to have our agenda of things to discuss. A copy as of today is included in this email. If you have something to discuss, add the topic to the discussion area with your name. If you would like to teach other folks about some application or setup in our infrastructure, please add that topic and your name to the learn about section. = Introduction = We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. = Meeting start stuff = #startmeeting Infrastructure (2019-03-07) #meetingname infrastructure #topic aloha #chair nirik pingou puiterwijk relrod smooge tflink threebean cverna mizdebsk = Let new people say hello = #topic New folks introductions #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted = Status / Information / Trivia / Announcements = (We put things here we want others on the team to know, but don't need to discuss) (Please use #info <the thing> - your name) #topic announcements and information #info Beta Freeze Begins 2019-03-05 #info Staging Koji sync planned for 2019-03-08 (ticket 7600) #info = Things we should discuss = We use this section to bring up discussion topics. Things we want to talk about as a group and come up with some consensus /suor decision or just brainstorm a problem or issue. If there are none of these we skip this section. (Use #topic your discussion topic - your username) #topic Oncall #info smooge is on call from 2019-02-28 -> 2019-03-07 #info mizdebsk is on call from 2019-03-07 -> 2019-03-14 #info bowlofeggs is on call from 2019-03-14 -> 2019-03-21 #info ?????? is on call from 2019-03-21 -> 2019-03-28 #info Summary of last week: (from smooge ) #topic Monitoring discussion #info https://nagios.fedoraproject.org/nagios #info Go over existing out items and fix #topic Tickets discussion #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket Go thru each ticket one by one #topic Priorities for next week? #info please put tickets needing to be focused on here #topic Discuss: Is the Fedora pastebin still useful? - relrod #info how many users are using it? [3000 posts a day from 350 ips] #info should we look at converging with CentOS one so simpler setup? = Apprentice office hours = #topic Apprentice Open office minutes #info A time where apprentices may ask for help or look at problems. Here we will discuss any apprentice questions, try and match up people looking for things to do with things to do, progress, testing anything like that. = Learn about some application or setup in infrastructure = (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the i/nfo in this section. In the event we don't find someone to teach about something, we skip this section and just move on to open floor.) #info = Meeting end stuff = #topic Open Floor #endmeeting -- Stephen J Smoogen.

1 0

[Freeze Break Request ] gnome-backups: remove cloud.gnome.org and also unset freeze flag
by kevin＠scrye.com 06 Mar '19

06 Mar '19

From: Kevin Fenzi <kevin(a)scrye.com> Per averi, cloud.gnome.org is to be removed. This host has no impact on fedora releases and shouldn't freeze. Signed-off-by: Kevin Fenzi <kevin(a)scrye.com> --- inventory/group_vars/gnome-backups | 1 + roles/gnome_backups/files/backup.sh | 1 - roles/gnome_backups/files/ssh_config | 2 +- roles/gnome_backups/tasks/main.yml | 1 - 4 files changed, 2 insertions(+), 3 deletions(-) diff --git a/inventory/group_vars/gnome-backups b/inventory/group_vars/gnome-backups index 5c4a8b5..9e55342 100644 --- a/inventory/group_vars/gnome-backups +++ b/inventory/group_vars/gnome-backups @@ -1,3 +1,4 @@ +freezes: False csi_purpose: GNOME Infrastructure Backups facility csi_relationship: | Provides rdiff-backup based backups to all the GNOME Infrastructure diff --git a/roles/gnome_backups/files/backup.sh b/roles/gnome_backups/files/backup.sh index 348dce1..4b4c619 100644 --- a/roles/gnome_backups/files/backup.sh +++ b/roles/gnome_backups/files/backup.sh @@ -9,7 +9,6 @@ MACHINES='signal.gnome.org webapps2.gnome.org palette.gnome.org webapps.gnome.org - cloud.gnome.org bastion.gnome.org spinner.gnome.org master.gnome.org diff --git a/roles/gnome_backups/files/ssh_config b/roles/gnome_backups/files/ssh_config index 630a652..e6f8ecb 100644 --- a/roles/gnome_backups/files/ssh_config +++ b/roles/gnome_backups/files/ssh_config @@ -1,4 +1,4 @@ -Host puppetmaster01.gnome.org cloud.gnome.org webapps3.gnome.org +Host puppetmaster01.gnome.org webapps3.gnome.org User root IdentityFile /usr/local/etc/gnome_backup_id.rsa ProxyCommand ssh -W %h:%p bastion.gnome.org -F /usr/local/etc/gnome_ssh_config diff --git a/roles/gnome_backups/tasks/main.yml b/roles/gnome_backups/tasks/main.yml index 368ccdc..5de98fc 100644 --- a/roles/gnome_backups/tasks/main.yml +++ b/roles/gnome_backups/tasks/main.yml @@ -34,7 +34,6 @@ - webapps.gnome.org - socket.gnome.org - bugzilla.gnome.org - - cloud.gnome.org - bastion.gnome.org - spinner.gnome.org - master.gnome.org -- 1.8.3.1

3 2

[Freeze Break Request ] batcave01: add sysadmin-gnome so they can login and commit to ansible if need be.
by kevin＠scrye.com 06 Mar '19

06 Mar '19

From: Kevin Fenzi <kevin(a)scrye.com> Gnome folks have a gnome-backups01 vm that has a netapp volume for backups. They manage it via our ansible repo and playbooks. Signed-off-by: Kevin Fenzi <kevin(a)scrye.com> --- inventory/group_vars/batcave | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/inventory/group_vars/batcave b/inventory/group_vars/batcave index c518391..13e8d21 100644 --- a/inventory/group_vars/batcave +++ b/inventory/group_vars/batcave @@ -8,7 +8,7 @@ tcp_ports: [ 80, 443, 8443, 8444 ] # Neeed for rsync from log01 for logs. custom_rules: [ '-A INPUT -p tcp -m tcp -s 10.5.126.13 --dport 873 -j ACCEPT', '-A INPUT -p tcp -m tcp -s 192.168.1.59 --dport 873 -j ACCEPT' ] -fas_client_groups: sysadmin-ask,sysadmin-atomic,sysadmin-cvs,sysadmin-main,sysadmin-web,sysadmin-noc,sysadmin-hosted,sysadmin-releng,sysadmin-qa,sysadmin-tools,sysadmin-cloud,sysadmin-bot,sysadmin-centos,sysadmin-koschei,sysadmin-datanommer,sysadmin-fedimg,fi-apprentice,sysadmin-regcfp,sysadmin-badges,sysadmin-mbs,sysadmin-veteran,sysadmin-coreos,sysadmin-upstreamfirst,sysadmin-releasemonitoring,sysadmin-fpdc,sysadmin-messaging,sysadmin-libravatar +fas_client_groups: sysadmin-ask,sysadmin-atomic,sysadmin-cvs,sysadmin-main,sysadmin-web,sysadmin-noc,sysadmin-hosted,sysadmin-releng,sysadmin-qa,sysadmin-tools,sysadmin-cloud,sysadmin-bot,sysadmin-centos,sysadmin-koschei,sysadmin-datanommer,sysadmin-fedimg,fi-apprentice,sysadmin-regcfp,sysadmin-badges,sysadmin-mbs,sysadmin-veteran,sysadmin-coreos,sysadmin-upstreamfirst,sysadmin-releasemonitoring,sysadmin-fpdc,sysadmin-messaging,sysadmin-libravatar,sysadmin-gnome ansible_base: /srv/web/infra freezes: false -- 1.8.3.1

3 2

FBR to add fedora-30 key to bodhi pungi configs
by Mohan Boddu 06 Mar '19

06 Mar '19

Please review the FBR and +1 it, thanks. diff --git a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 b/roles/bodhi2/backend/templates/pungi.rpm. index b0742ec..8d9e9a3 100644 --- a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 +++ b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 @@ -27,6 +27,8 @@ sigkeys = [ '9db62fb1', [% elif release.version_int == 29 %] '429476b4', +[% elif release.version_int == 30 %] + 'cfc659b9', [% elif release.version_int == 6 %] '0608b895', [% elif release.version_int == 7 %]

3 2

External access to the AMQP broker
by Aurelien Bompard 06 Mar '19

06 Mar '19

Hey y'all, Fedora Messaging, the replacement for fedmsg, is using AMQP and thus a message broker. The current clusters we have deployed in staging and prod are only accessible from inside our infrastructure. There are two needs for an externally accessible broker: - the CentOS folks, who are outside of our infrastructure, would like to send messages - people from the community would like to subscribe to messages and do things based on them We have several options to make that happen. 1. Use our existing cluster and expose it to the world The advantage is we don't maintain another cluster, but the downside is in the case of a DoS attack we're directly affected. With RabbitMQ 3.7 there are some limits[0] you can set on vhosts (max connections and max queues), but we're not yet on 3.7. [0] https://www.rabbitmq.com/vhosts.html#limits 2. Use a separate cluster and copy messages over We could deploy a separate cluster that would get a copy of all messages, and would be more limited in resources. It truly isolates infrastructure, so it's better protected against DoS, but it's more work for sysadmins. In both cases, there are several paths we can take as regards to authentication. A: make a single readonly account for everybody in the community to use, and a few read-write accounts (with X509 certs) for people who need to publish, ie CentOS CI. If we choose a separate broker we can copy those messages back to the main cluster. The issue here is that everybody in the community will be using the same account, so it's harder to shut down bad actors. It would also be theoretically possible for someone to consume from somebody else's queue (unless people make sure they use UUIDs in queues, I think we can enforce that but it way have side effects). However, it enables the same kind of usage that fedmsg provided before. B: require authentication with username & password but make it easy to get accounts. People could require accounts via tickets for example. It will make it much harder to abuse the service, and we could easily shut down bad actors. However it's an obviously heavier load on the people who will handle the tickets and create the accounts. My personal preference would be option 2A, so an external broker with an anonymous read-only account, but all combinations of options inflict different loads on the sysadmin (on deployment and in the longer term), so I think it's really up to them. What do you guys think? Thanks Aurélien

5 11

Bodhi stakeholder's meeting
by Randy Barlow 06 Mar '19

06 Mar '19

Greetings! I've scheduled another quarterly Bodhi stakeholder's meeting for March 22 at 15:00 UTC in #fedora-meeting-2. There will be an agenda in Gobby by March 20. We will talk about what's been going on in Bodhi, and what is planned future. https://apps.fedoraproject.org/calendar/infrastructure/2019/3/22/#m9485

1 0

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure March 2019