infrastructure March 2018

infrastructure@lists.fedoraproject.org

38 participants
74 discussions

FBR: enable gating on fedora-27's Atomic CI results
by Pierre-Yves Chibon 27 Mar '18

27 Mar '18

Good Morning Everyone, The Atomic CI pipeline running at: https://jenkins-continuous-infra.apps.ci.centos.org/job/continuous-infra-ci… is running on Fedora 27 but greenwave hasn't been instructed to gate based on its results. So I would like to make the following change to greenwave's policy and run the greenwave playbook that should rebuild the containers with the new policy. Change: --- a/roles/openshift-apps/greenwave/templates/configmap.yml +++ b/roles/openshift-apps/greenwave/templates/configmap.yml @@ -136,6 +136,7 @@ data: id: "atomic_ci_pipeline_results" product_versions: - fedora-26 + - fedora-27 decision_context: bodhi_update_push_testing blacklist: [] relevance_key: original_spec_nvr Thoughts? Thanks, Pierre

3 3

[Fedocal] Reminder meeting : Bodhi stakeholder's meeting
by bowlofeggs＠fedoraproject.org 26 Mar '18

26 Mar '18

Dear all, You are kindly invited to the meeting: Bodhi stakeholder's meeting on 2018-03-27 from 15:00:00 to 16:00:00 UTC At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Source: https://apps.fedoraproject.org/calendar/meeting/4671/

1 0

[PATCH] changes to releng-team email alias
by dennis＠ausil.us 26 Mar '18

26 Mar '18

From: Dennis Gilmore <ausil(a)fedoraproject.org> Remove myself from releng-team as I am no longer involved Add Rob Marshall to the releng-team alias Signed-off-by: Dennis Gilmore <ausil(a)fedoraproject.org> --- roles/fas_client/files/aliases.template | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/fas_client/files/aliases.template b/roles/fas_client/files/aliases.template index 0de0049..41682da 100644 --- a/roles/fas_client/files/aliases.template +++ b/roles/fas_client/files/aliases.template @@ -342,7 +342,7 @@ packaging-team: ffesti,james,pmatilai,timlau,zpavlas,jnovy,jbowes,lmacken blockerbugs: tflink+blockerbugs(a)redhat.com # fedora release engineering -releng-team: ausil,mohanboddu,parasense +releng-team: mohanboddu,parasense,kellin containerbuild: maxamillion # Fedora Community Action and Impact Coordinator -- 1.8.3.1

6 6

FBR: Bodhi 3.5.2
by Randy Barlow 26 Mar '18

26 Mar '18

Greetings! There was a bit of a severe bug introduced with Bodhi 3.5.1 that I would like to create a 3.5.2 release to address. Patrick has already hotfixed production so the issue is technically not present there, but we also technically now have a non-reproducible deployment (since there isn't an RPM with the fix yet). For cleanliness, I'd like to make and deploy a 3.5.2 with that patch, though I could see an argument to wait for the freeze to lift if that is preferrable. The patch[0] is actually quite small. [0] https://github.com/fedora-infra/bodhi/pull/2243

3 3

Meeting Agenda Item: Introduction Leonardo Rossetti
by Leonardo Rossetti 26 Mar '18

26 Mar '18

Hello there, My name is Leonardo Rossetti, nickname Leo, I am currently located in São Paulo - Brazil (GMT-3) and my irc username in freenode is "lrossetti". I have been working as a software engineer for the last 10 years with a focus on automation and backend services (with a particular heavy usage in python). My most recent work is an open source project that enables mobile ci/cd on top of openshift and jenkins[1]. I am currently employed by Red Hat but I am joining as an individual and not on behalf of my employer. I've been using fedora for a while now (former debian and slackware user) so I started looking on how to contribute it back to fedora and that's how I found about the fedora infrastructure community. I would like to work on this issue if possible: https://pagure.io/fedora-infrastructure/issue/6505 (I am already familiar with nagios). I would like to learn more about fedora's infrastructure and its system environment with a low level flavor if possible (I began to study linux kernel modules development a few months ago). I can currently work around 20 hours a week since I would be using my free time to work on any issues - Looking forward to meet a mentor :) [1] - https://github.com/aerogear/aerogear-digger

2 1

FBR: Fix for processing F28 Cloud & Atomic messages in Autocloud
by Sayan Chowdhury 26 Mar '18

26 Mar '18

Hi, This is similar to the fedimg patch sent to the list earlier. Autocloud right now is not processing the messages because of the change in the F28 messages. This patch would fix the issue. This also contains a commit to remove a earlier hotfix in autocloud. That hotfix was deployed during the last release but I forgot to remove the hotfix. Don't know if this is proper but if needed I can break this into two FBRs too. Here is the PR to the corresponding change[0] Thoughts? +1s? [0] https://github.com/kushaldas/autocloud/pull/62 From: Sayan Chowdhury <sayan.chowdhury2012(a)gmail.com> Date: Mon, 26 Mar 2018 18:42:33 +0530 Subject: [PATCH 1/3] autocloud, hotfix: Remove the autocloud hotfix for the PR#56 Signed-off-by: Sayan Chowdhury <sayan.chowdhury2012(a)gmail.com> --- files/hotfix/autocloud/consumer.py | 135 --------------------------------- roles/autocloud/backend/tasks/main.yml | 14 ---- 2 files changed, 149 deletions(-) delete mode 100644 files/hotfix/autocloud/consumer.py diff --git a/files/hotfix/autocloud/consumer.py b/files/hotfix/autocloud/consumer.py deleted file mode 100644 index c216553..0000000 --- a/files/hotfix/autocloud/consumer.py +++ /dev/null @@ -1,135 +0,0 @@ -# -*- coding: utf-8 -*- -from datetime import datetime - -import requests -import fedmsg.consumers -import fedfind.release - -from sqlalchemy import exc - -import autocloud - -from autocloud.models import init_model, ComposeDetails, ComposeJobDetails -from autocloud.producer import publish_to_fedmsg -from autocloud.utils import is_valid_image, produce_jobs - -import logging -log = logging.getLogger("fedmsg") - -DEBUG = autocloud.DEBUG - - -class AutoCloudConsumer(fedmsg.consumers.FedmsgConsumer): - """ - Fedmsg consumer for Autocloud - """ - - if DEBUG: - topic = [ - 'org.fedoraproject.dev.__main__.pungi.compose.status.change' - ] - - else: - topic = [ - 'org.fedoraproject.prod.pungi.compose.status.change' - ] - - config_key = 'autocloud.consumer.enabled' - - def __init__(self, *args, **kwargs): - self.supported_archs = [arch for arch, _ in ComposeJobDetails.ARCH_TYPES] - - log.info("Autocloud Consumer is ready for action.") - super(AutoCloudConsumer, self).__init__(*args, **kwargs) - - def consume(self, msg): - """ This is called when we receive a message matching the topic. """ - - log.info('Received %r %r' % (msg['topic'], msg['body']['msg_id'])) - - STATUS_F = ('FINISHED_INCOMPLETE', 'FINISHED',) - VARIANTS_F = ('CloudImages',) - - images = [] - compose_db_update = False - msg_body = msg['body'] - status = msg_body['msg']['status'] - compose_images_json = None - - if status in STATUS_F: - location = msg_body['msg']['location'] - json_metadata = '{}/metadata/images.json'.format(location) - resp = requests.get(json_metadata) - compose_images_json = getattr(resp, 'json', False) - - if compose_images_json is not None: - compose_images_json = compose_images_json() - compose_images = compose_images_json['payload']['images'] - compose_details = compose_images_json['payload']['compose'] - compose_images = dict((variant, compose_images[variant]) - for variant in VARIANTS_F - if variant in compose_images) - compose_id = compose_details['id'] - rel = fedfind.release.get_release(cid=compose_id) - release = rel.release - compose_details.update({'release': release}) - - compose_images_variants = [variant for variant in VARIANTS_F - if variant in compose_images] - - for variant in compose_images_variants: - compose_image = compose_images[variant] - for arch, payload in compose_image.iteritems(): - - if arch not in self.supported_archs: - continue - - for item in payload: - relative_path = item['path'] - if not is_valid_image(relative_path): - continue - absolute_path = '{}/{}'.format(location, relative_path) - item.update({ - 'compose': compose_details, - 'absolute_path': absolute_path, - }) - images.append(item) - compose_db_update = True - - if compose_db_update: - session = init_model() - compose_date = datetime.strptime(compose_details['date'], '%Y%m%d') - try: - cd = ComposeDetails( - date=compose_date, - compose_id=compose_details['id'], - respin=compose_details['respin'], - type=compose_details['type'], - status=u'q', - location=location, - ) - - session.add(cd) - session.commit() - - compose_details.update({ - 'status': 'queued', - 'compose_job_id': cd.id, - }) - publish_to_fedmsg(topic='compose.queued', - **compose_details) - except exc.IntegrityError: - session.rollback() - cd = session.query(ComposeDetails).filter_by( - compose_id=compose_details['id']).first() - log.info('Compose already exists %s: %s' % ( - compose_details['id'], - cd.id - )) - session.close() - - num_images = len(images) - for pos, image in enumerate(images): - image.update({'pos': (pos+1, num_images)}) - - produce_jobs(images) diff --git a/roles/autocloud/backend/tasks/main.yml b/roles/autocloud/backend/tasks/main.yml index bc50df1..7ff5811 100644 --- a/roles/autocloud/backend/tasks/main.yml +++ b/roles/autocloud/backend/tasks/main.yml @@ -139,17 +139,3 @@ tags: - autocloud - autocloud/backend - -# -# Install hotfix to ignore new architectures -# See PR - https://github.com/kushaldas/autocloud/pull/56/ -# -- name: hotfix - copy over consumer files - copy: src='{{ files }}/{{ item.src }}' dest={{ item.dest }} - with_items: - - { src: 'hotfix/autocloud/consumer.py', dest: '/usr/lib/python2.7/site-packages/autocloud' } - notify: - - restart fedmsg-hub - tags: - - autocloud - - hotfix -- 2.9.4 From 7593e3dddaf85011b27c989a3f14c9794f4265bc Mon Sep 17 00:00:00 2001 From: Sayan Chowdhury <sayan.chowdhury2012(a)gmail.com> Date: Mon, 26 Mar 2018 18:50:42 +0530 Subject: [PATCH 2/3] autocloud, hotfix: Add the hotfix to process F28 Atomic, Cloud messages Signed-off-by: Sayan Chowdhury <sayan.chowdhury2012(a)gmail.com> --- files/hotfix/autocloud/consumer.py | 136 +++++++++++++++++++++++++++++++++ roles/autocloud/backend/tasks/main.yml | 14 ++++ 2 files changed, 150 insertions(+) create mode 100644 files/hotfix/autocloud/consumer.py diff --git a/files/hotfix/autocloud/consumer.py b/files/hotfix/autocloud/consumer.py new file mode 100644 index 0000000..a7b2aa5 --- /dev/null +++ b/files/hotfix/autocloud/consumer.py @@ -0,0 +1,136 @@ +# -*- coding: utf-8 -*- +from datetime import datetime + +import requests +import fedmsg.consumers +import fedfind.release + +from sqlalchemy import exc + +import autocloud + +from autocloud.models import init_model, ComposeDetails, ComposeJobDetails +from autocloud.producer import publish_to_fedmsg +from autocloud.utils import is_valid_image, produce_jobs + +import logging +log = logging.getLogger("fedmsg") + +DEBUG = autocloud.DEBUG + + +class AutoCloudConsumer(fedmsg.consumers.FedmsgConsumer): + """ + Fedmsg consumer for Autocloud + """ + + if DEBUG: + topic = [ + 'org.fedoraproject.dev.__main__.pungi.compose.status.change' + ] + + else: + topic = [ + 'org.fedoraproject.prod.pungi.compose.status.change' + ] + + config_key = 'autocloud.consumer.enabled' + + def __init__(self, *args, **kwargs): + self.supported_archs = [arch for arch, _ in ComposeJobDetails.ARCH_TYPES] + + log.info("Autocloud Consumer is ready for action.") + super(AutoCloudConsumer, self).__init__(*args, **kwargs) + + def consume(self, msg): + """ This is called when we receive a message matching the topic. """ + + log.info('Received %r %r' % (msg['topic'], msg['body']['msg_id'])) + + STATUS_F = ('FINISHED_INCOMPLETE', 'FINISHED',) + VARIANTS_F = ('CloudImages',) + + images = [] + compose_db_update = False + msg_body = msg['body'] + status = msg_body['msg']['status'] + compose_images_json = None + + if status in STATUS_F: + location = msg_body['msg']['location'] + json_metadata = '{}/metadata/images.json'.format(location) + resp = requests.get(json_metadata) + compose_images_json = getattr(resp, 'json', False) + + if compose_images_json is not None: + compose_images_json = compose_images_json() + compose_images = compose_images_json['payload']['images'] + compose_details = compose_images_json['payload']['compose'] + compose_images = dict((variant, compose_images[variant]) + for variant in VARIANTS_F + if variant in compose_images) + compose_id = compose_details['id'] + rel = fedfind.release.get_release(cid=compose_id) + release = rel.release + compose_details.update({'release': release}) + + compose_images_variants = [variant for variant in VARIANTS_F + if variant in compose_images] + + for variant in compose_images_variants: + compose_image = compose_images[variant] + for arch, payload in compose_image.iteritems(): + + if arch not in self.supported_archs: + continue + + for item in payload: + relative_path = item['path'] + if not is_valid_image(relative_path): + continue + absolute_path = '{}/{}'.format(location, relative_path) + item.update({ + 'compose': compose_details, + 'absolute_path': absolute_path, + }) + images.append(item) + compose_db_update = True + + if compose_db_update: + session = init_model() + compose_date = datetime.strptime(compose_details['date'], '%Y%m%d') + try: + cd = ComposeDetails( + date=compose_date, + compose_id=compose_details['id'], + respin=compose_details['respin'], + type=compose_details['type'], + status=u'q', + location=location, + ) + + session.add(cd) + session.commit() + + compose_details.update({ + 'status': 'queued', + 'compose_job_id': cd.id, + }) + publish_to_fedmsg(topic='compose.queued', + **compose_details) + except exc.IntegrityError: + session.rollback() + cd = session.query(ComposeDetails).filter_by( + compose_id=compose_details['id']).first() + log.info('Compose already exists %s: %s' % ( + compose_details['id'], + cd.id + )) + session.close() + + num_images = len(images) + for pos, image in enumerate(images): + image.update({'pos': (pos+1, num_images)}) + + produce_jobs(images) + diff --git a/roles/autocloud/backend/tasks/main.yml b/roles/autocloud/backend/tasks/main.yml index 7ff5811..e1024c3 100644 --- a/roles/autocloud/backend/tasks/main.yml +++ b/roles/autocloud/backend/tasks/main.yml @@ -139,3 +139,17 @@ tags: - autocloud - autocloud/backend + +# +# Install hotfix to ignore new architectures +# See PR - https://github.com/kushaldas/autocloud/pull/62/ +# +- name: hotfix - copy over consumer files + copy: src='{{ files }}/{{ item.src }}' dest={{ item.dest }} + with_items: + - { src: 'hotfix/autocloud/consumer.py', dest: '/usr/lib/python2.7/site-packages/autocloud' } + notify: + - restart fedmsg-hub + tags: + - autocloud + - hotfix -- 2.9.4 From 1403d8118ab9483d0753f1995bec5ce6e78f0df2 Mon Sep 17 00:00:00 2001 From: Sayan Chowdhury <sayan.chowdhury2012(a)gmail.com> Date: Mon, 26 Mar 2018 18:53:01 +0530 Subject: [PATCH 3/3] autocloud, hotfix: Add the patch for the hotfix (F28 messages) Signed-off-by: Sayan Chowdhury <sayan.chowdhury2012(a)gmail.com> --- files/hotfix/autocloud/consumer.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/files/hotfix/autocloud/consumer.py b/files/hotfix/autocloud/consumer.py index a7b2aa5..0ff8317 100644 --- a/files/hotfix/autocloud/consumer.py +++ b/files/hotfix/autocloud/consumer.py @@ -48,7 +48,6 @@ class AutoCloudConsumer(fedmsg.consumers.FedmsgConsumer): log.info('Received %r %r' % (msg['topic'], msg['body']['msg_id'])) STATUS_F = ('FINISHED_INCOMPLETE', 'FINISHED',) - VARIANTS_F = ('CloudImages',) images = [] compose_db_update = False @@ -56,6 +55,16 @@ class AutoCloudConsumer(fedmsg.consumers.FedmsgConsumer): status = msg_body['msg']['status'] compose_images_json = None + # Till F27, both cloud-base and atomic images were available + # under variant CloudImages. With F28 and onward releases, + # cloud-base image compose moved to cloud variant and atomic images + # moved under atomic variant. + prev_rel = ['26', '27'] + if msg_body['msg']['release_version'] in prev_rel: + VARIANTS_F = ('CloudImages',) + else: + VARIANTS_F = ('AtomicHost', 'Cloud') + if status in STATUS_F: location = msg_body['msg']['location'] json_metadata = '{}/metadata/images.json'.format(location) @@ -133,4 +142,3 @@ class AutoCloudConsumer(fedmsg.consumers.FedmsgConsumer): image.update({'pos': (pos+1, num_images)}) produce_jobs(images) - -- 2.9.4 -- Sayan Chowdhury <https://sayanchowdhury.dgplug.org/> Senior Software Engineer, Fedora Engineering - Emerging Platform GPG Fingerprint : 0F16 E841 E517 225C 7D13 AB3C B023 9931 9CD0 5C8B Proud to work at The Open Organization!

3 2

FBR: Update fedmsg on compose-x86-01
by Patrick マルタインアンドレアス Uiterwijk 26 Mar '18

26 Mar '18

Hi, Last Friday, pungi got updated on compose-x86-01, but that requires fedmsg-1.1.0 (there's currently 1.0.1 on that box) because of json encoding issues. +1 to run "dnf update --refresh fedmsg" on compose-x86-01? Patrick

4 3

Retroactive FBR: Run proxy playbook with -t whatcanidoforfedora.org
by Ricky Elrod 23 Mar '18

23 Mar '18

Our cert for whatcanidoforfedora.org expired. I've renewed the cert and deployed it. I had to run the proxy playbook with -t whatcanidoforfedora.org, which caused an httpd reload on proxies to see the new cert. Retroactive +1's please.

3 2

FBR: Enable --sni for nagios ssl cert checks, and add a few missing ones.
by Ricky Elrod 23 Mar '18

23 Mar '18

I'd like to enable --sni on the Nagios cert check to force it to check the correct certificate on some of our sites (which use SNI). I'd also like to add checks for whatcanidoforfedora.org, release-monitoring.org, and pagure.io. +1's? diff --git a/roles/nagios_server/files/nagios/commands/httpd.cfg b/roles/nagios_server/files/nagios/commands/httpd.cfg index 944cb50..21843f4 100644 --- a/roles/nagios_server/files/nagios/commands/httpd.cfg +++ b/roles/nagios_server/files/nagios/commands/httpd.cfg @@ -63,7 +63,7 @@ define command{ define command{ command_name check_ssl_cert - command_line $USER1$/check_http -I $HOSTADDRESS$ -H $ARG1$ -C $ARG2$ + command_line $USER1$/check_http --sni -I $HOSTADDRESS$ -H $ARG1$ -C $ARG2$ } define command{ diff --git a/roles/nagios_server/files/nagios/services/ssl.cfg b/roles/nagios_server/files/nagios/services/ssl.cfg index 150411d..81e4b4a 100644 --- a/roles/nagios_server/files/nagios/services/ssl.cfg +++ b/roles/nagios_server/files/nagios/services/ssl.cfg @@ -32,3 +32,24 @@ define service { check_command check_ssl_cert!pkgs.fedoraproject.org!60 use defaulttemplate } + +define service { + hostgroup_name proxies + service_description https-whatcanidoforfedora-cert + check_command check_ssl_cert!whatcanidoforfedora.org!25 + use defaulttemplate +} + +define service { + host_name anitya-frontend01.fedoraproject.org + service_description https-release-monitoring-cert + check_command check_ssl_cert!release-monitoring.org!60 + use defaulttemplate +} + +define service { + host_name pagure-proxy01.fedoraproject.org + service_description https-pagure-cert + check_command check_ssl_cert!pagure.io!60 + use defaulttemplate +}

3 2

FBR: To update pungi on compose-x86-01.phx2.fedoraproject.org to pungi-4.1.22-10.fc27
by Mohan Boddu 22 Mar '18

22 Mar '18

We updated pungi on branched and rawhide composer a lot during this freeze, here's the list of the FBR and why we updated pungi https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapr… https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapr… https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapr… https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapr… Now that we are getting close to RC request and all the known issues are fixed in pungi, I would like to update pungi on compose-x86-01.phx2.fedoraproject.org from which we run the RC compose.

3 3

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure March 2018