infrastructure August 2015

infrastructure@lists.fedoraproject.org

41 participants
45 discussions

[PATCH] kojid: arm: fix host triplet on arm builders
by Nicolas Chauvet 15 Aug '15

15 Aug '15

armhfp architecture is using GNU Extended ABI to differenciate from previous armv7 softfp/soft GNU one. Currently, the fedora infrastructure doesn't maintain this old arm 32bit abi so we can assume every koji arm* builders to use this "new" ABI. Also, aarch64 builders are using another aarch64 prefix for the name which this patch handles right since aarch64 is using redhat-linux-gnu. This patch should have limited or no impact to most gcc based packages because gcc knowns the %{_host} triplet for itself. But it will fix llvm/clang build usability which rely on the appropriate host triplet to interface with gcc. This was tested with llvm 3.5.2/3.6.2 to build mesa with clang. reported as https://fedorahosted.org/fedora-infrastructure/ticket/4858 Signed-off-by: Nicolas Chauvet <kwizart(a)gmail.com> --- roles/koji_builder/templates/kojid.conf | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/roles/koji_builder/templates/kojid.conf b/roles/koji_builder/templates/kojid.conf index 9bb9a29..7c1b55b 100644 --- a/roles/koji_builder/templates/kojid.conf +++ b/roles/koji_builder/templates/kojid.conf @@ -33,7 +33,11 @@ packager=Fedora Project distribution=Fedora Project ; The _host string to use in mock +{% if ansible_hostname.startswith('arm') %} +mockhost=redhat-linux-gnueabi +{% else %} mockhost=redhat-linux-gnu +{% endif %} ; The URL for the xmlrpc server server={{koji_server_url}} -- 1.7.2.1

1 0

Move to mailman3
by Pierre-Yves Chibon 12 Aug '15

12 Aug '15

Good morning everyone, As you may know, mailman3 has been released and with it the new archiver bridging mailing lists and forums: HyperKitty. Some of the automated mailing list have been already migrated to mailman3, but we still need to migrate 'live' mailing list (ie: having user actually discussing, not just bot sending emails). There has been thoughts to use the infrastructure list as a potential first list to migrate. Thoughts on this? Pierre To give you an idea: HyperKitty for scm-commits: https://lists.fedoraproject.org/archives/list/scm-commits@lists.fedoraproje…

15 26

END OF FREEZE: Re: Fedora 23 Alpha Freeze now in effect
by Stephen John Smoogen 12 Aug '15

12 Aug '15

The Fedora 23 Alpha freeze is over as we have shipped an alpha. Thank you to everyone who worked on things. On 28 July 2015 at 14:02, Kevin Fenzi <kevin(a)scrye.com> wrote: > Greetings. > > we are now in the infrastructure freeze leading up to the Fedora 23 > Alpha release. This is a pre-release freeze. > > We do this to ensure that our infrastructure is stable and ready to > release the Fedora 23 Alpha when it's available. > > You can see a list of hosts that do not freeze by checking out the > ansible repo and running the freezelist script: > > git clone https://infrastructure.fedoraproject.org/infra/ansible.git > scripts/freezelist -i inventory > > Any hosts listed as freezes is frozen until 2015-08-12. (or later if > Alpha slips). Frozen hosts should have no changes made to them > without a sign-off on the change from at least 2 sysadmin-main or > rel-eng members, along with (in most cases) a patch of the exact > change to be made to this list. > > Thanks, > > kevin > > _______________________________________________ > infrastructure mailing list > infrastructure(a)lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/infrastructure -- Stephen J Smoogen.

1 0

Freeze break request: make only fas01 clear sessions, resolving a deadlock
by Patrick Uiterwijk 11 Aug '15

11 Aug '15

Change in controllers.py is at line 101: "if socket.gethostname()...." commit d0f4e6f6f956133da4116025eead691d4d96fbb7 Author: Patrick Uiterwijk <puiterwijk(a)redhat.com> Date: Mon Aug 10 22:29:44 2015 +0000 HOTFIX: Make sure that only fas01clears sessions This will prevent deadlocks in the SQL server diff --git a/roles/fas_server/files/controllers.py b/roles/fas_server/files/controllers.py new file mode 100644 index 0000000..6b15a46 --- /dev/null +++ b/roles/fas_server/files/controllers.py @@ -0,0 +1,263 @@ +# -*- coding: utf-8 -*- +# +# Copyright © 2008 Ricky Zhou +# Copyright © 2008-2014 Red Hat, Inc. +# +# This copyrighted material is made available to anyone wishing to use, modify, +# copy, or redistribute it subject to the terms and conditions of the GNU +# General Public License v.2. This program is distributed in the hope that it +# will be useful, but WITHOUT ANY WARRANTY expressed or implied, including the +# implied warranties of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. +# See the GNU General Public License for more details. You should have +# received a copy of the GNU General Public License along with this program; +# if not, write to the Free Software Foundation, Inc., 51 Franklin Street, +# Fifth Floor, Boston, MA 02110-1301, USA. Any Red Hat trademarks that are +# incorporated in the source code or documentation are not subject to the GNU +# General Public License and may only be used or replicated with the express +# permission of Red Hat, Inc. +# +# Author(s): Ricky Zhou <ricky(a)fedoraproject.org> +# Mike McGrath <mmcgrath(a)redhat.com> +# Toshio Kuratomi <toshio(a)redhat.com> +# +from bunch import Bunch + +from turbogears import expose, config, identity, redirect +from turbogears.database import session +from cherrypy import request + +import turbogears +import cherrypy +import time + +from fedora.tg import controllers as f_ctrlers +from fedora.tg.utils import request_format + +from fas import release +from fas.user import User +from fas.group import Group +from fas.configs import Config +from fas.fpca import FPCA +from fas.json_request import JsonRequest +from fas.help import Help +from fas.model import Session, People +from fas.model import SessionTable + + +from fas.auth import undeprecated_cla_done +from fas.util import available_languages + +from fas import plugin + +import os + +import datetime + +import socket + +try: + import cPickle as pickle +except ImportError: + import pickle + +class SQLAlchemyStorage: + def __init__(self): + pass + + def load(self, session_id): + s = Session.query.get(session_id) + if not s: + return None + expiration_time = s.expiration_time + pickled_data = s.data + data = pickle.loads(pickled_data.encode('utf-8')) + return (data, expiration_time) + + # This is an iffy one. CherryPy's built in session + # storage classes use delete(self, id=None), but it + # isn't called from anywhere in cherrypy. I think we + # can do this as long as we're careful about how we call it. + def delete(self, session_id=None): + if session_id is None: + session_id = cherrypy.session.id + s = Session.query.get(session_id) + session.delete(s) + session.flush() + + def save(self, session_id, data, expiration_time): + pickled_data = pickle.dumps(data) + s = Session.query.get(session_id) + if not s: + s = Session() + s.id = session_id + s.data = pickled_data + s.expiration_time = expiration_time + session.flush() + + def acquire_lock(self): + pass + + def release_lock(self): + pass + + def clean_up(self, sess): + # This is to make sure that only one server cleans up sessions + if socket.gethostname() != 'fas01.phx2.fedoraproject.org': + return + result = SessionTable.delete( + SessionTable.c.expiration_time.__lt__(datetime.datetime.now()) + ).execute() + +config.update({'session_filter.storage_class': SQLAlchemyStorage}) + +def get_locale(locale=None): + if locale: + return locale + try: + return turbogears.identity.current.user.locale + except AttributeError: + pass + try: + return cherrypy.request.simple_cookie['fas_locale'].value + except KeyError: + pass + + default_language = config.get('default_language', + turbogears.i18n.utils._get_locale()) + return default_language + +config.update({'i18n.get_locale': get_locale}) + + +def add_custom_stdvars(variables): + return variables.update({'gettext': _, "lang": get_locale(), + 'available_languages': available_languages(), + 'fas_version': release.VERSION, + 'webmaster_email': config.get('webmaster_email')}) +turbogears.view.variable_providers.append(add_custom_stdvars) + +# from fas import json +# import logging +# log = logging.getLogger("fas.controllers") + +#TODO: Appropriate flash icons for errors, etc. +# mmcgrath wonders if it will be handy to expose an encrypted mailer with fas +# over json for our apps + +class Root(plugin.RootController): + + user = User() + group = Group() + fpca = FPCA() + json = JsonRequest() + config = Config() + help = Help() + + def __init__(self): + # TODO: Find a better place for this. + os.environ['GNUPGHOME'] = config.get('gpghome') + plugin.RootController.__init__(self) + + def getpluginident(self): + return 'fas' + + @expose(template="fas.templates.welcome", allow_json=True) + def index(self): + if turbogears.identity.not_anonymous(): + if request_format() == 'json': + # redirects don't work with JSON calls. This is a bit of a + # hack until we can figure out something better. + return dict() + turbogears.redirect('/home') + return dict(now=time.ctime()) + + @identity.require(identity.not_anonymous()) + @expose(template="fas.templates.home", allow_json=True) + def home(self): + user_name = turbogears.identity.current.user_name + person = People.by_username(user_name) + (cla_done, undeprecated_cla) = undeprecated_cla_done(person) + + person = person.filter_private() + return dict(person=person, memberships=person['memberships'], cla=undeprecated_cla) + + @expose(template="fas.templates.about") + def about(self): + return dict() + + @expose(template="fas.templates.login", allow_json=True) + def login(self, forward_url=None, *args, **kwargs): + '''Page to become authenticated to the Account System. + + This shows a small login box to type in your username and password + from the Fedora Account System. + + :kwarg forward_url: The url to send to once authentication succeeds + ''' + actual_login_dict = f_ctrlers.login(forward_url=forward_url, *args, **kwargs) + + try: + login_dict = Bunch() + login_dict['user'] = Bunch() + for field in People.allow_fields['complete']: + login_dict['user'][field] = None + for field in People.allow_fields['self']: + login_dict['user'][field] = getattr(actual_login_dict['user'], field) + # Strip out things that the user shouldn't see about their own + # login + login_dict['user']['internal_comments'] = None + login_dict['user']['emailtoken'] = None + login_dict['user']['security_answer'] = None + login_dict['user']['alias_enabled'] = None + login_dict['user']['passwordtoken'] = None + + # Add things that are needed by some other apps + login_dict['user'].approved_memberships = list( + actual_login_dict['user'].approved_memberships) + login_dict['user'].memberships = list(actual_login_dict['user'].memberships) + login_dict['user'].unapproved_memberships = list( + actual_login_dict['user'].unapproved_memberships) + login_dict['user'].group_roles = list(actual_login_dict['user'].group_roles) + login_dict['user'].roles = list(actual_login_dict['user'].roles) + login_dict['user'].groups = [g.name for g in actual_login_dict['user'].approved_memberships] + return login_dict + except KeyError, e: + # No problem, this usually means that we failed to login and + # therefore we don't have a user field. + login_dict = actual_login_dict + + if not identity.current.anonymous and identity.was_login_attempted() \ + and not identity.get_identity_errors(): + # Success that needs to be passed back via json + return login_dict + + if identity.was_login_attempted() and request.fas_provided_username: + if request.fas_identity_failure_reason == 'status_inactive': + turbogears.flash(_('Your old password has expired. Please' + ' reset your password below.')) + if request_format() != 'json': + redirect('/user/resetpass') + if request.fas_identity_failure_reason == 'status_account_disabled': + turbogears.flash(_('Your account is currently disabled. For' + ' more information, please contact %(admin_email)s' % + {'admin_email': config.get('accounts_email')})) + if request_format() != 'json': + redirect('/login') + + return login_dict + + @expose(allow_json=True) + def logout(self): + return f_ctrlers.logout() + + @expose() + def language(self, locale): + if locale not in available_languages(): + turbogears.flash(_('The language \'%s\' is not available.') % locale) + redirect(request.headers.get("Referer", "/")) + return dict() + #turbogears.i18n.set_session_locale(locale) + cherrypy.response.simple_cookie['fas_locale'] = locale + redirect(request.headers.get("Referer", "/")) + return dict() + diff --git a/roles/fas_server/tasks/main.yml b/roles/fas_server/tasks/main.yml index 57370a8..980013d 100644 --- a/roles/fas_server/tasks/main.yml +++ b/roles/fas_server/tasks/main.yml @@ -355,3 +355,12 @@ - config - fas - hotfixfas + +- name: HOTFIX make sure only fas01 cleans up sessions + copy: src={{ roles }}/fas_server/files/controllers.py + dest=/usr/lib/python2.6/site-packages/fas/controllers.py + mode=644 owner=root group=root + tags: + - config + - fas + - hotfixfas

3 4

Retroactive freeze break request: Add indexes for ipsilon
by Patrick Uiterwijk 11 Aug '15

11 Aug '15

Hi, So it turned out that the slowness of Ipsilon was caused by the lack of keys on the database, so it got slower over time, as more entries got added. I just ran the following queries to create new indexes on production, could I get any +1s retroactive? ----------------------------------------------------- ALTER TABLE association ADD PRIMARY KEY (uuid,name); CREATE INDEX ON association (uuid); ALTER TABLE dbinfo ADD PRIMARY KEY (name, option); CREATE INDEX ON dbinfo (name); ALTER TABLE openid_data ADD PRIMARY KEY (name, option); CREATE INDEX ON openid_data (name); ALTER TABLE sessions ADD PRIMARY KEY (id); CREATE INDEX ON sessions (expiration_time); ALTER TABLE transactions ADD PRIMARY KEY (uuid, name); CREATE INDEX ON transactions (uuid); ALTER TABLE users ADD PRIMARY KEY (name, option); CREATE INDEX ON users (name); ----------------------------------------------------- With kind regards, Patrick Uiterwijk Fedora Infra

3 2

bodhi2 plans
by Kevin Fenzi 10 Aug '15

10 Aug '15

Greetings. So, in the releng meeting last week we discussed about landing bodhi2 before the alpha freeze started, however there were just too many things still up in the air to do that, so we held off. However, I'd like to propose a concrete date to enable it and try and make sure we have everything we need lined up for making that work. :) My understanding of the current status: (Luke: please correct me if anything is wrong or unclear here) * We have bodhi2 web frontend up and running fine in staging: https://admin.stg.fedoraproject.org/updates/ (well, right now it's down as Luke is working on a update id bug there and is repopulating the db, but it was and will be up again soon). * We have a bodhi2 backend up in staging and it's successfully mashed updates pushes. * bodhi2 isn't yet packaged in Fedora (it and it's deps are in a copr): http://copr.fedoraproject.org/coprs/lmacken/bodhi2/ So, to get that last bit we need: * API users. I thought we had a list somewhere, but I cannot seem to find it. At least fedora-easy-karma and fedora-gooey-karma use it. Are all of these using python-fedora for the api? Do we need to update that and bodhi-client? * We need bodhi2 in fedora/epel. Were we going to do it as a seperate package, or just an update to the existing 'bodhi' package. I assume maintainers would need the new client to do 'fedpkg update' ? * Do we need any fedpkg changes? Or anything else? How do we want to handle the switchover? * Is taskotron all ready to go with bodhi2? or more changes there? * We will need an outage to switch over. I assume the bodhi1->bodhi2 db conversion shouldn't take too long? But once we convert and make live we are commmited. * Anything else I missed? I'd like to propose we schedule the switchover/outage for 2015-08-19. (Provided alpha doesn't slip). This will allow us a few days to recover from flock and get anything lined up we need to. Thoughts? additions? kevin

3 3

Freeze Break Request - fedmsg authz policy
by Ralph Bean 07 Aug '15

07 Aug '15

I found a bug! The impact of it is that the releng primary arch 'compose' fedmsg messages were being considered invalid by our consuming services (ircbot, etc..) but the s390 compose messages were being let through. First -- here's the patch, then the explanation: diff --git a/inventory/host_vars/branched-composer.phx2.fedoraproject.org b/inventory/host_vars/branched-composer.phx2.fedoraproject.org index bfe9b94..a5d3514 100644 --- a/inventory/host_vars/branched-composer.phx2.fedoraproject.org +++ b/inventory/host_vars/branched-composer.phx2.fedoraproject.org @@ -7,12 +7,3 @@ volgroup: /dev/vg_bvirthost08 kojipkgs_url: kojipkgs.fedoraproject.org kojihub_url: koji.fedoraproject.org/kojihub kojihub_scheme: https - -# These are consumed by a task in roles/fedmsg/base/main.yml -fedmsg_certs: -- service: shell - owner: root - group: root -- service: bodhi - owner: root - group: masher diff --git a/inventory/host_vars/rawhide-composer.phx2.fedoraproject.org b/inventory/host_vars/rawhide-composer.phx2.fedoraproject.org index a0d17a6..9cb3409 100644 --- a/inventory/host_vars/rawhide-composer.phx2.fedoraproject.org +++ b/inventory/host_vars/rawhide-composer.phx2.fedoraproject.org @@ -6,12 +6,3 @@ volgroup: /dev/vg_bvirthost06 kojipkgs_url: kojipkgs.fedoraproject.org kojihub_url: koji.fedoraproject.org/kojihub kojihub_scheme: https - -# These are consumed by a task in roles/fedmsg/base/main.yml -fedmsg_certs: -- service: shell - owner: root - group: root -- service: bodhi - owner: root - group: masher It is just *removing* lines from the host_vars files for rawhide-composer and branched-composer. But why? First, those fedmsg_certs vars are already defined at the group_vars level here: https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/inventory/gr… That more fully-defined and correct statement at the group level was being overwritten by the less fully defined statements at the host level (see ansible var precedence rules). Everything appeared to be working normally while this was the case because, since no hosts declared that they could send those fedmsg topics there was no explicit check for who could send them. It didn't matter until we added the s390 koji hub a week or so ago which is allowed to broadcast those same topics. It had its fedmsg_certs correctly defined, and since it declared that it could send those topics -- and no other hosts made the same declarations -- the primary compose messages suddenly started being considered invalid (unauthorized). By removing these old crufty definitions at the host level and letting the correct definition at the group level prevail -- all those hosts should show up correctly in the fedmsg authz policy and things should start working again. This will require a master playbook run on just the fedmsgdconfig tag to push out making this a "high touch" change to do during freeze, but I'm quite certain it is correct. Can I get two +1s? -Ralph

3 2

RE: Move to mailman3
by Sander Hoentjen 07 Aug '15

07 Aug '15

Hey yeah I figured as much, haven't touched my procmailrc for as long as I can remember :) Will go fix it now. -----Original message----- From: Aurelien Bompard <abompard(a)fedoraproject.org> Sent: Friday 7th August 2015 20:18 To: Fedora Infrastructure <infrastructure(a)lists.fedoraproject.org> Subject: Re: Move to mailman3 > And for me things show up in my inbox instead of in a folder, because I > filter on X-BeenThere. This disappeared, not sure if it counts as a > regression though :) If I understand correctly, the X-BeenThere header has been obsoleted some time ago (read: years ;-)), you can now filter on List-Id or List-Post. https://bugs.launchpad.net/mailman/+bug/883158 Sorry for the trouble, that's clearly something to warn people of. Aurélien

1 0

RE: Move to mailman3
by Sander Hoentjen 07 Aug '15

07 Aug '15

And for me things show up in my inbox instead of in a folder, because I filter on X-BeenThere. This disappeared, not sure if it counts as a regression though :) Sander -----Original message----- From: Michael Cronenworth <mike(a)cchtml.com> Sent: Friday 7th August 2015 19:42 To: infrastructure(a)lists.fedoraproject.org Subject: Re: Move to mailman3 On 08/07/2015 12:37 PM, Aurelien Bompard wrote: >> Might be an idea to send one when you start the migration as well :] >> Just that we are not surprised or that we watch if one of our email doesn't land >> on the list. > > Yeah, that would have been a good idea, but it's too late: your > message was processed by Mailman3 already ;-) > https://lists.fedoraproject.org/archives/list/infrastructure%40lists.fedora… > > I'm currently checking for bugs & regressions, feel free to ping me :-) > > Aurélien > Did you use an old subscription list? I have not been subscribed to @infra for about a month and today I start receiving these from you guys. Performing the unsubscribe process again does not send me a confirmation e-mail.

3 2

Time to work on a Plan: Moving VM's
by Stephen John Smoogen 07 Aug '15

07 Aug '15

We have new hardware in to replace some of our 4+ year old IBM x3650's and need to do so in the next month or so to make sure we have a good list of hardware to go onto extended warranty this fall. I would like to come up with a plan of attack on getting them all moved by September 1st to virthost19->virthost22. Hardware Server Virtual Machine virthost05.phx2.fedoraproject.org bastion02.phx2.fedoraproject.org virthost05.phx2.fedoraproject.org db-fas01.phx2.fedoraproject.org virthost05.phx2.fedoraproject.org proxy01.phx2.fedoraproject.org virthost06.phx2.fedoraproject.org ask01.phx2.fedoraproject.org virthost06.phx2.fedoraproject.org notifs-web02.phx2.fedoraproject.org virthost07.phx2.fedoraproject.org datagrepper02.phx2.fedoraproject.org virthost07.phx2.fedoraproject.org elections01.phx2.fedoraproject.org virthost07.phx2.fedoraproject.org hotness01.phx2.fedoraproject.org virthost07.phx2.fedoraproject.org nuancier01.phx2.fedoraproject.org virthost08.phx2.fedoraproject.org darkserver01.phx2.fedoraproject.org virthost08.phx2.fedoraproject.org ns03.phx2.fedoraproject.org virthost09.phx2.fedoraproject.org busgateway01.phx2.fedoraproject.org virthost09.phx2.fedoraproject.org fedocal01.phx2.fedoraproject.org virthost09.phx2.fedoraproject.org notifs-backend01.phx2.fedoraproject.org virthost09.phx2.fedoraproject.org nuancier02.phx2.fedoraproject.org virthost10.phx2.fedoraproject.org bodhi01.stg.phx2.fedoraproject.org virthost10.phx2.fedoraproject.org bodhi02.stg.phx2.fedoraproject.org virthost10.phx2.fedoraproject.org fas01.stg.phx2.fedoraproject.org virthost10.phx2.fedoraproject.org koji01.stg.phx2.fedoraproject.org virthost10.phx2.fedoraproject.org notifs-web02.stg.phx2.fedoraproject.org virthost10.phx2.fedoraproject.org summershum01.stg.phx2.fedoraproject.org There are a couple of ways we do these transitions. 1) Spin up a new virtual machine with an incremented hostname: Example: a) check to see which ask systems exist. (ask01, ask02) b) create a new virtual machine with an incremented number: ask03 c) ansible the system to be clone of ask01 d) either turn off ask01 and rename ask03 to be ask01 OR d) configure other servers to point to ask03 instead of ask01 e) fix problems as needed f) shutdown and remove ask01. 2) Move virtual machine to another server. a) Schedule a downtime b) Shutdown the server c) network dd the lvm image to other server. d) copy over the /etc/libvirt/qemu/___.xml file over to other server. e) spin up server f) fix problems as needed g) remove files from old server 3) If the image is on an iscsi share versus local disks... a) shutdown the image on server A. b) copy the xml files over to server B. c) get libvirt to see them. d) start the image on server B e) remove the xml files from server A. It looks like none of the servers in question are on the iscsi share so we won't be able to do 3. [Unless there is one or two that are good candidates to be on the iscsi share... then a variant of 2 would be used.] Downtimes except for the fas system will be in the 20 minute range. The fas database might be 1-3 hours due to a 100 GB image to be copied over and the usual 'what we have to reboot that because this was down? WHY?' problems we end up with. Other plans and ideas can be replied to here. -- Stephen J Smoogen.

5 4

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure August 2015