infrastructure March 2015

infrastructure@lists.fedoraproject.org

42 participants
65 discussions

Freeze break: Install new zanata-python-client on bapp02
by Robert Mayr 06 Mar '15

06 Mar '15

Hi, websites translations have been migrated to zanata, but the actual zanata-python-client release is buggy and doesnt work at all to pull translations (1.3.13-1) I've worked a bit on it and Anish recently released an updated el6 package of the client which is working fine on staging (app01). The latest client is still in updates-testing, but will be pushed to stable at next push (pending). http://koji.fedoraproject.org/koji/buildinfo?buildID=617577 The zanata-python-client-1.3.21-1.el6 has been already added to the infra repo, so I'm requesting this freeze break to install it on bapp02. The package has no new deps and it's just an update of the client. +1's from your side to install it and be ready for tuesday's Alpha release? Thank you. -- Robert Mayr (robyduck)

4 4

Freeze Break: fix pkgdb_sync_git_branches.py for git being too clever
by Kevin Fenzi 06 Mar '15

06 Mar '15

See ticket: https://fedorahosted.org/fedora-infrastructure/ticket/4680 Basically git decides sometimes if it thinks it's a good idea to pack the refs for a repo. This includes the head refs. Then they no longer appear under refs/heads/ but instead are in a packed-refs file listing them all. Currently pkgdb_sync_git_branches.py just looks for refs/heads/* in 2 places to decide what it should do. We replace one with a 'git show-ref --heads' and the other with a 'git branch' to get existing branches. This is needed to fix sending an error to libguestfs maintainers everytime genacls runs in production. It's been tested on pkgs01.stg and works there (with the libguestfs synced from prod). +1s? kevin -- diff --git a/roles/distgit/templates/pkgdb_sync_git_branches.py b/roles/distgit/templates/pkgdb_sync_git_branches.py index 8643165..fc4e6b3 100644 --- a/roles/distgit/templates/pkgdb_sync_git_branches.py +++ b/roles/distgit/templates/pkgdb_sync_git_branches.py @@ -100,6 +100,15 @@ def _invoke(program, args): e.message = 'Error, "%s" returned %s' % (e.cmd, e.returnCode) raise e +def _call_cmd(command): + ''' Call the given command with the given arguments, all provided as a list + and return the output from this command. + ''' + if isinstance(command, basestring): + command = command.split() + output = subprocess.Popen(command, stdout=subprocess.PIPE).stdout.read()[:-1] + return output + def _create_branch(pkgname, branch): '''Create a specific branch for a package. @@ -108,23 +117,23 @@ def _create_branch(pkgname, branch): :arg branch: Name of the branch to create ''' + branch = branch.replace('*', '').strip() if branch == 'master': print 'ERROR: Proudly refusing to create master branch. Invalid repo?' print 'INFO: Please check %s repo' % pkgname return - branchpath = os.path.join( - GIT_FOLDER, '%s.git' % pkgname, 'refs/heads', branch) - if not os.path.exists(branchpath): - try: + os.chdir(os.path.join(GIT_FOLDER, '%s.git' % pkgname)) + branches = get_git_branch(pkgname) + if branch not in branches: + try: _invoke(MKBRANCH, [branch, pkgname]) - except ProcessError, e: + except ProcessError, e: if e.returnCode == 255: # This is a warning, not an error return raise - finally: - fedmsg.publish( + fedmsg.publish( topic='branch', modname='git', msg=dict( @@ -133,9 +142,8 @@ def _create_branch(pkgname, branch): branch=branch, ), ) - elif VERBOSE: - print 'Was asked to create branch %s of package %s, but it '\ - 'already exists' % (pkgname, branch) + else: + print 'Odd I was asked to create a branch %s that exists' % branch def pkgdb_pkg_branch(): @@ -171,8 +179,11 @@ def get_git_branch(pkg): print 'Could not find %s' % git_folder return set() - head_folder = os.path.join(git_folder, 'refs', 'heads') - return set(os.listdir(head_folder)) + branches = [ + lclbranch.replace('*', '').strip() + for lclbranch in _call_cmd(['git', 'branch']).split('\n') + ] + return set(branches) def branch_package(pkgname, branches): @@ -187,19 +198,18 @@ def branch_package(pkgname, branches): # Create the devel branch if necessary if not os.path.exists( - os.path.join(GIT_FOLDER, '%s.git/refs/heads/master' % pkgname)): + os.path.join(GIT_FOLDER, '%s.git' % pkgname)): _invoke(SETUP_PACKAGE, [pkgname]) - if 'master' in branches: - branches.remove('master') # SETUP_PACKAGE creates master - fedmsg.publish( - topic='branch', - modname='git', - msg=dict( - agent='pkgdb', - name=pkgname, - branch='master', - ), - ) + branches.remove('master') # SETUP_PACKAGE creates master + fedmsg.publish( + topic='branch', + modname='git', + msg=dict( + agent='pkgdb', + name=pkgname, + branch='master', + ), + ) # Create all the required branches for the package # Use the translated branch name until pkgdb falls inline

3 4

Freeze break request: adjust nrpe max procs on larger proxies
by Kevin Fenzi 05 Mar '15

05 Mar '15

When I moved proxies over to ansible, I made the max number of apache workers configurable, as we have some proxies that have more memory available than others. The default for the 'large' proxies is 900 apache workers. Our default nagios 'warn' is at 900 and critical is at 1000 for processes. We can and are easily hitting that limit on these larger proxies when they get a lot of requests + the normal pile of threads and such they have running all the time. So, I would like to change the nagios limits on those proxies. This will require applying the patch here and then running the proxy playbook. It should ideally result in less anoying nagios alerts where there is no action we actually need to take. +1s? kevin -- diff --git a/inventory/host_vars/proxy01.phx2.fedoraproject.org b/inventory/host_vars/proxy01.phx index 8f965b0..c9fbdcb 100644 --- a/inventory/host_vars/proxy01.phx2.fedoraproject.org +++ b/inventory/host_vars/proxy01.phx2.fedoraproject.org @@ -14,5 +14,5 @@ eth0_ip: 10.5.126.52 # This is consumed by the roles/fedora-web/main role sponsor: redhat -nrpe_procs_warn: 900 -nrpe_procs_crit: 1000 +nrpe_procs_warn: 1200 +nrpe_procs_crit: 1400 diff --git a/inventory/host_vars/proxy02.fedoraproject.org b/inventory/host_vars/proxy02.fedorapr index 4d4625e..21249f1 100644 --- a/inventory/host_vars/proxy02.fedoraproject.org +++ b/inventory/host_vars/proxy02.fedoraproject.org @@ -16,5 +16,5 @@ sponsor: internetx datacenter: internetx postfix_group: vpn -nrpe_procs_warn: 900 -nrpe_procs_crit: 1000 +nrpe_procs_warn: 1200 +nrpe_procs_crit: 1400 diff --git a/inventory/host_vars/proxy06.fedoraproject.org b/inventory/host_vars/proxy06.fedorapr index e807acb..055a56d 100644 --- a/inventory/host_vars/proxy06.fedoraproject.org +++ b/inventory/host_vars/proxy06.fedoraproject.org @@ -16,5 +16,5 @@ sponsor: osuosl datacenter: osuosl postfix_group: vpn -nrpe_procs_warn: 900 -nrpe_procs_crit: 1000 +nrpe_procs_warn: 1200 +nrpe_procs_crit: 1400 diff --git a/inventory/host_vars/proxy10.phx2.fedoraproject.org b/inventory/host_vars/proxy10.phx index 2cdcead..c16be55 100644 --- a/inventory/host_vars/proxy10.phx2.fedoraproject.org +++ b/inventory/host_vars/proxy10.phx2.fedoraproject.org @@ -13,3 +13,6 @@ eth0_ip: 10.5.126.51 # This is consumed by the roles/fedora-web/main role sponsor: redhat + +nrpe_procs_warn: 1200 +nrpe_procs_crit: 1400

4 3

Plan for today's Fedora Infrastructure meeting (2015-03-05)
by Kevin Fenzi 05 Mar '15

05 Mar '15

The infrastructure team will be having it's weekly meeting tomorrow, 2015-02-26 at 18:00 UTC in #fedora-meeting on the freenode network. This week we are trying something new. We have a gobby document (see: https://fedoraproject.org/wiki/Gobby ) fedora-infrastructure-meeting-next is the document. Please try and review and edit that document before the meeting and we will use it to have our agenda of things to discuss. A copy as of this morning is included in this email. kevin -- = Introduction = This shared document is for the next fedora infrastructure meeting. We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. = Meeting start stuff = #startmeeting Infrastructure (2015-03-05) #meetingname infrastructure #topic aloha #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk #topic New folks introductions / Apprentice feedback = Status / information / Trivia / Announcements = (We put things here we want others on the team to know, but don't need to discuss) (Please use #info <the thing> - your name) #info Trying new meeting process this week with shared document - kevin #info Figured out why squid smp mode wasnt working and posted to list - kevin #info Ipsilon is now in staging, please test and give me your problems - puiterwijk #info A large number of bug/RFEs have been coming in on anitya, the-new-hotness, and fmn. Any help appreciated. - ralph #info reminder: we are in Fedora 22 Alpha Freeze - kevin #info worked on getting gobby back in shape - kevin #info made progress on new openstack cloud and kept old one stumbling along - kevin and puiterwijk #info nagios alerts this week - http://ur1.ca/jr7j4 - kevin #info moved osu hosts around, should help nagios alerting - kevin #info fixed issue with varnish caching old wiki pages - puiterwijk = Things we should discuss = 1. kevin - new meeting process #topic New meeting process 2. kevin - fedorahosted discussion. what to do moving forward? = Learn about some application or setup in infrastructure = (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the info in this section) #topic = Action Items / todos = (Prefer probibly to make a ticket or track these outside the meeting process) 1. = Meeting end stuff = #topic Open Floor #endmeeting

1 1

Freeze Break Request: Docs Maintenance
by Pete Travis 05 Mar '15

05 Mar '15

I want to remove a bunch of docs fedorahosted trac instances that aren't being used, and update the docs.fp.o redirects in puppets to the most current versions of applicable guides. Nothing serious, just maintenance stuff that needs to be done. -- -- Pete

3 2

Freeze break request: [2 patches] Allow internal hosts to connect to the Varnish port and configure apache correctly
by Patrick Uiterwijk 05 Mar '15

05 Mar '15

========================== PATCH 1/2 ========================== commit 3f625948af36dc8047ffcbba0496bf008d77fcb5 Author: Patrick Uiterwijk <puiterwijk(a)redhat.com> Date: Thu Mar 5 00:41:37 2015 +0000 Allow direct varnish access for internal hosts This allows internal that are in the purge acl to issue purge requests. Apache won't forward purge, since it doesn't know what that is. diff --git a/inventory/group_vars/proxies b/inventory/group_vars/proxies index 3953b71..c86440a 100644 --- a/inventory/group_vars/proxies +++ b/inventory/group_vars/proxies @@ -34,10 +34,14 @@ custom_rules: [ '-A INPUT -p tcp -m tcp -s 192.168.1.59 --dport 873 -j ACCEPT', '-A INPUT -p tcp -m tcp -s 209.132.181.102 --dport 873 -j ACCEPT', - # only allow varnish from localhost + # allow varnish from localhost '-A INPUT -p tcp -m tcp -s 127.0.0.1 --dport 6081 -j ACCEPT', '-A INPUT -p tcp -m tcp -s 127.0.0.1 --dport 6082 -j ACCEPT', + # also allow varnish from internal for purge requests + '-A INPUT -p tcp -m tcp -s 192.168.1.0/24 --dport 6081 -j ACCEPT', + '-A INPUT -p tcp -m tcp -s 10.5.126.0/24 --dport 6081 -j ACCEPT', + # Allow koschei.cloud to talk to the inbound fedmsg relay. '-A INPUT -p tcp -m tcp --dport 9941 -s 209.132.184.151 -j ACCEPT', # Allow jenkins.cloud to talk to the inbound fedmsg relay. diff --git a/inventory/group_vars/proxies-stg b/inventory/group_vars/proxies-stg index 1b8fef2..2520ff1 100644 --- a/inventory/group_vars/proxies-stg +++ b/inventory/group_vars/proxies-stg @@ -33,10 +33,14 @@ custom_rules: [ '-A INPUT -p tcp -m tcp -s 10.5.126.13 --dport 873 -j ACCEPT', '-A INPUT -p tcp -m tcp -s 192.168.1.59 --dport 873 -j ACCEPT', - # only allow varnish from localhost + # allow varnish from localhost '-A INPUT -p tcp -m tcp -s 127.0.0.1 --dport 6081 -j ACCEPT', '-A INPUT -p tcp -m tcp -s 127.0.0.1 --dport 6082 -j ACCEPT', + # also allow varnish from internal for purge requests + '-A INPUT -p tcp -m tcp -s 192.168.1.0/24 --dport 6081 -j ACCEPT', + '-A INPUT -p tcp -m tcp -s 10.5.126.0/24 --dport 6081 -j ACCEPT', + # Allow koschei.cloud to talk to the inbound fedmsg relay. '-A INPUT -p tcp -m tcp --dport 9941 -s 209.132.184.151 -j ACCEPT', # Allow jenkins.cloud to talk to the inbound fedmsg relay. ========================== PATCH 2/2 ========================== commit baac9baa8b20ceda6e422acd7bf9a7f9f2a5686f Author: Patrick Uiterwijk <puiterwijk(a)redhat.com> Date: Thu Mar 5 00:56:10 2015 +0000 Fix mediawiki to determine proxies and send correct PURGE requests As commented: wgSquidServers is the set it sends a PURGE request to diff --git a/roles/mediawiki/templates/LocalSettings.php.fp.j2 b/roles/mediawiki/templates/LocalSettings.php.fp.j2 index a8e0142..b2620e0 100644 --- a/roles/mediawiki/templates/LocalSettings.php.fp.j2 +++ b/roles/mediawiki/templates/LocalSettings.php.fp.j2 @@ -322,15 +322,16 @@ $wgSkipSkins = array("chick", "cologneblue", "monobook", "myskin", "nostalgia", $wgSVGConverter = 'rsvg'; -#We use apache, but apparently it's the same difference +# This series of settings is used for reverse proxies $wgUseSquid = true; -$wgSquidServers = array( +# The SquidNoPurge setting is used to determine reverse proxies +$wgSquidServersNoPurge = array( {% if environment == "staging" %} # proxy01.stg - "10.5.126.88", + "10.5.126.88:6081", {% else %} # proxy01 - "10.5.126.52", + "10.5.126.52:6081", "192.168.1.11", # proxy02 @@ -368,7 +369,32 @@ $wgSquidServers = array( "192.168.1.17", {% endif %} ); -$wgSquidServersNoPurge = array('127.0.0.1'); +# This setting is used to send PURGE requests to varnish on reverse proxies upon page changes +$wgSquidServers = array( +{% if environment == "staging" %} + # proxy01.stg + "10.5.126.88:6081", +{% else %} + # proxy01 + "10.5.126.52:6081", + # proxy02 + "192.168.1.12:6081", + # proxy03 + "192.168.1.7:6081", + # proxy04 + "192.168.1.14:6081", + # proxy06 + "192.168.1.63:6081", + # proxy07 + "192.168.1.52:6081", + # proxy08 + "192.168.1.78:6081", + # proxy09 + "192.168.1.15:6081", + # proxy10 + "10.5.126.51:6081", +{% endif %} +); $wgSquidMaxage = 432000; # Don't add rel="nofollow"

3 3

Freeze break request: Fixing varnish purge requests
by Patrick Uiterwijk 04 Mar '15

04 Mar '15

Hi, This should fix tickets like https://fedorahosted.org/fedora-infrastructure/ticket/4679 from happening, since wiki can (and will) send a PURGE request whenever someone updates a page. I updated the IPs to include wiki01, wiki02, lockbox, and wiki01.stg and their VPN IPs. Any +1s? commit 621c373b1714f76b933b5b41253941586ea9136d Author: Patrick Uiterwijk <puiterwijk(a)redhat.com> Date: Wed Mar 4 21:31:18 2015 +0000 Fix varnish PURGE requests These are used by the wiki to purge updated pages Signed-off-by: Patrick Uiterwijk <puiterwijk(a)redhat.com> diff --git a/roles/varnish/files/proxy.vcl b/roles/varnish/files/proxy.vcl index 549d0a1..14e8846 100644 --- a/roles/varnish/files/proxy.vcl +++ b/roles/varnish/files/proxy.vcl @@ -124,33 +124,23 @@ backend mirrormanager2 { } -#acl purge { -# "192.168.1.3"; -# "192.168.1.4"; -# "192.168.1.5"; -# "192.168.1.6"; -# "192.168.1.13"; -# "192.168.1.24"; -# "192.168.1.23"; -# "192.168.1.41"; -# "10.5.126.31"; -# "10.5.126.32"; -# "10.5.126.33"; -# "10.5.126.34"; -# "10.5.126.37"; -# "10.5.126.38"; -#} +acl purge { + "10.5.126.60"; // wiki01.stg + "10.5.126.63"; // wiki01 + "10.5.126.73"; // wiki02 + "10.5.126.23"; // lockbox01 + "192.168.1.129"; // wiki01.vpn + "192.168.1.130"; // wiki02.vpn + "192.168.1.58"; //lockbox01.vpn +} sub vcl_recv { -# if (req.request == "PURGE") { -# if (!client.ip ~ purge) { -# error 405 "Not allowed."; -# } -# if (req.url ~ "^http://") { -# set req.url = regsub(req.url, "http://localhost:6081",""); -# } -# purge_url(req.url); -# } + if (req.method == "PURGE") { + if (!client.ip ~ purge) { + return (synth(405, "Not allowed")); + } + return(purge); + } if (req.url ~ "^/wiki/") { set req.backend_hint = wiki;

3 2

Freeze break: fix up gobby/sobby
by Kevin Fenzi 04 Mar '15

04 Mar '15

There's 2 issues with our sobby server: 1. It doesn't autosave/remember old documents that it has after restarting. 2. The git repo it has hasn't had any commits since 2013. ;) The following fixes both of those issues hopefully (Although I might need to tweak it a little as I have not tested it). For the autosave, we pass the autosave file on the command line when starting. For the git commits, it was using a link to /srv/sobby/sobby-commit, but /srv is noexec so that didn't work. This change instead adds a post-save-hook to just commit after every save. +1s? should be a minor issue if gobby is down or needs to be reverted back to it's current state. kevin -- diff --git a/modules/sobby/files/sobby.init b/modules/sobby/files/sobby.init index 42808cc..c246df4 100755 --- a/modules/sobby/files/sobby.init +++ b/modules/sobby/files/sobby.init @@ -32,7 +32,7 @@ start() { [ -x $exec ] || exit 5 [ -f $config ] || exit 6 echo -n $"Starting $prog: " - daemon --user sobby "$exec -c $config &" + daemon --user sobby "$exec -c $config --post-save-hook=/usr/local/bin/sobby-commit /srv/sobby/autosave.obby &" retval=$? echo [ $retval -eq 0 ] && touch $lockfile diff --git a/modules/sobby/manifests/init.pp b/modules/sobby/manifests/init.pp index b39f5a8..ff360dc 100644 --- a/modules/sobby/manifests/init.pp +++ b/modules/sobby/manifests/init.pp @@ -33,7 +33,7 @@ class sobby { source => "puppet:///sobby/sobby.init", } - file { "/srv/sobby/sobby-commit": + file { "/usr/local/bin/sobby-commit": owner => "root", group => "root", mode => 0755, @@ -53,9 +53,4 @@ class sobby { mode => 0755, ensure => directory, } - - file { "/etc/cron.hourly/sobby-commit": - ensure => link, - target => "/srv/sobby/sobby-commit", - } }

4 3

Re: Freeze break: fix up gobby/sobby
by Patrick Uiterwijk 04 Mar '15

04 Mar '15

+1 from me. Although if we go save the data, we might want to look at purging it every once in a while. But that's a problem for later. On Mar 4, 2015 3:42 PM, Kevin Fenzi <kevin(a)scrye.com> wrote: > > There's 2 issues with our sobby server: > > 1. It doesn't autosave/remember old documents that it has after > restarting. > > 2. The git repo it has hasn't had any commits since 2013. ;) > > The following fixes both of those issues hopefully (Although I might > need to tweak it a little as I have not tested it). For the autosave, > we pass the autosave file on the command line when starting. For the > git commits, it was using a link to /srv/sobby/sobby-commit, but /srv > is noexec so that didn't work. This change instead adds a > post-save-hook to just commit after every save. > > +1s? should be a minor issue if gobby is down or needs to be reverted > back to it's current state. > > kevin > -- > diff --git a/modules/sobby/files/sobby.init b/modules/sobby/files/sobby.init > index 42808cc..c246df4 100755 > --- a/modules/sobby/files/sobby.init > +++ b/modules/sobby/files/sobby.init > @@ -32,7 +32,7 @@ start() { > [ -x $exec ] || exit 5 > [ -f $config ] || exit 6 > echo -n $"Starting $prog: " > - daemon --user sobby "$exec -c $config &" > + daemon --user sobby "$exec -c $config --post-save-hook=/usr/local/bin/sobby-commit /srv/sobby/autosave.obby &" > retval=$? > echo > [ $retval -eq 0 ] && touch $lockfile > diff --git a/modules/sobby/manifests/init.pp b/modules/sobby/manifests/init.pp > index b39f5a8..ff360dc 100644 > --- a/modules/sobby/manifests/init.pp > +++ b/modules/sobby/manifests/init.pp > @@ -33,7 +33,7 @@ class sobby { > source => "puppet:///sobby/sobby.init", > } > > - file { "/srv/sobby/sobby-commit": > + file { "/usr/local/bin/sobby-commit": > owner => "root", > group => "root", > mode => 0755, > @@ -53,9 +53,4 @@ class sobby { > mode => 0755, > ensure => directory, > } > - > - file { "/etc/cron.hourly/sobby-commit": > - ensure => link, > - target => "/srv/sobby/sobby-commit", > - } > }

1 0

Re: Freeze break request: moving around osuosl machines
by Patrick Uiterwijk 04 Mar '15

04 Mar '15

+1 for meOn Mar 4, 2015 2:12 PM, Kevin Fenzi <kevin(a)scrye.com> wrote: > > Last week we moved around some osuosl vm's in order to avoid an outage > with collab (lists) and hosted. Also, we wanted to reinstall osuosl02 > anyhow (it was still rhel6/puppet). > > The osuosl work was completed monday as was the osuosl02 reinstall. > > I'd like to do the following today: > > * Move mirrorlists-osuosl back to osuosl02 and give it more cpus. > Currently it's on osuosl01 and can't seem to keep up with requests, > leading to lots of anoying alerts. > > * Proxy04 is on osuosl01 as well, I'd like to give it some more > cpus/memory once we get mirrorlists-osuosl moved. This should help > with alerting on it as well. > > +1s? > > kevin

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure March 2015