infrastructure February 2015

infrastructure@lists.fedoraproject.org

34 participants
43 discussions

firewall rules on builders (iptables, firewalld, libvirt...)
by Matthew Miller 30 Apr '15

30 Apr '15

It's my understanding (Dennis please correct if I'm wrong) that the problem with cloud image creation was due to libvirt iptables rules being lost when iptables was restarted. This is a fundamental known issue (see last paragraph of <http://libvirt.org/firewall.html>), and one of the things firewalld was meant to solve. Dennis says that there are lot of complicated rules on the builders making switching to firewalld difficult. One possibility might be to move those complicated rules from the builders to a network firewall, and keep the host rules simple and functional. But that's probably a big undertaking. In the meantime, any time iptables is restarted or reloaded, libvirt needs a SIGHUP. (I suppose this means: ansible playbooks and also added to any manual procedures.) [cc rel-eng, reply-to infrastructure] -- Matthew Miller <mattdm(a)fedoraproject.org> Fedora Project Leader

6 8

Fedora 22 Alpha Freeze now in effect
by Kevin Fenzi 11 Mar '15

11 Mar '15

Greetings. we are now in the infrastructure freeze leading up to the Fedora 22 Alpha release. This is a pre-release freeze. We do this to ensure that our infrastructure is stable and ready to release the Fedora 22 Alpha when it's available. You can see a list of hosts that do not freeze by checking out the ansible repo and running the freezelist script: git clone https://infrastructure.fedoraproject.org/infra/ansible.git scripts/freezelist -i inventory Any hosts listed as freezes is frozen until 2015-03-10. (or later if Alpha slips). Frozen hosts should have no changes made to them without a sign-off on the change from at least 2 sysadmin-main or rel-eng members, along with (in most cases) a patch of the exact change to be made to this list. Thanks, kevin

1 1

New OpenStack instance - status
by Miroslav Suchý 09 Mar '15

09 Mar '15

Since I'm leaving for one week vacation, I think I may write down current status of our new OpenStack instance and write down TODO list. Just in case someone is desperate enough to do some fixes. I updated docs.git/cloud.txt - mainly which playbooks we use right now and where to write down IP, when you add new compute node. Controller - should be OK. At least I see no problems there right now. Network is stable. I can log to EqualLogic (credentials are at bottom of cinder.conf). Volumes are created correctly. I can reach compute nodes. AMQP works and is reachable from Compute nodes (do not try to play with SSL&RabbitMQ it will never work on RHEL7). Horizon works (over https). Compute nodes - it looks good until you try to start VM. :) I fixed several problems, but new ones still pop ups. If you want to debug it, just go to dashboard and start new VM (note that m1.tiny is too small for Fedora image) and on controller do: tail -f /var/log/nova/nova-scheduler.log And look for something like: Choosing host WeighedHost [host: fed-cloud13.cloud.fedoraproject.org, weight: 1.0] for instance 75f1b5ca-88d5-4e57-8c18-8d6554e1f2bc then log to that instance (right now root@fed-cloud09 can ssh directly as root@fed-cloudXX) and tail -f /var/log/nova/nova-compute.log /var/log/neutron/openvswitch-agent.log When spin up of VM fail, then controller try 2 next machines before giving up. Right now there is some error: TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'\n" which is new to me and which I will not manage to fix before I will leave today. It may be last one problem or they may be dozen other still waiting in queue. It's hard to tell. Smaller fixes to do: * playbook hosts/fed-cloud09.cloud.fedoraproject.org.yml can be enhanced that after packstack execution the machine should be restarted. Right now I am waiting for first error after packstack and then I restart the machine manualy and re-run playbook again. This is last manual workaround. Everything else was already automated. * routing between compute nodes and controller using public IP does not work. Not fatal right now, but nice to have. -- Miroslav Suchy, RHCE, RHCDS Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

3 18

Freeze break request: Make setup_git_package also handle recreating the repo
by Patrick マルタインアンドレアス Uiterwijk 27 Feb '15

27 Feb '15

>From 6e98a2dc667365633e47c63730feefc8d752886c Mon Sep 17 00:00:00 2001 From: Patrick Uiterwijk <puiterwijk(a)redhat.com> Date: Fri, 27 Feb 2015 23:39:59 +0000 Subject: [PATCH] Fix setup_git_package to handle setting up after gitolite setup a broken repo setup_git_package did the same check as pkgdb_sync did [1]. Also, if gitolite already came by, it will have created an update hook, which blocks us from pushing the initial commit (since gitolite has no idea who we are). So in that case, we will just remove the update hook: gitolite will recreate it when it comes by the repo with genacls.sh. The change to mkdir -p is just syntax to make it not print an error in case the directory already existed. [1]: https://lists.fedoraproject.org/pipermail/infrastructure/2015-February/0156… Signed-off-by: Patrick Uiterwijk <puiterwijk(a)redhat.com> --- roles/distgit/files/setup_git_package | 10 ++++++++-- 1 files changed, 8 insertions(+), 2 deletions(-) diff --git a/roles/distgit/files/setup_git_package b/roles/distgit/files/setup_git_package index b22b31f..67103ec 100644 --- a/roles/distgit/files/setup_git_package +++ b/roles/distgit/files/setup_git_package @@ -68,11 +68,17 @@ fi # Sanity checks before we start doing damage [ $VERBOSE -gt 1 ] && echo "Checking package $PACKAGE..." -if [ -d $GITROOT/$PACKAGE.git ] ; then +if [ -f $GITROOT/$PACKAGE.git/refs/heads/master ] ; then echo "ERROR: Package module $PACKAGE already exists!" >&2 exit -1 fi +# A cleanup in case gitolite came by this repo +if [ -f $GITROOT/$PACKAGE.git/hooks/update ] ; then + echo "Gitolite already initialized this repo. Will remove its hooks" + rm -f $GITROOT/$PACKAGE.git/hooks/update +fi + # "global" permissions check if [ ! -w $GITROOT ] ; then echo "ERROR: You can not write to $GITROOT" @@ -85,7 +91,7 @@ fi TMPDIR=$(mktemp -d /tmp/tmpXXXXXX) # First create the master repo -mkdir $GITROOT/$PACKAGE.git +mkdir -p $GITROOT/$PACKAGE.git pushd $GITROOT/$PACKAGE.git >/dev/null git init -q --shared --bare echo "$PACKAGE" > description # This is used to figure out who to send mail to. -- 1.7.2.1

3 2

Freeze break request: fixing race condition breakage for pkgdb_sync
by Patrick Uiterwijk 27 Feb '15

27 Feb '15

Any +1s? Explanation in the commit message. Note: we should work on improving the script in general, but that is not something we should consider pushing during freeze, so this is a patch that hopefully gets us through the freeze. commit 2d98b6748663747698cd6a0bfc23d05bbc371a01 Author: Patrick Uiterwijk <puiterwijk(a)redhat.com> Date: Fri Feb 27 08:44:57 2015 +0000 Make pkgdb_sync more resilient by always calling SETUP_PACKAGE if master does not exist This should make it recover in case there are any repos in a weird state (aka, no master branch). This could for example happen if the package was created in pkgdb after pkgdb_sync ran, but before genacls.pkgdb runs, because then gitolite will see it as a new repo, and create a totally blank repo. Especially since pkgdb_sync will abort after the first error, it would then leave all later created repos broken as well. Calling setup_git_package in a repo without master branch has no ill effect: - Running git init on a pre-initialized repo is a no-op (or at least doesn't destroy current git data) - Since it doesn't have master, it didn't run the script, so adding the first commit and hooks etc are what we want the script to do Signed-off-by: Patrick Uiterwijk <puiterwijk(a)redhat.com> diff --git a/roles/distgit/templates/pkgdb_sync_git_branches.py b/roles/distgit/templates/pkgdb_sync_git_branches.py index ced139b..8643165 100644 --- a/roles/distgit/templates/pkgdb_sync_git_branches.py +++ b/roles/distgit/templates/pkgdb_sync_git_branches.py @@ -187,7 +187,7 @@ def branch_package(pkgname, branches): # Create the devel branch if necessary if not os.path.exists( - os.path.join(GIT_FOLDER, '%s.git' % pkgname)): + os.path.join(GIT_FOLDER, '%s.git/refs/heads/master' % pkgname)): _invoke(SETUP_PACKAGE, [pkgname]) if 'master' in branches: branches.remove('master') # SETUP_PACKAGE creates master

3 3

retroactive freeze break request: kojipkgs01
by Kevin Fenzi 26 Feb '15

26 Feb '15

We have been having some persistent issues with kojipkgs01 lately. kojipkgs01 is our squid proxy in front of koji builds. It allows users and builders to get fast access to packages. (When it's working). Lately, it's been working fine at first, then in a few days or so it starts getting really slow. Downloads go from 25M/s to 200k/sec and sometimes things even just timeout. Restarting squid seems to fix this... for a few more days. There is never any errors on the box, i/o, load and everything is fine. I looked this morning a bunch at options and adjusted the memory cache down in case we were hitting some kind of issue with memory cache. I'd like +1's for that change, and also to solicit ideas for what we can do to fix this once and for all (if these changes don't do so). diff --git a/roles/kojipkgs/files/squid.conf b/roles/kojipkgs/files/squid.conf index b011143..a0d5312 100644 --- a/roles/kojipkgs/files/squid.conf +++ b/roles/kojipkgs/files/squid.conf @@ -6,8 +6,8 @@ hierarchy_stoplist cgi-bin ? cache_swap_low 98 cache_swap_high 99 -cache_mem 50 GB -maximum_object_size 700 MB +cache_mem 10 GB +maximum_object_size 200 MB minimum_object_size 0 KB cache_replacement_policy heap LFUDA maximum_object_size_in_memory 100 MB

4 4

Next weeks meeting - shared document info
by Kevin Fenzi 26 Feb '15

26 Feb '15

Greetings. In the meeting today I talked about restructuring how we do our meetings some. The goal would be to make the meetings more about things we want to discuss than gathering status and such, as well as letting us remember better what we wanted to discuss or note. So, I have made a document in our gobby instance: https://fedoraproject.org/wiki/Gobby Called fedora-infrastructure-meeting-next If you don't have access to fedorapeople to get the super sekret password, just ping someone who does. :) Basically over the next week if people could login and add things to that document as they go we can use it for the next weeks meeting. I've attached the current contents here, and when I send out the meeting reminder next wed, I will copy out the document then too. Hopefully it makes sense and helps us out. ;) ideas and changes welcome. kevin -- = Introduction = This shared document is for the next fedora infrastructure meeting. We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. = Meeting start stuff = #startmeeting Infrastructure (2015-03-05) #meetingname infrastructure #topic aloha #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk #topic New folks introductions / Apprentice feedback = Status / information / Trivia / Announcements = (We put things here we want others on the team to know, but don't need to discuss) (Please use #info <the thing> - your name) #info Trying new meeting process this week with shared document - kevin = Things we should discuss = 1. kevin - new meeting process #topic New meeting process 2. = Learn about some application or setup in infrastructure = (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the info in this section) #topic = Action Items / todos = (Prefer probibly to make a ticket or track these outside the meeting process) 1. = Meeting end stuff = #topic Open Floor #endmeeting

1 0

Plan for tomorrow's Fedora Infrastructure meeting (2015-02-26)
by Kevin Fenzi 26 Feb '15

26 Feb '15

The infrastructure team will be having it's weekly meeting tomorrow, 2015-02-26 at 18:00 UTC in #fedora-meeting on the freenode network. Suggested topics: #topic New folks introductions and Apprentice tasks. If any new folks want to give a quick one line bio or any apprentices would like to ask general questions, they can do so in this part of the meeting. Don't be shy! #topic Applications status / discussion Check in on status of our applications: pkgdb, fas, bodhi, koji, community, voting, tagger, packager, dpsearch, etc. If there's new releases, bugs we need to work around or things to note. #topic Sysadmin status / discussion Here we talk about sysadmin related happenings from the previous week, or things that are upcoming. #topic nagios/alerts recap Here we go over the last weeks alerts and see if we can find ways to make it so they don't happen again. http://ur1.ca/jr7j4 #topic Upcoming Tasks/Items https://apps.fedoraproject.org/calendar/list/infrastructure/ #topic Open Floor Submit your agenda items, as tickets in the trac instance and send a note replying to this thread. More info here: https://fedoraproject.org/wiki/Infrastructure/Meetings#Meetings Thanks kevin

1 1

Fwd: To Whomever: You Made My Day
by Stephen John Smoogen 26 Feb '15

26 Feb '15

The little things hidden in plain view make the best of a bad day. Thank you to the people involved. ---------- Forwarded message ---------- From: John Florian <john.florian(a)dart.biz> Date: 26 February 2015 at 06:31 Subject: To Whomever: You Made My Day To: devel <devel(a)lists.fedoraproject.org> Yesterday I was browsing the Fedora pkgdb/git/bohdi pages and this morning I returned to go backwards thru my web browser history when I stumbled upon a real hidden gem for a HTTP 500 response. Our hot dog armed with a ray gun against a nuclear panda… oh man that was great. Best 500 I’ve seen yet. -- John Florian -- devel mailing list devel(a)lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- Stephen J Smoogen.

1 0

[release] anitya 0.2.0
by Pierre-Yves Chibon 26 Feb '15

26 Feb '15

Hi all, Since anitya is not frozen and I had a bunch of changes merged, I just cut a new release and push it to prod. Here is the changelog: * Thu Feb 26 2015 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 0.2.0 - Fix doc typos (reported by tibbs, fixed by Ralph Bean) - Fix typo when unable to retrieve the latest version (Thomas Spura) - Autocomplete distribution name when mapping a project - Dynamically check existing projects while adding a new one - Check now button available after an edit - More information for the custom backend - Fix the drupal backends to make them suppor project named 'drupal6: foo' or 'drupal7: bar' - Fix pagination on quite some pages - Allow users to browse their own log - On postgresql '_' is a special character that we should escape - User friendly url for example: /project/guake - More info when mapping a project fails (including a link to the project causing the failure to map) Enjoy! Pierre

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure February 2015