infrastructure January 2011

infrastructure@lists.fedoraproject.org

35 participants
19 discussions

Meeting Re: Eucalyptus hosting
by Garrett Holmstrom 18 Jan '11

18 Jan '11

Hi, At the infrastructure meeting yesterday Seth brought up the topic of cloud hosting for Fedora services. With EC2 and Rackspace being closed platforms, I suggested Eucalyptus as an alternative. I wasn't able to find any hosting providers in the US that use it, but Rich Wolski, the CTO of Eucalyptus Systems, expressed an interest in chatting with you infrastructure folks about whether or not we at Eucalyptus can just host it ourselves, depending on Fedora's needs and such. Would there be a good time and place to discuss this? The weekly IRC meeting? (For Rich's benefit: the weekly IRC meeting is at noon PST on Thursdays in #fedora-meeting on irc.freenode.net.) Another IRC meeting in case the weekly one would be too pressed for time? A conference call? Might an email thread be better? -- Garrett Holmstrom

3 3

Wiki extension request
by Adam Williamson 17 Jan '11

17 Jan '11

Hi, everyone. I'd like to ask about adding an extension to the Fedora wiki instance. Many of you may have noticed the package-specific test case project I've been working on lately - https://fedorahosted.org/fedora-qa/ticket/154 . Part of the plan for this project is to allow tools such as Bodhi to use mediawiki categories to figure out certain sets of test cases. One important one is 'all critical path test cases for package foo'. However, this requires the ability to do multi-category search in the wiki - find all pages that are members of two (or more) different categories. We've identified a couple of extensions which can potentially achieve this: http://www.mediawiki.org/wiki/Extension:Multi-Category_Search http://www.mediawiki.org/wiki/Extension:DynamicPageList_%28Wikimedia%29 I'd like to ask that we look at including one of these (or someone tell me that we already do :>). There are a couple of concerns that I don't have sufficient expertise to address. There would need to be some kind of, I guess, API access to the searches, so that an external tool could do the search - both extensions seem rather geared to doing the search entirely within the Wiki, for instance, including the results of the search in another page. I don't know how to tell which, if either, extension is most suited to this. The other concern is recursion - does either extension handle this case: page is a member of categories A and Z, but not directly of category B category A is a member of category B if searching for 'all pages that are in category B and Z', does the page show up? For our purposes I'd like that the be the case. If anyone can help with this it'd be greatly appreciated! Thanks. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net

2 2

Querying FAS details through IRC
by Misha Shnurapet 13 Jan '11

13 Jan '11

Hello. It is possible to query contributors' information on IRC with a command that fetches data from FAS. While it's OK to provide extensive information on users through the web interface (which is now only possible for the community members, AFAIK, as it requires to log in), I think it might come reasonable not to provide email addresses on such queries on IRC. -- Best regards, Misha Shnurapet, Fedora Project Contributor http://fedoraproject.org/wiki/User:Shnurapet shnurapet AT fedoraproject.org, GPG: 00217306

6 8

Outage notes
by Seth Vidal 12 Jan '11

12 Jan '11

Hi Everyone, I took some notes while we were rebooting boxes I wanted to share them with everyone for future outages. Ordering of the bounces: 1. xen14: puppet is on there and if that is back up first we have a place to stand for pushing out any changes (dns changes for example via puppet) - xen14 takes about 4 minutes to restart/POST 2. xen15: bastion01, db02 are on there - same 4 minute restart window once this is up you'll want to logout of bastion02 and into bastion01 so you have a firm place to do the xen05 reboots from which will take out bastion02. 3. edit dns on puppet to remove proxy01 from the wildcard/roundrobin and push that to the ns* servers and verify. 4. xen05: bastion02 (openvpn), proxy01 - 4-5 minutes for this machine to restart. once xen05 is completely up log back in and verify the vpn is back online 5. edit dns remove all the other proxy hosts and put proxy01 back in. Push and verify 6. virthost01 - I had to halt each of the kvms from a login - virsh shutdown didn't work. - 4 minute restart time on the hw. Note: make sure virthost01 is completely up - especially fas03. since taking down virthost02 next will take out fas02- you want to make sure you don't leave fas01 all by itself. 7. virthost02 - fas02 was not setup to autostart - that's now fixed. 8. virthost13 - uneventful 9. xen03 - spin01 spewed lots of umount issues - those are from the spin creation paths - they can be safely ignored - fas01.stg was running on xen03 according to the logs but there's no definition for it on the system - so not sure what the story is there. - neither of the other staging hosts were set to autostart 10. xen04: we apparently have a number of hosts w/only one dns record internally and they point to ns03 only. B/c when ns03 went away - lots of things got VERY VERY SLOW trying to resolve names. This is on my list to address. You must wait for xen04 to be completely up and ns03 running before you can take down xen07. Otherwise we'll be w/o dns internally to phx. 11. xen07: iscsi disks didn't come up right away - this kept ns04 from coming up immediately - needed to run /etc/init.d/iscsi start and they showed up. 12. xen09: uneventful 13. xen10: log01 needed an fsck b/c of the time since last mount - this took a long time. 14. xen11: secondary1 needed an fsck. also a 5-6 minute hw reboot time. 15. xen12: db1->db01 naming change kept it from coming up at boot b/c of the 'auto' symlink to db1. db01 had to fsck 16. cnode01 - 6-7 minute reboot time - nothing was set to autostart in xen - this is now fixed - autoqa01 and dhcp02 are set to autostart 17. db03: fsck took FOREVER to complete and this takes a lot of things done - for the future move db03 reboot higher up the stack, just in case. This machine's restart/POST time is REALLY high like 7-10 minutes. The console for it is less than forthcoming, too. 18. backup01: uneventful At this point internal was back online - except for the build xen systems and servers. External hosts: 19. - bodhost01: 5-6 minute machine reboot time - people01 - uneventful. - ibiblio01 - 5-7 minute machine reboot time. uneventful 20. - internetx01: uneventful - osuosl01: uneventful 21. - sb2 - must wait for ibiblio01 to be up b/c of not having any external name servers - sb3 - uneventful - sb4 - hosted1 listed more 'maxmem' in its config that sb4 had available - so that had to be edited down. Not sure how that EVER started - sb5 - uneventful 22. telia01 - proxy5 did not restart on its own - unknown as to WHY yet - but it did start manually. - retrace01 was not set to autostart tummy1 - uneventful Now all the proxy* rebooting is over so we can: 23. edit dns: put the other proxy hosts in the wildcard/RR - push and verify Build boxes: - bxen03 had koji2 listed in its set of hosts - but it wasn't running. This led to some confusion as to how to start the hosts on bxen03 b/c of insufficient memory for all guests. Eventually I realized bxen04 is where koji02 was running and that the left over guest file was never cleaned up on bxen03. Things to think about post-outage: - check all the raid arrays for lost disks - we saw this a couple of times - it's not pleasant. - check for downed vpns and/or broken resolution - we need to get a firm handle on why this is a hassle so often. Overall things to think about for the future: 1. dumping a complete virsh list - including how much memory is actually being used per vm per server before we start reboots 2. checking what disks need fscks because of mounted time and doing those earlier or separately. 3. verifying that all running vms are: a. intended to be running b. have a config file c. are set to autostart 4. verifying that all NOT running vms are: a. intended to be off b. are NOT set to autostart thoughts welcome. -sv

4 6

Outage: Fedora related services 2010-01-11 1900 UTC -> 2300 UTC.
by Stephen John Smoogen 11 Jan '11

11 Jan '11

Outage: Rolling update outages - 2010-01-11 19:00 UTC There will be an outage starting at 2010-01-11 19:00 UTC, which will last approximately 4 hours. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2010-01-11 19:00 UTC' Reason for outage: Kernel and glibc updates require servers to be rebooted. Outages will be occurring on our virtual servers requiring all services to be down at some point. However due to rolling nature of outages no service should be affected long or deeply. Affected Services: BFO - http://boot.fedoraproject.org/ Bodhi - https://admin.fedoraproject.org/updates/ Buildsystem - http://koji.fedoraproject.org/ CVS / Source Control DNS - ns1.fedoraproject.org, ns2.fedoraproject.org Docs - http://docs.fedoraproject.org/ Email system Fedora Account System - https://admin.fedoraproject.org/accounts/ Fedora Community - https://admin.fedoraproject.org/community/ Fedora Hosted - https://fedorahosted.org/ Fedora People - http://fedorapeople.org/ Fedora Talk - http://talk.fedoraproject.org/ Main Website - http://fedoraproject.org/ Mirror List - https://mirrors.fedoraproject.org/ Mirror Manager - https://admin.fedoraproject.org/mirrormanager/ Package Database - https://admin.fedoraproject.org/pkgdb/ Smolt - http://smolts.org/ Spins - http://spins.fedoraproject.org/ Start - http://start.fedoraproject.org/ Torrent - http://torrent.fedoraproject.org/ Translation Services - http://translate.fedoraproject.org/ Wiki - http://fedoraproject.org/wiki/ Unaffected Services: Ticket Link: https://fedorahosted.org/fedora-infrastructure/ticket/2554 Contact Information: Please join #fedora-admin in irc.freenode.net or respond to this email to track the status of this outage. -- Stephen J Smoogen. "The core skill of innovators is error recovery, not failure avoidance." Randy Nelson, President of Pixar University. "Let us be kind, one to another, for most of us are fighting a hard battle." -- Ian MacLaren

1 1

Turbogears downgraded on fas servers
by Toshio Kuratomi 10 Jan '11

10 Jan '11

I've re-downgraded TurboGears to 1.0.9 on the fas servers. Another incompatibility was discovered i nthe TG-.1.1.x series. EPEL6 is going out of beta soon so on Monday I'll need to decide whether to revert TG in EPEL6 to TG-1.0.x or work on fixing fas further. -Toshio

7 13

REMINDER: publictest box motd [this means you :)]
by Stephen John Smoogen 05 Jan '11

05 Jan '11

Ok when testing a project it is important to update the /etc/motd file with the following information: [fas username] - [date started] [date updated] project title - project summary directories/areas looked at. Example: Project Owner: smooge - Started: 2010-12-31 Updated: 2010-01-30 gnusong - a project to digitize the reading of all source code to a dance beat. /usr/local/bin, /srv/web/songs 1) Systems that DO NOT have updated motd's will be wiped, reinstalled to a fresh state and turned off until an RFR is needed for it. Systems whose last update was over 90 days will also be subject to wiping/reinstall/turn off. 2) publictest systems are NOT backed up. they are test/scratch systems that are meant for eventual integration with some sort of development/staging infrastructure. 3) due to communal nature of systems, people need to work together on them to make sure they are not overwriting and beating each other up. -- Stephen J Smoogen. "The core skill of innovators is error recovery, not failure avoidance." Randy Nelson, President of Pixar University. "Let us be kind, one to another, for most of us are fighting a hard battle." -- Ian MacLaren

5 5

delta rpm compose debugging change
by Kevin Fenzi 03 Jan '11

03 Jan '11

Greetings. Of late openoffice.org and a few other packages have been not generating delta rpms in pushes. ;( They error with: Error genDeltaRPM for chunkd: exitcode was 256 - Reported Error: bad cpio archive But looking later at both rpms, things seem fine. ;( I've talked with jdieter and we would like to add in more debugging: <jdieter> nirik: We're still having the "bad cpio archive" with openoffice.org updates <jdieter> I have a patched version of deltarpm available at: http://www.lesloueizeh.com/jdieter/deltarpm-3.6-0.1.20101230git.fc14.src.rpm <jdieter> If run with -vv, it will report old rpm, new rpm, and tell us what the actual first six bytes of what's actually in the rpm <jdieter> vs the 070701 that it's supposed to be <jdieter> I don't have an el6 system, so it will have to be rebuilt for releng2 <jdieter> /usr/lib/python2.4/site-packages/createrepo/deltarpms.py needs to modified as well <jdieter> The line with "makedeltarpm -v" needs to be changed to "makedeltarpm -vv" epel5 scratch build at: http://koji.fedoraproject.org/koji/taskinfo?taskID=2694315 (releng02 is rhel5). If I can get enough +'s I'd be happy to put this in place whenever. Otherwise we can wait until next week after the freeze. Thoughts? Comments? Other ideas? kevin

3 4

Back
by Stephen John Smoogen 01 Jan '11

01 Jan '11

I am back and starting to catch up with emails. I ended up with almost 0 internet access last week so thanks to those who covered and watched ibiblio go up and down ( I was drawing talking with Crush the Turtle at the time :)). -- Stephen J Smoogen. "The core skill of innovators is error recovery, not failure avoidance." Randy Nelson, President of Pixar University. "Let us be kind, one to another, for most of us are fighting a hard battle." -- Ian MacLaren

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure January 2011