infrastructure October 2008

infrastructure@lists.fedoraproject.org

37 participants
59 discussions

Outage Notification - 2008-10-15 13:30 UTC
by Mike McGrath 14 Oct '08

14 Oct '08

There will be an outage starting at 2008-10-15 13:30 UTC, which will last approximately 1 hour. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2008-10-15 13:30 UTC' Affected Services: Buildsystem Database Fedora Hosted Translation Services Websites Unaffected Services: DNS CVS / Source Control Fedora People Fedora Talk Mail Mirror System Torrent Ticket Link: https://fedorahosted.org/fedora-infrastructure/ticket/911 Reason for Outage: Migrating to postgres 8.3. This should require minutes of outage time. We're going to do a dump, delete, update, import. Doing this one in the morning because its short, so we can monitor it during the day instead of have it fail at night, and so we don't interrupt any of the builds or mashes that would happen in the evening. Actual downtime should be about 5-10 minutes Contact Information: Please join #fedora-admin in irc.freenode.net or respond to this email to track the status of this outage.

1 0

Maps
by Mike McGrath 14 Oct '08

14 Oct '08

Anyone interested in getting our fedoraproject.org/maps/ infrastructure back in place? It was up and running in FC6, after the rebuild and no more FC6 boxes though, we no longer have active maps. -Mike

4 5

pre-release freeze on the 21st
by Mike McGrath 13 Oct '08

13 Oct '08

Just a note to everyone, as long as we stay on schedule another pre-release freeze will happen starting on the 21st ending on the 5th. The final full infrastructure freeze does not begin until the 11th. I'd also encourage those with the proper skill set to focus on rawhide during these times, tackle the final bugs, focus on blockers, QA, etc. -Mike

2 2

How to upload to alt.fedoraproject.org?
by Warren Togami 13 Oct '08

13 Oct '08

http://alt.fedoraproject.org/pub/alt/ltsp/beta1/i686/ Where is it documented how I can upload to this tree? I will be uploading "beta2" soon. Warren Togami wtogami(a)redhat.com

2 1

Outage Notification - 2008-10-11 03:00 UTC
by Seth Vidal 10 Oct '08

10 Oct '08

Outage Notification - 2008-10-11 03:00 UTC There will be an outage starting at 2008-10-11 03:00 UTC, which will last approximately 2 hours. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2008-10-11 03:00 UTC' Affected Services: Fedora People Unaffected Services: Buildsystem CVS / Source Control Database DNS Fedora Hosted Fedora Talk Mail Mirror System Torrent Translation Services Websites Reason for Outage: Migrating fedorapeople.org from Duke University hosting to Boston University hosting. Contact Information: Please join #fedora-admin in irc.freenode.net or respond to this email to track the status of this outage.

1 1

Meeting Log - 2008-10-09
by Ricky Zhou 10 Oct '08

10 Oct '08

20:00 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Who's here? 20:00 < G> boo 20:00 < dgilmore> not I 20:00 < jcollie> hello 20:00 < mdomsch> barely 20:00 * ValHolla grabs a seat 20:00 < mmcgrath> G: boo? 20:01 < mmcgrath> its not halloween yet! 20:01 < G> mmcgrath: :) 20:01 * ivazquez is here for once 20:01 < mmcgrath> heh, lets look at the tickets. 20:01 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- The Dow is way down 20:01 -!- ValHolla [i=0c29727e(a)gateway/web/ajax/mibbit.com/x-d4aae136aecdd915] has quit Client Quit 20:02 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Tickets 20:02 < mmcgrath> .tiny https://fedorahosted.org/fedora-infrastructure/query?status=new&status=assi… 20:02 < zodbot> mmcgrath: http://tinyurl.com/2hyyz6 20:02 -!- ValHolla [i=0c29727e(a)gateway/web/ajax/mibbit.com/x-8aa88275c5dc8cbb] has joined #fedora-meeting 20:02 < mmcgrath> .ticket 395 20:02 < zodbot> mmcgrath: #395 (Audio Streaming of Fedora Board Conference Calls) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/395 20:02 < mmcgrath> jcollie: ping 20:02 < jcollie> nothin' :) 20:02 < mmcgrath> heheh k 20:02 < mmcgrath> .ticket 446 20:02 < zodbot> mmcgrath: #446 (Possibility to add external links on spins page) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/446 20:02 < mmcgrath> dgilmore: ^^^ 20:02 * f13 is here 20:03 * ValHolla sits back down 20:03 < kcadieux> hello 20:04 -!- JSchmitt [n=s4504kr(a)p4FDD213F.dip0.t-ipconnect.de] has joined #fedora-meeting 20:04 * ricky 20:05 < jcollie> i thought we were going to close this one? 20:05 < mmcgrath> we'll skip that for now 20:05 < mmcgrath> I thought so too. 20:05 * mmcgrath also hasn't heard much on 20:05 < mmcgrath> .ticket 740 20:05 < zodbot> mmcgrath: #740 (Loaning out system time to OLPC participants) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/740 20:05 < mmcgrath> ok so 20:06 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- App farm and new bapp servers 20:06 < mdomsch> yea! 20:06 < mmcgrath> Did anyone have any questions about how our new farm works, bapp, etc? 20:06 < mmcgrath> any comments? 20:06 -!- brothers [n=brothers(a)66-234-43-147.nyc.cable.nyct.net] has joined #fedora-meeting 20:06 -!- finalzone [n=luya@fedora/Finalzone] has joined #fedora-meeting 20:06 < mdomsch> bapp* will be in phx 20:07 < mdomsch> we expect 20:07 < mmcgrath> likely 20:07 < mdomsch> ? 20:07 < mdomsch> good 20:07 < mmcgrath> There's still some jobs to move to bapp1, in particular the damned lies stuff 20:08 < mmcgrath> K, well if there's nothing else there.. 20:08 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- telia1 and ibiblio1 20:08 < mmcgrath> We now have proxy servers in 4 locations. 20:09 < mmcgrath> 1 in germany, 3 in the states. 20:09 < mmcgrath> 5 proxy servers total (two of them are in PHX) 20:09 < mmcgrath> this will give us far more flexability during problems as well as distribute network load more evenly on release day. 20:09 < mmcgrath> it will also let us take a closer look at geoIP and DNS 20:09 < G> mmcgrath: what happens if WWI happens all over again? :) 20:10 < mmcgrath> we'll have to find an oil rig 20:10 < mmcgrath> or one of google's new cruise ships :) 20:10 < mmcgrath> At present, I'm thinking that 5 proxy servers more then suits our needs. 20:10 < G> sounds good, I thought maybe host a server in Iraq or something :) 20:11 < jcollie> G: i nominate anyone but me to go install that one 20:11 < mmcgrath> I'd like to focus on our caching policies and stuff. But I don't see a need to add additional proxy servers at this time. And if a new site comes up in asia, europe or south america, we may move one of our proxy servers there. 20:11 < G> mmcgrath: from an APAC view, I'd like to be bias and say, one somewhere out this way would be nice 20:11 < mmcgrath> G: yeah, that area could certainly use one 20:12 < jcollie> something in africa would be good too, but that'd be almost as fun as the iraq one 20:12 < mmcgrath> heh 20:12 < G> if we can get a site, Australia/New Zealand/Singapore/Japan or something we'd be covering all the regions 20:12 < mmcgrath> if there's nothing else there we'll move on :) 20:12 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Duke, planet, buildsys, people, torrent. 20:13 < jcollie> if fedora on olpc takes off there could be a lot if activity from africa 20:13 < G> isn't buildsys at SB now? 20:13 < mmcgrath> At this time the old buildsys and torrent boxes are no longer being used (hosted at duke) 20:13 < G> jcollie: true 20:13 < mmcgrath> G: it is. 20:13 < mmcgrath> seth is also working to move planet and people to a combined machine at Boston University. 20:13 < G> k, I was thinking I was going even more potty :) 20:13 < mmcgrath> this is for a couple of reasins, the primary one being that the hardware that people and planet is on right now are just woefully out of date. 20:13 < jcollie> are the systems at duke worth re-purposing? 20:14 < mmcgrath> jcollie: not that we can use but, IIRC, they've already been taken care of. 20:14 * mmcgrath is almost afraid to know how old that torrent box is. 20:14 < mmcgrath> but new torrent is in ibiblio 20:14 < mmcgrath> buildsys is at SB 20:14 < G> mmcgrath: loan boxes to OLPC :) 20:14 < mmcgrath> heh 20:15 < mmcgrath> so thats just kind of a heads up. I'm hoping to update our global map to reflect that soon as its completed. 20:15 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Puppet (sysctl) 20:15 -!- tibbs [n=tibbs@fedora/tibbs] has quit "Konversation terminated!" 20:16 < mmcgrath> For those of you that aren't familiar we added sysctl support to puppet a couple days back. I'm going to be making more change that as we go forward. 20:16 < mmcgrath> I'm also, very slowly, completing our conversion to module 20:16 < mmcgrath> s 20:16 < mmcgrath> Anyone have any questions on that? or anything thats come up since puppet training? 20:17 < ricky> We should push for high quality modules that we can make public 20:17 * jcollie says modules good 20:17 < mmcgrath> ricky: yeah 20:17 < ricky> Some modules, like the haproxy one contain Fedora-specific configs, so we need to think about how to do that the "puppet" way 20:17 < mmcgrath> ricky: well, even that module is useful. Just takes someone to alter that file (which they'd have to do anyway for their environment) 20:18 < mmcgrath> it gets trickier if haproxy required something strange that was specific to Fedora 20:18 < mmcgrath> Ok, anyone have anything else on that? 20:19 < mmcgrath> Ok, well thats really all I had this week. 20:19 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Open Floor 20:19 < mmcgrath> anyone have anything they'd like to discuss related to Infrastructure ? 20:19 < abadger1999> publictest machines. 20:19 < mmcgrath> abadger1999: sure, whats up? 20:20 < abadger1999> How many do we have up now? Do we have resources to make more? 20:20 < abadger1999> Is there a SOP? :-) 20:20 < ricky> The new "rule" is that they just stay out of PHX, right? 20:20 -!- JSchmitt [n=s4504kr@fedora/JSchmitt] has quit "Konversation terminated!" 20:20 < mmcgrath> I think we just have one or two up, and we can make more at telia1 and ibiblio1 20:20 < mmcgrath> abadger1999: no SOP really, just the kickstart and dns SOP's are needed. 20:21 -!- che [n=rkastl@redhat/che] has joined #fedora-meeting 20:21 < mmcgrath> abadger1999: do you want one? I can walk you through making one if you have any questions, or I can just throw one together. 20:22 < abadger1999> mmcgrath: You shouuld probably walk me through making one. 20:22 < jcollie> do you have to be in -main to create a new PT system? 20:22 < G> jcollie: yep 20:22 < G> jcollie: shall we say "damn" together ? :P 20:22 < abadger1999> I think dmalcolm's rpmgrok and rnorwood's amber have yet to land somewhere post-incident. 20:22 < abadger1999> ricky: Am I right on that? 20:22 < G> abadger1999: thats my understanding... 20:22 < ricky> Yup. 20:22 -!- fbijlsma [n=fbijlsma(a)92.50.110.191] has quit "Leaving" 20:23 < G> abadger1999: + QA don't have anywhere for testopia/etc\ 20:23 < G> to errr test anyway 20:23 < mmcgrath> abadger1999: I'd assume it had ended up on pt15. 20:24 < ricky> Haha, we all logged in at the same minute :-) 20:24 < abadger1999> heh :-) 20:25 < abadger1999> Hmm.... Okay, there's plenty of free memory there right now. 20:25 < abadger1999> I'll get those guys using it and you can walk me through creating a new pt instance if they outgrow it. 20:26 < mmcgrath> sure thing 20:26 < mmcgrath> Ok, well, we're only about a half hour in. Does anyone else have anything to discuss? if not we'll close the meeting early 20:26 < abadger1999> Cool. That's all I have. 20:27 * G thinks 20:27 < ricky> Infrastructure contact info? :-) 20:28 < G> wwoods says they can't use Testopia because of legal 20:28 < mmcgrath> ricky: I actually have that. I'll email you after the meeting 20:28 < ricky> Coool 20:28 < G> wwoods: are you guys actually looking at any other options? 20:29 < ricky> Compose hosts stuff done? Will F10 be made entirely on our infrastructure? :-) 20:29 < wwoods> at the moment, no. dmalcolm was working on a new frontend that used the same database 20:29 < wwoods> but I haven't heard anything about that in a couple of months 20:29 < G> k, so I guess QA doesn't need a PT machine then 20:29 < wwoods> at the moment, no 20:30 < G> wwoods: I guess that means you guys want the QA wiki namespace? 20:30 * G recalls that as the alternative 20:31 < mmcgrath> well, if they do need one wwoods knows who to tell :) 20:31 < wwoods> too busy dealing with F10 release process stuff to mess with that now, but... exactly 20:31 < mmcgrath> Anyone have anything else to discuss? If not I'll close the meeting in 30 20:31 < G> mmcgrath: nope 20:32 < mmcgrath> 15 20:32 < G> except git is a pain... 20:32 < mmcgrath> yes it is :) 20:32 < mmcgrath> 5 20:32 < G> I symlink the wrong update script and it goes haywire :) 20:32 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Meeting End

3 2

Puppet SELinux support and an introduction
by Sean E. Millichamp 09 Oct '08

09 Oct '08

Hi everyone, I have been working on getting some SELinux support into Puppet, based on some initial work by Frank Sweetser. I know the Fedora Infrastructure team uses Puppet and I heard a rumor that there was interest here in better SELinux support. ReductiveLabs just cut a 0.24.6rc1 containing this support. If you would like to test it I would certainly appreciate any feedback and bugreports. I believe they intend to release 0.24.6 as soon as possible, so the sooner the better. Now a brief introduction, since I know that is the thing to do on this mailing list: I ran my first Linux-in-production system in 1996, started the switch to Red Hat Linux 4.2 as soon as it was released, and have been using RHL, RHEL, CentOS, and Fedora since on both servers and on my desktop (as my exclusive OS since around 1999). These days I am a senior systems architect for a web-based firm that did around 1.2 billion page views in 2007 supported by around 300 Linux servers running our various dev, test, and prod environments. Some of the key technologies in use here are Apache, Perl/mod_perl, MySQL, MySQL cluster, RH Cluster Suite/GFS, Nagios, Cobbler/Kickstart for deployment, cfengine (and soon to be Puppet) for configuration management. My current main task is reimplementing our cfengine configurations in Puppet so I've been deep in Puppet recently. I have this idea that I would like to help on the Fedora Infrastructure team, to help contribute something back to Fedora, but between job and family feel that I probably don't have the time needed to dedicate to it, so I had just been lurking here until now. Cheers, Sean

3 2

Some architectural changes
by Mike McGrath 08 Oct '08

08 Oct '08

So we've finally completed some consolidation we've been working on for the past couple of months. We've also added a new class of server (bappX servers). What does this all mean? Well, app1+ are now all identical, for the most part. This allows us to think of our app servers as truly being part of a farm. The exception to this right now is those apps requiring our nfs mount in PHX. I'm still debating what to do about this but so far haven't really come up with a good answer. We'll be distributing load with haproxy. Just because an app is on a server doesn't mean that it will get hit. This allows us to distribute load intelligently and hopefully allow us to get local proxies preferring local app servers so geoIP dns can be deployed. The bapp servers (there's only 1 right now) will be our job control servers. Right now we kind of just have cron jobs spread all over the place and in some instances this has caused problems with production traffic. For example, while damned lies is checking out and updating all of its repos to get the latest translation stats for http://translate.fedoraproject.org/ app1 and app2 would typically see high load and if you happened to hit one of them while trying to get to pkgdb, you might experience slow response time. Now though we'll be running this job on bapp1 and pushing or pulling the relevant content via rsync. We use this model pretty successfully for other applications like mirrormanager and smolt, but now we have a dedicated server for it. Additionally we can add the bapp servers to the haproxy farms as a backup server, no traffic goes to it unless its needed. -Mike

4 4

New puppet trick - sysctl
by Mike McGrath 07 Oct '08

07 Oct '08

Just so you guys know we're using a sysctl plugin now so we can do things like sysctl { 'net.ipv4.ip_forward': value => '1' } I'm going to go through tomorrow and convert some of our existing configs. We don't use sysctl heavily but this will be handy in places like our database servers where each db server has different settings depending on the host. -Mike

1 0

netatalk for EPEL?
by Matt Domsch 05 Oct '08

05 Oct '08

Jiri, I see you're the owner of netatalk in Fedora. netatalk includes the 'timeout' command which I'd like to use with some scripts in Fedora Infrastructure, specifically some of MirrorManager's scripts. Can I talk you into building and maintaining netatalk in EPEL, on EL-5 branch at least? Thanks, Matt -- Matt Domsch Linux Technology Strategist, Dell Office of the CTO linux.dell.com & www.dell.com/linux

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure October 2008