============================================ #fedora-meeting: Infrastructure (2012-04-12) ============================================

Meeting started by nirik at 20:00:01 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2012-04-12/infrastructure.20... .

Meeting summary --------------- * Robot Roll Call (nirik, 20:00:01)

* New folks introductions and Apprentice tasks. (nirik, 20:02:52)

* two factor auth status (nirik, 20:04:27)

* Staging re-work status (nirik, 20:08:21)

* Applications status / discussion (nirik, 20:08:53) * LINK: http://gitorious.org/sticky-notes/sticky-notes/ (nirik, 20:10:47) * LINK: http://paste.kde.org/ (nirik, 20:11:07)

* Upcoming Tasks/Items (nirik, 20:27:37)

* Open Floor (nirik, 20:32:44)

Meeting ended at 20:37:47 UTC.

Action Items ------------

Action Items, by person ----------------------- * **UNASSIGNED** * (none)

People Present (lines said) --------------------------- * nirik (91) * mdomsch (29) * herlo (15) * smooge (9) * zodbot (8) * skvidal (7) * CodeBlock (5) * nb (4) * abadger1999 (4) * athmane (3) * wolfkit (3) * lmacken (2) * ianweller (1) * threebean (1) * adrianhannah (1) * kwame (1) * ricky (0) * dgilmore (0) * Codeblock (0) -- 20:00:01 <nirik> #startmeeting Infrastructure (2012-04-12) 20:00:01 <zodbot> Meeting started Thu Apr 12 20:00:01 2012 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 20:00:01 <nirik> #meetingname infrastructure 20:00:01 <zodbot> The meeting name has been set to 'infrastructure' 20:00:01 <nirik> #topic Robot Roll Call 20:00:01 <nirik> #chair smooge skvidal Codeblock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean 20:00:01 <zodbot> Current chairs: Codeblock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean 20:00:07 * skvidal is here 20:00:25 * adrianhannah here 20:00:28 * lmacken 20:00:30 * abadger1999 here 20:00:31 * athmane is around 20:00:33 * mdomsch 20:00:36 * kwame is here o/ 20:00:52 * CodeBlock here 20:01:09 * wolfkit is here 20:01:17 * nirik waits another minute for folks to wander in 20:02:12 * herlo is here 20:02:48 <nirik> ok, lets go ahead and dive in... 20:02:52 <nirik> #topic New folks introductions and Apprentice tasks. 20:02:53 <nirik> 20:02:53 <nirik> If any new folks want to give a quick one line bio or any apprentices 20:02:53 <nirik> would like to ask general questions, they can do so now. anyone? 20:03:33 <smooge> is here. 20:03:59 <nirik> I'll note there's at least one apprentice ticket waiting for me to comment. I'm going to try and get to it in the next few days... 20:04:25 <nirik> ok, moving along then... 20:04:27 <nirik> #topic two factor auth status 20:04:31 <nirik> any news here? 20:05:09 <skvidal> not much change 20:05:18 <skvidal> mricon is working on radius support for somethign for them for cisco vpn 20:05:24 <wolfkit> I've been quite busy and the latest rawhide grub2-efi / kernel updates have been playing up on my laptop so haven't been able to hack on it 20:05:33 <nirik> ok. 20:05:43 <nirik> anything we can do to help move it along? 20:06:09 <skvidal> I just need some time to sit down and focus on it, I suspect 20:06:21 <herlo> where are we stuck? 20:06:24 <nirik> ok. I'd be happy to help too.. .would love to get it going. 20:06:26 <skvidal> we're not stuck 20:06:32 * herlo hasn't been able to help in a while, wants to help again 20:06:37 <skvidal> there aren't any roadblocks afaik 20:06:42 <skvidal> it's more just integrate + test 20:06:42 <wolfkit> as do I, time seems to be my most shrinking resource, heh 20:06:42 <herlo> k 20:06:52 <nirik> wolfkit: always the way I fear. ;) 20:07:13 <nirik> ok, lets try and move it forward in the next week, revisit next meeting. 20:08:21 <nirik> #topic Staging re-work status 20:08:28 <nirik> still waiting on freeze on this one... 20:08:41 <nirik> but the happy news is that we have a beta and freeze is over next week. :) 20:08:53 <nirik> #topic Applications status / discussion 20:09:05 <nirik> any application news? 20:09:21 <CodeBlock> dpsearch is evil. That is all ;) -- I have emailed daMaestro asking for help, waiting for response. 20:09:31 <nirik> CodeBlock: ok, thanks. 20:09:53 <nirik> Oh, I have a bit of news on paste... I'm looking at sticky-notes as a possibly fpaste.org replacement server. 20:10:09 <nirik> it might work out and let us take over fpaste.org... 20:10:20 <ianweller> nirik: sticky-notes? 20:10:30 <nirik> It's php, but it has an upstream and they sound like they might be responsive. 20:10:47 <nirik> http://gitorious.org/sticky-notes/sticky-notes/ 20:11:06 <nirik> kde uses it. 20:11:07 <nirik> http://paste.kde.org/ 20:11:31 <CodeBlock> nirik: is there a CLI app for it? I could look at hacking one up if needed 20:11:32 <herlo> it looks pretty good 20:11:41 <nirik> pastebinit can talk to it apparently. 20:11:44 <nirik> (already in fedora) 20:11:56 <herlo> CodeBlock: the plan there was to maybe modify fpaste to take a configuration file 20:11:56 <smooge> cool 20:12:00 <herlo> nirik: good news! 20:12:13 * herlo thinks that's better anyway 20:12:19 <nirik> so, possibly we could work something with fpaste calling that or have pastebinit ship a fpaste wrapper or something. 20:12:26 <herlo> yeah 20:12:29 <herlo> easy to do 20:12:46 <nirik> pastebinit already defaults to fpaste.org in fedora too. 20:13:03 <herlo> we'll have to change something 20:13:14 <nirik> yeah. 20:13:19 <herlo> but we could address that later 20:13:22 <nirik> can burn that bridge when we come to it... 20:13:40 <nirik> abadger1999 / threebean / pingou / lmacken: any other app news from this week/upcoming? 20:13:48 <abadger1999> nothing upcoming 20:14:04 <threebean> still working on packaging for the message bus 20:14:23 * mdomsch has been hacking MM late at night 20:14:40 <nirik> mdomsch: oh, did you see we had another issue with mirrorlist_server last night... 20:14:42 <mdomsch> v1.4 in progress again, mostly works, needs more testing 20:14:46 <mdomsch> nirik: no... 20:14:47 * athmane reported two vuls to sticky-notes (see merge request), will check the admin console this weekend 20:15:07 <mdomsch> nirik: what now? 20:15:17 <nirik> mdomsch: started taking up memory and causing problems, but then we stopped it all and I tried to get some debugging and started it and it all started working again. 20:15:33 <mdomsch> ah - heisenbug 20:15:41 <nirik> I have no idea if it was a DOS or what was happening. ;( Have any debugging tips or info we can gather if it happens again? 20:15:58 <mdomsch> get a copy of /var/lib/mirrormanager/ 20:16:00 <nirik> athmane: yeah, dcr226 also mailed the author a pointer to your patches. ;) 20:16:12 <nirik> mdomsch: I have one from the last issue on april 2nd... 20:16:13 <mdomsch> so we can see if there's corruption of the netblock lists or the pickle 20:16:16 <lmacken> nirik: nothing upcoming, aside from finishing up the apps.fp.o/{packages,tagger} deployment, which I haven't had any cycles for lately 20:16:32 <nirik> mdomsch: bastion:~kevin/mirorrmanager.tar.xz 20:16:39 <athmane> nirik: feel free to ping me, if you need help with the deployment 20:16:52 <nirik> rorry, mirrormanager-2012-04-02.tar.xz is the file 20:17:02 <nirik> athmane: excellent. will do. 20:17:02 <mdomsch> I did spend considerable time teaching MM how to do database schema updates for various oddities not covered by SQLObject itself 20:17:02 <nb> nirik, so are we not planning on sticking with th current fpaste-server? 20:17:15 <mdomsch> particularly adding indexes 20:17:30 <herlo> nb: it's still a possibility, but there's a ton of work that needs to be put into fpaste-server before it's ready 20:17:45 <nirik> nb: we tried, but herlo and daMaestro both thought the hacks were too gross for a clean deployment, and they also were not really wanting to be an upstream for it... 20:18:02 <nb> oh ok 20:18:03 <herlo> I wasn't aware of the amount of work it would take and it's going to take some serious amount of time to hack upon it 20:18:11 <herlo> to get it into shape 20:18:38 <nirik> mdomsch: what does mirrorlist_server do? which part? anyhow, if you could look at that dump and/or suggest things for us to collect I can do that if it happens again. 20:19:06 <mdomsch> nirik: that's the URL endpoint that yum hits 20:19:20 <nirik> ok, so it could have been some kind of DOS... ? 20:19:22 <mdomsch> and that download.fp.o URLs hit to generate the redirect 20:19:26 <mdomsch> yes, could be a DOS 20:19:33 <mdomsch> though that'd be surprising 20:19:37 * nirik can try checking logs for unusual hits. 20:19:44 <nirik> yeah, it was odd. 20:19:55 <nirik> it happened on app01 first... caused it to crash. 20:20:03 <nirik> then a few minutes later the other apps started doing it too. 20:20:18 <mdomsch> normally when it does this, it means the communication path between apache wsgi app and mirrorlist_server.py app got interrupted 20:20:32 <mdomsch> and both sides go a little nuts waiting on the other 20:20:50 <mdomsch> but yes, pls check the proxy logs for spikes 20:20:57 <nirik> will do. 20:21:06 <mdomsch> I suspect it's a bug in mirrorlist_server.py somehow 20:21:24 <nirik> yeah, I tried to change 'log_file' and 'debug' but it didn't seem to do anything. ;( 20:21:24 <abadger1999> nirik: oh I took a look at the ipnetblocks file -- it's not corrupt although it is very big. 20:21:27 <mdomsch> that doesn't log output much if at all - and it's run by supervisor 20:21:34 <nirik> yeah 20:21:42 <mdomsch> yeah, these netblock lists look sane at a glance 20:21:53 <abadger1999> Loads fine using the same code as is in mirrormanager 20:22:06 <nirik> so there goes that theory. ;( 20:22:29 <mdomsch> the pickle has a much newer timestamp 20:22:37 <mdomsch> than the netblock lists 20:23:12 <mdomsch> anyhow - not obvious... 20:23:30 <mdomsch> page me if it happens again 20:23:38 <mdomsch> ideally while it's happening 20:23:54 <mdomsch> and look at ps auxgw | grep mirror 20:24:20 <mdomsch> to see what's there. normally there are a few mirrorlist_server.py processes there, 1 master plus one for each request in process 20:24:34 <nirik> mdomsch: ok. we do have OOM dumps in logs from the kernel if it helps any? 20:24:50 <mdomsch> if there are more than say 20, and if any are in S state, that's a problem 20:25:17 <mdomsch> it might, yes 20:26:19 <nirik> ok. can get you that. 20:26:33 <nirik> ok, any more app news? or shall we move on? 20:27:37 <nirik> #topic Upcoming Tasks/Items 20:28:34 <nirik> 2012-03-20 to 2012-04-17 - F17 Beta Freeze 20:28:34 <nirik> 2012-04-17 - F17Beta release day 20:28:34 <nirik> 2012-05-08 to 2012-05-22 - F17 Final Freeze. 20:28:34 <nirik> 2012-05-01 - nag fi-apprentices. 20:28:34 <nirik> 2011-05-03 - gitweb-cache removal day. 20:28:35 <nirik> 2012-05-09 - Check if puppet works on f17 yet. 20:28:37 <nirik> 2012-05-10 - drop inactive fi-apprentices 20:28:39 <nirik> 2012-05-22 - F17 release 20:28:44 <nirik> so beta comes out tuesday.. 20:29:03 <nirik> we have our usual release tickets we will need to act on once the trees are staged. 20:29:25 <nirik> .ticket 3218 20:29:27 <zodbot> nirik: #3218 (Fedora17 Beta - Verify mirror space) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3218 20:29:29 <nirik> .ticket 3219 20:29:30 <CodeBlock> nirik: safe tavels back in time to 2011 to remove gitweb-cache ;) 20:29:31 <zodbot> nirik: #3219 (Fedora17 Beta - new website) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3219 20:29:33 <nirik> .ticket 3220 20:29:35 <zodbot> nirik: #3220 (Fedora17 Beta - release day ticket) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3220 20:29:41 <nirik> .ticket 3221 20:29:43 <zodbot> nirik: #3221 (Fedora17 Beta - verify permissions on content) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3221 20:29:49 <nirik> CodeBlock: ha. Paying attention I see... 20:29:51 * nirik fixes 20:30:31 <nirik> anything else upcoming anyone would like to note or schedule? 20:31:04 <nirik> I've added a wiki page with our private cloud plans: https://fedoraproject.org/wiki/Infrastructure_private_cloud 20:31:24 <nirik> we don't have concrete dates on it yet, but I can add them in when we have finalized hardware and such 20:32:23 <nirik> use ideas, suggestions or comments welcome on the list 20:32:44 <nirik> #topic Open Floor 20:32:50 <nirik> anyone have items for open floor? 20:33:27 <nb> not i 20:33:40 <smooge> yay release 20:33:48 <nirik> nb: oh, any news on the keyserver thing? or haven't had a chance to try yet? 20:33:49 <smooge> not i 20:33:51 <smooge> oh wait 20:34:04 <CodeBlock> my online presence will be iffy tomorrow and the weekend of April 28th. Not schedule worthy, but just noting. 20:34:12 <nb> nirik, nope, not had time yet, just got pw yesterday 20:34:25 <nirik> nb: no worries. Will be interesting to see if it works 20:34:32 <smooge> I am currently working on removing ALL mailing list aliases from redhat.com -> lists.fedoraproject.org 20:34:40 <nirik> smooge: hurray! 20:34:52 <smooge> I will then work on moving epel and some other lists over to lists.fedoraproject.org 20:35:00 <smooge> which is always fun 20:35:30 <nirik> oh, I had another thing to note: We were going to migrate hosted monday, but postponed it. We want to shake out a timeout/failover issue with the new glusterfs setup before making it live. 20:35:38 <nirik> Hopefully we can migrate that next week or so. 20:37:01 <nirik> ok, if nothing else we can call it a meeting and get back to work. ;) 20:37:17 <smooge> call it 20:37:44 <nirik> thanks for coming everyone! 20:37:47 <nirik> #endmeeting