============================================ #fedora-meeting: Infrastructure (2014-12-11) ============================================

Meeting started by nirik at 18:00:36 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2014-12-11/infrastructure.20... .

Meeting summary --------------- * aloha (nirik, 18:00:37)

* New folks introductions and Apprentice tasks (nirik, 18:02:05)

* FAD recap (nirik, 18:06:48) * got lots of details about mirrormanager 1 from Matt Domsch (nirik, 18:07:31) * got Mirrormanager 2 pretty much all implemented (nirik, 18:08:38) * got some test coverage added to Mirrormanager 2 (nirik, 18:08:50) * got Mirrormanager2 mostly added to ansible and deployed in staging (nirik, 18:09:09) * LINK: https://admin.stg.fedoraproject.org/mirrormanager2/ (pingou, 18:09:19) * targeting mid january for production deployment. Might do some kind of phased approach with mirrorlist vs the rest. (nirik, 18:12:01) * ansible migrations took up the rest of the FAD time. (nirik, 18:13:01) * puiterwijk got nameservers in ansible and all redeployed with it. (nirik, 18:13:34) * various other playbooks written, just need to finish things up. (nirik, 18:14:13) * down to 57 machines in puppet. (nirik, 18:14:21)

* Applications status / discussion (nirik, 18:15:38) * puiterwijk got single login in badges, and spec for single logout (more to come) (nirik, 18:17:00) * proxy playbooks are about 75% done. TODO list is in playbooks/groups/proxies-reverseproxy.yml (nirik, 18:20:05) * hopefully roll out a bodhi2 instance in stg asap and see what all we need for production (nirik, 18:27:35)

* Sysadmin status / discussion (nirik, 18:28:28) * monday outage - updates/reboots (nirik, 18:29:59) * tuesday outage - database server migrations (nirik, 18:30:08) * wed outage - more ansible/rhel7 migrations (nirik, 18:30:19)

* nagios recap (nirik, 18:35:31)

* Upcoming Tasks/Items (nirik, 18:37:28) * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/ (nirik, 18:37:28) * we will meet next week as normal, but not the following 2 weeks (nirik, 18:38:34) * ACTION: nirik to send list of ansible work we still need done to list. (nirik, 18:40:33)

* Open Floor (nirik, 18:40:39) * talk to puiterwijk if you want single login on your fedora app (nirik, 18:41:43)

Meeting ended at 18:43:04 UTC.

Action Items ------------ * nirik to send list of ansible work we still need done to list.

Action Items, by person ----------------------- * nirik * nirik to send list of ansible work we still need done to list. * **UNASSIGNED** * (none)

People Present (lines said) --------------------------- * nirik (102) * pingou (20) * puiterwijk (16) * lmacken (13) * wallison (11) * tflink (9) * threebean (8) * zodbot (5) * adimania (3) * mirek-hm (3) * Khaytsus (1) * bwood09 (1) * danielbruno (1) * relrod (1) * abadger1999 (0) * smooge (0) * mdomsch (0) * dgilmore (0

-- 18:00:36 <nirik> #startmeeting Infrastructure (2014-12-11) 18:00:37 <zodbot> Meeting started Thu Dec 11 18:00:36 2014 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:37 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:37 <nirik> #meetingname infrastructure 18:00:37 <zodbot> The meeting name has been set to 'infrastructure' 18:00:37 <nirik> #topic aloha 18:00:37 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 18:00:37 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 18:00:42 * lmacken 18:00:42 * threebean waves 18:00:45 * puiterwijk is here 18:00:48 * danielbruno is here 18:00:52 * mirek-hm is here 18:01:00 * adimania is here 18:01:03 * pingou 18:01:09 * bwood09 here 18:02:05 <nirik> #topic New folks introductions and Apprentice tasks 18:02:20 <nirik> any new folks like to provide a short introduction of themselves? 18:02:28 <nirik> or apprentices with questions or comments ? 18:02:48 <wallison> Guess I'll go... 18:03:23 <wallison> My name is Will, I'm still working through the Primary Steps of Getting Started 18:03:33 <nirik> cool. welcome wallison 18:03:44 <wallison> I'm not an IT pro, but have been using Linux at home for at least 15 years 18:03:53 <nirik> are you primarilly interested in sysadmin type things, or application development, or both? 18:04:00 <wallison> and want to contribute (finally) to the FOSS community 18:04:10 <wallison> Fedora seems to make that easy! 18:04:15 <threebean> :) 18:04:18 <nirik> we try. ;) 18:04:24 <wallison> nirik sysadmin stuff 18:04:40 <wallison> i'd like to do that as a second career after college 18:04:46 <nirik> cool. See me after the meeting over in #fedora-admin and I can get you added to our apprentice group... 18:04:49 <wallison> threebean: nice vids by the way 18:04:58 <nirik> and you can look around and look at easyfix tickets 18:05:18 <wallison> nirik: sounds awesome, I'll probably need some hand holding! 18:05:37 * relrod late, sorry 18:05:38 <nirik> sure, there are usually folks around in #fedora-admin or #fedora-noc that can help you out. 18:05:42 <wallison> Thanks for allowing my to introduce myself... 18:05:46 <threebean> wallison: thanks :) lemme know if you find holes in the information or important stuff left out. could maybe use an updated sequel.. 18:06:24 <nirik> any other new folks? or apprentices with questions? 18:06:24 <wallison> will do threebean 18:06:38 <pingou> welcome wallison 18:06:48 <nirik> #topic FAD recap 18:07:06 <nirik> So, we recently had a FAD (fedora activity days) in raleigh... 18:07:13 <pingou> it was nice :) 18:07:13 <nirik> I think we got some great stuff done, wanted to recap. ;) 18:07:31 <nirik> #info got lots of details about mirrormanager 1 from Matt Domsch 18:07:49 <nirik> did we actually get mm2 all implemented? 18:07:58 <nirik> or were there still some minor gaps? 18:08:22 <pingou> should be all there 18:08:23 <nirik> I think we got it all pretty much? 18:08:33 <pingou> including the bugs we missed :) 18:08:38 <nirik> #info got Mirrormanager 2 pretty much all implemented 18:08:50 <nirik> #info got some test coverage added to Mirrormanager 2 18:09:09 <nirik> #info got Mirrormanager2 mostly added to ansible and deployed in staging 18:09:19 <pingou> https://admin.stg.fedoraproject.org/mirrormanager2/ 18:09:26 <nirik> There is still some testing work and finishing things up in stg, but it's largely all there. ;) 18:09:42 <pingou> the mirrorlist needs a little tweaking :) 18:09:51 <nirik> did we decide on a timeline to deploy? 18:09:56 <Khaytsus> Sorry, my server is flipping out again, I"m reading back ;) 18:09:56 <pingou> we did not 18:09:56 <nirik> in prod I mean 18:10:28 <nirik> well, it would be good to get this rolled out before any freezes or milestones... 18:10:35 <nirik> perhaps mid january? 18:10:46 <pingou> once we get mirrorlist up and running we'll be able to test mm2 nicely 18:10:51 <pingou> but mid-january sounds good 18:11:15 <nirik> we could also do a phased approach... do mirrorlists first, then the rest... 18:11:18 <nirik> or vice versa. 18:11:23 <nirik> as long as the data is compatible 18:11:34 <pingou> should be :) 18:12:01 <nirik> #info targeting mid january for production deployment. Might do some kind of phased approach with mirrorlist vs the rest. 18:12:12 <nirik> anything else to note on MM2? 18:12:46 <pingou> looks good to me :) 18:12:48 <nirik> we also then spent serveral days working on ansible migrations. 18:13:01 <nirik> #info ansible migrations took up the rest of the FAD time. 18:13:18 <nirik> threebean: where were we on proxy playbooks? still needs more converting? 18:13:34 <nirik> #info puiterwijk got nameservers in ansible and all redeployed with it. 18:14:13 <nirik> #info various other playbooks written, just need to finish things up. 18:14:21 <nirik> #info down to 57 machines in puppet. 18:14:56 <nirik> anything else to note about the ansible side of the FAD? 18:15:38 <nirik> #topic Applications status / discussion 18:15:49 <puiterwijk> nirik: well, just one more thing about the FAD not about ansible 18:15:51 <nirik> anything in the applications side this week? I see there was a new pkgdb deployed 18:15:56 <nirik> puiterwijk: oh? 18:16:00 <puiterwijk> oh, can also go here 18:16:21 <puiterwijk> got single login into badges upstream, and a spec for single logout. more to come in the near future 18:16:33 <nirik> oh yeah. 18:16:48 <puiterwijk> (that was also during the FAD, though not officially part :) ) 18:17:00 <nirik> #info puiterwijk got single login in badges, and spec for single logout (more to come) 18:17:01 <lmacken> I worked on polishing up the atomic tree composer code and integrated it into bodhi during the first 1.5 days of the FAD 18:17:18 <nirik> lmacken: oh yeah, we were going to look at landing that sometime, right? 18:17:20 <lmacken> got bodhi mashes running in stg for the first time, and with atomic tree support too 18:17:37 <lmacken> nirik: yeah, I'd be happy to push that out today or tomorrow 18:17:47 <threebean> nirik: on proxy playbooks the redirects, websites, and reversepassproxies are done. 18:17:55 <nirik> lmacken: we might run it by dgilmore... but I think it's the way he wanted it. 18:17:56 <tflink> any farther thoughts on when bodhi2 might be deployed? 18:18:17 <lmacken> nirik: i'll have it compose trees to a one-off location and have him +1 the output before we throw it live 18:18:28 <threebean> prolly 75% done... there are some more little bits that need doing, with a commented-out TODO list in playbooks/groups/proxies-reverseproxy.yml 18:18:30 <lmacken> and walters too 18:18:31 <nirik> tflink: not sure, but we should try and come up with a timeline. 18:18:47 <tflink> not much concrete movement in qa apps, mostly discussion and planning. still working on an el7 build for blockerbugs so that can be migrated but keep hitting epel7 package problems 18:19:14 <nirik> lmacken: sounds good. 18:19:33 <nirik> tflink: let me know if I can help... or if you get to the end of the well and we can cut them over. ;) 18:19:38 <pingou> nirik: btw, rebuilt fas03 fine yesterday? 18:20:05 <nirik> #info proxy playbooks are about 75% done. TODO list is in playbooks/groups/proxies-reverseproxy.yml 18:20:08 <tflink> we're likely to try getting beaker working as more than just a dev machine next year, will be starting more conversations about how to deploy and secure that once it gets farther along 18:20:24 <nirik> pingou: I rebuilt fas03 and fas02. Both seem working fine. 18:20:29 <pingou> cool! 18:20:37 <tflink> nirik: yeah, I want to make sure Taskotron is ready for the switchover since we rely on bodhi for notifications right now 18:21:02 <tflink> nirik: it's mostly epel7 builds that were never tested ATM. things that build fine until you try to use them and they explode :-/ 18:21:02 <puiterwijk> nirik: well, I've seen some 2FA problems earlier today. but might've been just net timeouts or me overthinking things 18:21:20 <nirik> puiterwijk: hum, ok. It's working fine for me... 18:21:32 <puiterwijk> nirik: not all the time. next time I see it, I'll report back 18:21:37 <puiterwijk> and/or debug 18:22:00 <nirik> lmacken: thoughts on bodhi2 deployment? can we add a bodhi2 instance in stg now and see what else we need? 18:22:15 <lmacken> nirik: setting up a stg instance sounds good to me 18:22:33 <lmacken> nirik: the frontend should be close to ready, and having a stg setup that we can mash in now will help me polish up the final masher bits 18:22:49 <nirik> cool. I can make one this afternoon. ;) 18:22:51 <lmacken> (until we mash in koji) 18:23:00 <lmacken> nirik: awesome, thanks 18:23:32 <nirik> I'd be happy if we could roll it out in jan too... or sooner, but that may not be practical. 18:23:56 <tflink> nirik: you mean a production bodhi2? 18:24:06 <tflink> or the stg instance 18:24:14 <nirik> yeah. I want it out. It's been forever. 18:24:19 <nirik> stg I plan on doing today. ;) 18:24:29 <nirik> perhaps that will give us a feel for when we could go to prod 18:24:42 * tflink starts scrambling to get Taskotron ready for that 18:24:44 <lmacken> yeah, let's shoot for sometime in jan/feb :) 18:25:03 <nirik> tflink: is that impractical from your side? or how much code changes? 18:25:04 <lmacken> there's backwards compat work that still needs to be done in python-fedora, etc. 18:25:35 <nirik> oh true. 18:25:37 <tflink> nirik: the biggest problem is that we need to get fedmsg integration, i assume that API changes will have minimal impact from our end 18:26:01 <nirik> tflink: ok. having a stg instance should help there too... (so you have something to test/see) 18:26:12 <lmacken> threebean and I can help with the fedmsg stuff on all fronts 18:26:21 <nirik> the other big application deployment I want to see is mailman3 soonish. ;) 18:26:30 * pingou looks at abompard 18:26:32 <nirik> abompard: ^ any news on mailman3 18:27:35 <nirik> #info hopefully roll out a bodhi2 instance in stg asap and see what all we need for production 18:27:57 <nirik> cool. Any other applications side news? 18:28:28 <nirik> #topic Sysadmin status / discussion 18:28:33 <mirek-hm> openstack had just problems ergo we copr had problems today and there were lots of failing builds today. /me lost the patience and I'm scratching the idea of installin Juno as the installation still does not work and I want to install Icehouse, which works (err worked last time I tried that). 18:28:50 <nirik> mirek-hm: yeah, seems reasonable. 18:29:06 <nirik> mirek-hm: I asked Smooge to reinstall those. If he can't get to it soon I will. 18:29:50 <nirik> I'm also going to send out a outage announcement for next week... I'm going to plan an outage block monday/tuesday/wed. 18:29:59 <nirik> #info monday outage - updates/reboots 18:30:02 <mirek-hm> nirik: my plan is that once installed, I will run there few VM and will check how stable it is, and eventually start migrating copr 18:30:08 <nirik> #info tuesday outage - database server migrations 18:30:19 <nirik> #info wed outage - more ansible/rhel7 migrations 18:30:28 <nirik> mirek-hm: ok, sounds good. 18:30:43 <threebean> nirik: would you like to see the proxy stuff ready by then (next week)? 18:31:04 <puiterwijk> threebean: I think that might be a bit soon for something that complex 18:31:12 <nirik> threebean: I think it's too soon to roll that to prod... also, we can do that outside outage 18:31:21 * threebean nods 18:31:28 <nirik> ie, take a proxy out, convert it, add it back carefully and watch 18:31:28 <threebean> will get it ready and then let it sit, then. 18:31:48 <nirik> if it's ready looking I can try some conversions over the holidays... ;) 18:31:52 <pingou> nirik: on the other side threebean tested with with ansible --check to see if it changed anything 18:32:17 <nirik> yeah... 18:32:26 <pingou> the only thing I would have is: next week might be a little soon with the holidays around the corner :) 18:32:34 <pingou> but early jan sounds good to me 18:32:51 <nirik> I think the next step is to finish things up and put it in as proxy01.stg... 18:33:07 <nirik> then we can test it more... then if it looks all good there, we can start rolling out to prod proxies. 18:33:26 <nirik> also, at the same time we can try and update colo sites to rhel7 virthosts. 18:33:28 <puiterwijk> nirik: or we can push it to proxy01 and have everyone use that again. might provide good testing :) 18:33:43 <nirik> heh. indeed. 18:33:46 <pingou> ^^ 18:35:22 <nirik> Anything else on the sysadmin side? 18:35:31 <nirik> #topic nagios recap 18:35:37 <puiterwijk> .tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&dis... 18:35:37 <zodbot> puiterwijk: http://tinyurl.com/pzzwx2r 18:35:57 <puiterwijk> oh, hah. copied the wrong one I guess 18:35:59 <nirik> we had quite the pile of alerts this week... due to the proxy geoip issue puiterwijk tracked down and fixed (thanks again!) 18:36:40 <puiterwijk> we had lots of traffic as well (thanks to all contributors for Fedora 21!) 18:36:44 <nirik> hopefully things will relax a bit now. Anything to reduce alerts over the holidays would be good. 18:37:28 <nirik> #topic Upcoming Tasks/Items 18:37:28 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/ 18:37:37 <nirik> anything upcoming that folks would like to note or schedule? 18:37:56 <nirik> aside from mirrormanager2, mailman3 and bodhi2 deployed by jan. ;) 18:38:04 <pingou> \ó/ 18:38:11 <adimania> note: do we have an updated list of what is left for ansible? 18:38:20 <puiterwijk> nirik: RH shutdown end of december? 18:38:34 <nirik> #info we will meet next week as normal, but not the following 2 weeks 18:38:58 <nirik> adimania: I have a list. I am going to probibly post to the list... if you want to look at working on some more. ;) 18:39:19 <nirik> a lot of the ones left in puppet already have playbooks, just a matter of downtime and moving them. 18:39:25 <nirik> thus the outage blocks next week 18:39:35 <adimania> nirik, yes. I am willing to. If you can post the list today or tomorrow, then I can try to do something over the weekend :) 18:39:43 <nirik> cool. ;) 18:40:33 <nirik> #action nirik to send list of ansible work we still need done to list. 18:40:39 <nirik> #topic Open Floor 18:40:42 <nirik> anything for open floor? 18:41:26 <puiterwijk> if people want to add Single login to Fedora apps, please get in contact with me? 18:41:43 <puiterwijk> help add* 18:41:43 <nirik> #info talk to puiterwijk if you want single login on your fedora app 18:43:01 <nirik> ok, thanks for coming everyone. :) Lets continue in #fedora-admin, #fedora-noc and #fedora-apps. 18:43:04 <nirik> #endmeeting