============================================ #fedora-meeting: Infrastructure (2013-06-20) ============================================

Meeting started by nirik at 19:00:01 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2013-06-20/infrastructure.20... .

Meeting summary --------------- * welcome y'all (nirik, 19:00:02)

* New folks introductions and Apprentice tasks (nirik, 19:02:15)

* Applications status / discussion (nirik, 19:03:43) * mirrormanager 1.4 is humming along in production now. :) (nirik, 19:04:04) * badges work is moving rapidly ahead (nirik, 19:04:58) * badges has a number of open issues, see oddshocks or threebean if you want to help out implementing. (nirik, 19:10:10) * LINK: http://ambre.pingoured.fr/thisweekinfedora/ (pingou, 19:10:37) * LINK: http://ambre.pingoured.fr/thisweekinfedora/ <- results of tuesday work (pingou, 19:13:12) * http://ambre.pingoured.fr/thisweekinfedora/ hopefully themed and added to planet soon (nirik, 19:15:45)

* Sysadmin status / discussion (nirik, 19:21:31) * ansible migration continues new machines are in ansible and slowly we are porting things over to it. (nirik, 19:25:21) * LINK: http://paste.fedoraproject.org/19948/71747790/ <-- that's the basics (skvidal, 19:28:35) * working out how to let folks run specific playbook runs themselves. (nirik, 19:38:28) * working out how triggering playbook runs should work. (nirik, 19:38:46) * ansible help welcome. See skvidal. :) (nirik, 19:43:39) * new bladecenter is all arrived, trying to get network to it. (nirik, 19:44:42) * will be adding mem and replacing 2 bvirthosts sometime this quarter (nirik, 19:45:00) * phx2 on-site visit time probibly in mid/late july. (nirik, 19:45:16) * working on new rdiff-backup setup for backups, probibly after freeze (nirik, 19:46:04)

* Private Cloud status update / discussion (nirik, 19:46:50) * moving networking on a spare node so we can do a multihost grizzly setup (nirik, 19:47:49) * f19 test compose images available for testing/tweaking. (nirik, 19:48:03)

* Upcoming Tasks/Items (nirik, 19:48:21) * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/ (nirik, 19:48:21)

* Open Floor (nirik, 19:49:34)

Meeting ended at 19:52:44 UTC.

Action Items ------------

Action Items, by person ----------------------- * **UNASSIGNED** * (none)

People Present (lines said) --------------------------- * nirik (112) * skvidal (112) * pingou (16) * threebean (10) * abadger1999 (7) * smooge (6) * zodbot (4) * relrod (4) * oddshocks (3) * mdomsch (3) * kc4zvw_ (1) * lmacken (1) * LoKoMurdoK (1) * ricky (0) * dgilmore (0) * CodeBlock (0) -- 19:00:01 <nirik> #startmeeting Infrastructure (2013-06-20) 19:00:01 <zodbot> Meeting started Thu Jun 20 19:00:01 2013 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:01 <nirik> #meetingname infrastructure 19:00:01 <zodbot> The meeting name has been set to 'infrastructure' 19:00:02 <nirik> #topic welcome y'all 19:00:02 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean 19:00:02 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean 19:00:24 <nirik> who all is around for a nice meeting? 19:00:25 * relrod here 19:00:26 * threebean is here 19:00:31 * LoKoMurdoK here 19:00:35 <oddshocks> Holla! 19:00:50 * skvidal is here 19:01:07 * lmacken 19:01:31 <smooge> ere 19:02:12 <nirik> ok, lets go ahead and dive in... 19:02:15 <nirik> #topic New folks introductions and Apprentice tasks 19:02:25 * mdomsch 19:02:25 <nirik> any new folks want to say hi, or apprentices with questions or comments? 19:02:36 * abadger1999 here 19:03:24 * nirik waits a minute. 19:03:43 <nirik> #topic Applications status / discussion 19:03:49 <nirik> ok, any applications news? 19:04:00 <mdomsch> MM upgraded! 19:04:04 <nirik> #info mirrormanager 1.4 is humming along in production now. :) 19:04:10 <nirik> thanks for working so much on that mdomsch 19:04:17 <oddshocks> Tahrir, badges, etc all humming along. Threebean and I are slaying some code 19:04:20 <threebean> mdomsch: nice blog post on it too :) 19:04:43 <threebean> yeah, and the badges repo can be found here if anyone wants to poke through it http://infrastructure.fedoraproject.org/infra/badges/rules/ 19:04:44 <mdomsch> glad to have finally had the time. My apologies for the crappy overnight pages and debugging sessions all weekend long! 19:04:44 <nirik> oddshocks: excellent. :) 19:04:58 <nirik> #info badges work is moving rapidly ahead 19:05:08 * pingou late 19:05:18 <nirik> oddshocks / threebean: any milestones coming up or things others could help with? 19:05:26 <pingou> oh I got something app related (after badges) 19:05:27 <nirik> mdomsch: no worries. It was worth it to get it all sorted out. :) 19:05:36 <threebean> oh, a question on badge-land. in the next few weeks, can we set up the production badge machines even while we're still in freeze? 19:06:04 <smooge> mdomsch, we need to do that every now and then to remind of us why we don't like to have them 19:06:14 <nirik> we could, just would need a freeze break explaining exactly what is being done. 19:06:30 <skvidal> what is the badges production setup look like? 19:06:35 <threebean> cool. we're not there yet, but when we do get there I'll write it up 19:06:44 <skvidal> are we doing a separate db server for it or relying on db01,etc? 19:07:05 <threebean> skvidal: two "webapp" nodes and one "backend" node. db usage should be pretty light.. I was anticipating adding it to db01. 19:07:23 * nirik really hopes to work on our db 'story' at flock some 19:07:38 <skvidal> nirik: +1 19:07:48 <threebean> yeah, that story right now is "how many eggs do you think we can fit in this basket?" 19:07:54 <nirik> yeah. :( 19:08:14 * relrod has an idea for a web app that I ran by threebean and puiterwijk a while back, but haven't asked anyone else yet because I haven't had time to work on it - but could maybe get some thoughts: 19:08:16 <nirik> we have some mitigation of a db server dying, but it could be/should be much better. 19:08:55 * nirik waits for relrod. 19:09:01 <oddshocks> As far as badges milestones, we're going to be seeing major front-end changes now that we've got the backend all groovy, and there are plenty of issues open peeps can help with on the github repo, just ping me or threebean about em if ya want :) 19:09:26 <relrod> Basically it would be neat/handy for some of our apps (e.g. Fedora Packages and Fedora Mobile) if there was a way for package maintainers to upload screenshots of their packages, and we could show them on package info pages. This could also play into if we ever move to an app-store style system in Fedora, it'd be nice if you could see screenshots before you install an app. 19:09:51 <pingou> relrod: you should talk with hugshie about that 19:10:10 <nirik> #info badges has a number of open issues, see oddshocks or threebean if you want to help out implementing. 19:10:29 <nirik> yeah, we probibly need to discuss app store stuff at flock? 19:10:37 <pingou> http://ambre.pingoured.fr/thisweekinfedora/ 19:10:46 <nirik> I could see screenshots and/or screencasts being nice to have for that 19:10:47 <relrod> pingou: will do. It's pretty low priority on my list, but just a thought I had. I don't have time to do much with it yet, but if it's something that not everybody hates then I'll make a repo and at least hack on it once in a while ;) 19:11:38 <threebean> relrod: same story with me and fedmsg-notifications :) 19:11:54 <nirik> The app store story is still unclear to me. I know hugshie was working on it again, but we need to determine what of that falls on us to implment and if it makes sense to do things that way. 19:12:04 * threebean nods 19:12:24 <pingou> nirik: well afaiu fedora-tagger lift already a good part of the burden 19:12:27 * skvidal recuses himself from this discussion 19:12:32 <pingou> only icons and screenshots are missing 19:12:43 <nirik> pingou: ok. 19:12:49 <threebean> he unfortunately never responded to my pings after we launched the rewrite of fedora-tagger. that was supposed to enable app store stuff integration but its not being taken advantage of as far as I can tell. 19:12:50 <nirik> any other application type news? 19:13:12 <pingou> http://ambre.pingoured.fr/thisweekinfedora/ <- results of tuesday work 19:13:43 <nirik> pingou: very nice! 19:13:48 <pingou> tatica said she was interested in theming it and I hope to plug it onto the planet one of these days 19:14:20 <nirik> sounds great. ;) 19:14:28 <nirik> thats all fedmsg based right? 19:14:43 <pingou> datagrepper actually 19:15:30 <nirik> yeah, ok. 19:15:45 <nirik> #info http://ambre.pingoured.fr/thisweekinfedora/ hopefully themed and added to planet soon 19:16:09 <nirik> any other app news? or shall we move on? 19:16:53 <pingou> I have to put some pressure on abadger1999 to release python-fedora with the flask openid plugin by the end of freeze :) 19:17:02 <abadger1999> <nod> 19:17:05 <abadger1999> It will be done :) 19:17:17 <abadger1999> and hopefully not buggy as hell ;-) 19:17:18 <nirik> is that needed for fedocal openid? 19:17:19 <nirik> or ? 19:17:35 <abadger1999> yeah -- fedocal openid should wor with the beta package + hotfix in infra 19:17:38 <pingou> nirik: yes, that's the last change I have on the todo before releasing 0.2.0 19:17:51 <abadger1999> but it won't work with just hte packages in the fedora/epel repos 19:18:12 <nirik> ok. 19:18:16 <skvidal> question on flask and openid in general 19:18:32 <skvidal> is there a status on the groups/teams work 19:18:46 <pingou> the group extension? 19:18:52 <skvidal> so we can do group-limited openid logins to websites via flask or mod_auth_openid 19:18:53 <skvidal> pingou: yes 19:19:10 <nirik> yeah, I know puiterwijk was waiting for some package for that, but I don't know current status 19:19:11 <skvidal> so I can say 'allow openid from fedora who are in this group' 19:19:21 <nirik> we also want it for the trac openid on hosted. 19:19:22 <skvidal> thx - in the long term I think it will really help us 19:19:27 <skvidal> nirik: yes we do 19:19:40 <pingou> skvidal: afaik for the moment the app is responsible for that 19:19:42 <nirik> I'd love to kick mod_auth_pgsql to the curb. ;) 19:19:54 <skvidal> pingou: right - but I need some way of requesting which groups a user is in 19:20:00 <skvidal> nirik: it's just nagios, right? 19:20:00 <nirik> python-openid-teams I think is the package? 19:20:16 <nirik> nagios and logs I think? or were we going to leave logs alone? 19:20:17 <pingou> nirik: I think so yes 19:20:22 <skvidal> nirik: I'd like to move logs 19:20:28 <skvidal> but logs is not mod_auth_pgsl 19:20:30 <skvidal> afaik 19:20:33 <skvidal> it's that local htpasswd thing 19:20:37 <nirik> it would mean that if openid is down we couldn't get to logs, but meh 19:20:41 <nirik> yeah 19:20:43 <skvidal> nirik: yah we could 19:20:51 <skvidal> nirik: we could either do a fallback or just ssh into the frelling box 19:21:00 <nirik> well, we couldn't get to html versions of the epylogs via a remote browser. ;) 19:21:07 <nirik> so, no big deal. 19:21:14 <nirik> since we would be likely fixing openid anyhow then. ;) 19:21:17 <skvidal> nirik: scp log02:/path/to/file.html . ; firefox file.html :) 19:21:25 <nirik> yep. ;) 19:21:27 <nirik> anyhow... 19:21:28 <skvidal> I'm okay with that 19:21:31 <nirik> #topic Sysadmin status / discussion 19:21:38 <nirik> lets see... sysadmin items... 19:21:43 <abadger1999> skvidal: that's in the extension. 19:21:47 <skvidal> abadger1999: excellent 19:21:51 <skvidal> I went to ansiblefest last week 19:21:54 <nirik> we have a number of things that are going to be pending after freeze. 19:22:03 <skvidal> and mainly heard about how other people are doing things in their infrastructures 19:22:07 <nirik> skvidal: you should do a trip report thingie/blog post/etc. ;) 19:22:10 <skvidal> ah 19:22:16 * skvidal goes back to lurking 19:22:17 <skvidal> :) 19:22:42 <nirik> so it sounded like we aren't doing anything _too_ crazy from others, right? 19:22:51 <skvidal> nirik: not too bad 19:23:03 <skvidal> I will note that a lot of folks are using it to merge between sets of tools 19:23:13 <skvidal> lots of 'use ansible to orchestrate other systems' 19:23:21 <skvidal> 'use ansible to manage merging multiple inventory sources' 19:23:33 <skvidal> 'use ansible to make our devs shut up and break their own systems' 19:23:42 <skvidal> and then by far 19:23:47 <skvidal> some really great new modules 19:23:50 <skvidal> that folks have written 19:23:58 <skvidal> mysql_replication was by far the most exciting imo 19:24:09 <skvidal> their able to flap their master back and forth 19:24:14 <skvidal> orchestrated in a playbook 19:24:21 <nirik> very nice. 19:24:24 <skvidal> and it is kinda seriously neat 19:24:29 * nirik wishes for a similar postgresql one. ;) 19:24:29 <skvidal> yah - I'd like to be there 19:24:34 <skvidal> on both of our db servers 19:24:48 <skvidal> okay I'll be quiet now :) 19:25:06 <pingou> and start write the email ;) 19:25:21 <nirik> #info ansible migration continues new machines are in ansible and slowly we are porting things over to it. 19:25:27 <skvidal> one thing I'll bring up 19:25:34 <nirik> skvidal: so what are our next steps on ansible migration? 19:25:47 <skvidal> so 19:25:54 <skvidal> in order to allocate access to folks 19:26:02 <skvidal> 2 or 3 folks at the conference who spoke are using jenkins 19:26:10 <skvidal> to allocate access to non-root users to run ansible 19:26:19 <skvidal> so they setup jenkins to access the right keys 19:26:28 <skvidal> and then give the users in jenkins access to run certain scripts 19:26:41 <skvidal> then the user sees the whole output and jenkins logs it, etc 19:26:48 <skvidal> so if they break it - they can fix it, too :) 19:27:03 <pingou> neat 19:27:09 <skvidal> I have to admit I like that - but running jenkins as part of the admin process feels...... weird to me 19:27:15 <nirik> it's a nice idea (althought I think jenkins is not a solution for us, but perhaps one of the other things like it would be) 19:27:26 <skvidal> right 19:27:31 <skvidal> so my wonder is this 19:27:48 <skvidal> if we did RBAC via a script + your user/groups + sudo on lockbox 19:27:56 <skvidal> that let you see the output AND it captured the output 19:28:00 <skvidal> 1. is that ridiculous? 19:28:13 <skvidal> 2. is there something stupid about this I'm not thinking about? 19:28:35 <skvidal> http://paste.fedoraproject.org/19948/71747790/ <-- that's the basics 19:29:03 <nirik> I think that would work... but in addition I think we still would need a trigger based thing... so even if you couldn't run the playbook then, you could commit and let it trigger off. 19:29:10 <skvidal> nirik: triggers STILL have to happen 19:29:11 <skvidal> yes 19:29:14 <skvidal> nirik: no dispute 19:29:23 <nirik> yep. In agreement here then. :) 19:29:35 <skvidal> about triggers then 19:29:39 <skvidal> I was looking at our commits 19:30:05 <skvidal> and so man y of them would either actually be 'apply-global' or they are the kind that make it almost impossible to know what systems would definitely be impacted - which means 'apply-global' 19:30:09 <skvidal> so in the apply-global case 19:30:27 <skvidal> how often do we check for that 19:30:40 <skvidal> since clearly it cannot be 'apply-global immediately following any check in that could impact it' 19:30:48 <skvidal> that would just be a recipe for people making bad commit decisions 19:30:55 <nirik> so, that would run all playbooks on all affected hosts in those playbooks? 19:31:08 <nirik> yeah 19:31:24 <skvidal> apply-global means 'we dcannot tell which hosts are affected so we have to assume all' 19:31:54 <nirik> yeah. 19:32:03 <skvidal> I guess I was thinking 19:32:14 * nirik needs to ponder on this a bit more I think... I think we can come up with something... 19:32:16 <skvidal> in the abscence of any commits we should not run 19:32:29 <skvidal> that seems fairly obvious, right? 19:32:33 <nirik> yeah, that sounds reasonable. 19:32:42 <nirik> or if we did, it would be a nightly job or something. 19:32:56 <nirik> and it should SCREAM that something changed. 19:33:18 <skvidal> nirik: sfromm's ansible-report can produce email reports - which are kinda nice 19:33:46 <skvidal> I talked to him a bit last week and earlier this week - there's some.... tricks with it - but I think they can be worked around 19:33:52 <skvidal> okay so 19:33:57 <skvidal> let's say that a commit comes in 19:34:00 <nirik> we could put some of the burden on the committer... to indicate what host(s) or groups to run on... but thats not ideal I guess. 19:34:09 <skvidal> we look up which hosts it applies to 19:34:15 <skvidal> we write that out to a file 19:34:22 <skvidal> which a cron job checks for.... how often? 19:34:33 <nirik> 30min? 19:34:33 <skvidal> every hour? every half-hour? every 15 minutes? 19:34:54 <skvidal> so every 30 m the job will open up the file, look for which playbooks/hosts to run against and run those 19:34:58 <skvidal> if it exists 19:35:27 * nirik notes we can adjust down the road if it seems like the time isn't right... 19:35:40 <nirik> IMHO if it's urgent you want to run the playbook yourself or find someone to. 19:36:06 <nirik> if it's just a cleanup/whatever you want it to run and do those things before you forget about it, but with enough time to realize if you made a mistake/typo (or for other people to) 19:36:23 <skvidal> okay 19:36:52 <skvidal> so even more than before - this means if you're playbook is NOT idempotent - you need to make it be so 19:36:53 <skvidal> :) 19:37:05 <nirik> yes, I think we should strive to make them all so. 19:37:34 <nirik> that would help for a nightly run too... then if something changed, we know someone futzed with things. 19:38:16 <skvidal> okay 19:38:28 <nirik> #info working out how to let folks run specific playbook runs themselves. 19:38:39 <skvidal> so one more thing on the triggered runs 19:38:46 <nirik> #info working out how triggering playbook runs should work. 19:39:00 <skvidal> here's what i've been basing from 19:39:05 <skvidal> for determining which hosts 19:39:18 <skvidal> if the playbook is in playbooks/[hosts,groups] 19:39:21 <skvidal> then it is obvious and simple 19:39:30 <nirik> yeah, --list-hosts will show it. 19:39:41 <nirik> (for that easy case with a playbook) 19:39:48 <skvidal> if it is a task - then look up which playbooks that task shows up and run those playbooks 19:39:59 <skvidal> if the file is in $files then apply-global 19:40:15 <skvidal> I can probably make the files thing a BIT more specific - but it's kind a crapshoot :) 19:40:26 <skvidal> if the modification is in groupvars/hostvars - obvious rules apply 19:40:45 <nirik> roles would be like tasks? (ie, look up which it's included in)? 19:40:51 <skvidal> yah - that's the idea 19:41:25 <skvidal> so it's any mod/add/rm 19:41:33 <skvidal> does that make no sense to anyone? 19:41:54 <nirik> makes sense to me. I think there may be corner cases, but we can deal with them. 19:42:06 <skvidal> the paths i've not looked at are the plugins and library, handlers, etc 19:42:29 <nirik> we change those so rarely, just apply-global as a first cut 19:42:33 <nirik> IMHO 19:42:34 <skvidal> and finally people making 'inventory' changes makes things.. tricky 19:42:56 <skvidal> since it is impossible to know if the inventory change is substantive :) 19:43:00 <nirik> yeah. 19:43:10 <skvidal> okay that's all I have 19:43:12 <smooge> I am working a couple of tickets. Will be installing another PPC once it is racked and stacked and I get a kernel fix so I can run KVM sessions again 19:43:21 <nirik> and cases like if you remove a playbook, do nothing, etc. 19:43:21 <skvidal> if anyone is interested in any of this stuff - please let me know 19:43:39 <nirik> #info ansible help welcome. See skvidal. :) 19:43:48 <nirik> smooge: cool. 19:44:01 <smooge> KVM on PPC is not possible yet 19:44:01 <skvidal> nirik: removing a playbook - yes - would be doing nothing - but.... that's a little different ;) 19:44:06 <smooge> so it will be LPAR 19:44:12 <nirik> smooge: :( bummer. 19:44:29 <nirik> so other pending sysadmin items: 19:44:42 <nirik> #info new bladecenter is all arrived, trying to get network to it. 19:45:00 <nirik> #info will be adding mem and replacing 2 bvirthosts sometime this quarter 19:45:16 <nirik> #info phx2 on-site visit time probibly in mid/late july. 19:46:04 <nirik> #info working on new rdiff-backup setup for backups, probibly after freeze 19:46:25 <nirik> Anything else on the sysadmin side of things? 19:46:46 * skvidal has nothing else 19:46:50 <nirik> #topic Private Cloud status update / discussion 19:46:57 <nirik> not much cloud news recently. 19:47:05 <skvidal> well - sorta 19:47:10 <skvidal> openstack renamed more random crap 19:47:12 <nirik> I have some f19-tc5 images loaded in, need to update them to tc6. ;) 19:47:16 <nirik> yeah, boo. 19:47:30 <skvidal> nirik: hopefully we can get 03 moved to an external port 19:47:35 <skvidal> nirik: and we can do multihost grizzly setup 19:47:49 <nirik> #info moving networking on a spare node so we can do a multihost grizzly setup 19:48:03 <nirik> #info f19 test compose images available for testing/tweaking. 19:48:21 <nirik> #topic Upcoming Tasks/Items 19:48:21 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/ 19:48:33 <nirik> any upcoming tasks/items folks would like to schedule or note? 19:49:29 * nirik is crossing fingers for an on time release, but you know how it is... ;) 19:49:34 <nirik> #topic Open Floor 19:49:52 <nirik> anyone have items for open floor? 19:49:57 <nirik> questions? comments? suggestions? 19:50:59 <kc4zvw_> nirik: I have access issues to work after meeting 19:51:00 <skvidal> where is everyone today? 19:51:00 * nirik will close out the meeting in a minute if not 19:51:17 <smooge> ok 19:51:22 <nirik> kc4zvw_: ok. Can try and assist over in #fedora-admin 19:51:32 <nirik> skvidal: not sure. 19:52:42 <nirik> Thanks for coming everyone! 19:52:44 <nirik> #endmeeting