============================================ #fedora-meeting: Infrastructure (2011-07-28) ============================================

Meeting started by nirik at 19:00:00 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2011-07-28/infrastructure.20...

Meeting summary --------------- * Robot Roll Call (nirik, 19:00:01)

* New folks introductions and apprentice tasks/feedback (nirik, 19:02:44) * If you are an apprentice and want to get (re) involved, look at easyfix tickets and/or chime in on other topics in channel to get something to work on. ;) (nirik, 19:05:03)

* Moving SOP docs from wiki to git (nirik, 19:07:18) * ACTION: skvidal to make new repo (nirik, 19:16:04) * ACTION: nirik to write up migration plan (nirik, 19:16:11) * ACTION: nirik will investigate updating CSI docs, or see what we can do to update them. (nirik, 19:17:36)

* QA network setup (nirik, 19:22:19)

* Upcoming Tasks/Items (nirik, 19:29:02)

* Meeting tagged tickets (nirik, 19:32:09) * LINK: https://fedorahosted.org/fedora-infrastructure/query?status=new&status=a... (nirik, 19:32:24)

* Open Floor (nirik, 19:41:55)

Meeting ended at 20:07:04 UTC.

Action Items ------------ * skvidal to make new repo * nirik to write up migration plan * nirik will investigate updating CSI docs, or see what we can do to update them.

Action Items, by person ----------------------- * nirik * nirik to write up migration plan * nirik will investigate updating CSI docs, or see what we can do to update them. * skvidal * skvidal to make new repo * **UNASSIGNED** * (none)

People Present (lines said) --------------------------- * nirik (109) * skvidal (91) * abadger1999 (43) * herlo (33) * smooge (32) * zodbot (4) * CodeBlock (4) * ianweller (2) * StylusEater (1) * ricky (0) * codeblock (0) -- 19:00:00 <nirik> #startmeeting Infrastructure (2011-07-28) 19:00:00 <zodbot> Meeting started Thu Jul 28 19:00:00 2011 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:01 <nirik> #meetingname infrastructure 19:00:01 <zodbot> The meeting name has been set to 'infrastructure' 19:00:01 <nirik> #topic Robot Roll Call 19:00:01 <nirik> #chair smooge skvidal codeblock ricky nirik abadger1999 19:00:01 <zodbot> Current chairs: abadger1999 codeblock nirik ricky skvidal smooge 19:00:07 * skvidal is here 19:00:20 * abadger1999 here 19:00:29 <CodeBlock> hi there 19:00:51 <nirik> hello all. 19:01:14 * nirik waits another minute or so for more folks to wander in. 19:01:21 * herlo is here 19:01:24 <herlo> mostly 19:02:00 <skvidal> herlo: how goes your avian project? 19:02:09 <herlo> skvidal: it goes 19:02:25 <herlo> making decent progress, not as fast as I'd like, but good 19:02:31 <nirik> ok, I guess lets go ahead and get started... 19:02:37 <skvidal> waterfowl are gross, gross creatures. I wouldn't expect dealing with them would be a quick thing 19:02:42 <smooge> I am ready 19:02:44 <nirik> #topic New folks introductions and apprentice tasks/feedback 19:02:47 <herlo> skvidal: haha, yeah 19:02:47 <skvidal> herlo: I wish you luck 19:02:50 <herlo> thx 19:03:10 <nirik> any apprentice folks want to ask questions or note tickets? any new folks want to introduce themselves? 19:04:08 <nirik> I've seen a tapering off of apprentice activity of late... possibly due to schools restarting and people getting busy? or just general summer malaze? 19:04:23 <skvidal> both 19:04:24 <skvidal> I suspect 19:04:29 <skvidal> people chilling out for their summer 19:04:35 <skvidal> and prepping for school 19:04:38 <skvidal> I suspect in oct or so 19:04:49 <skvidal> when people are riding back comfortably in the reigns of school 19:04:53 <skvidal> then we'll see them come back 19:05:03 <nirik> #info If you are an apprentice and want to get (re) involved, look at easyfix tickets and/or chime in on other topics in channel to get something to work on. ;) 19:05:09 <nirik> yeah, could well be. 19:05:14 <skvidal> bonus points if you get the literary reference there 19:05:44 * nirik doesn't off hand. ;( 19:06:23 <herlo> dead poet's society? and I didn't google it, especially since I'm probably wrong :) 19:06:54 <nirik> carpe diem! 19:06:58 <herlo> :P 19:07:02 <skvidal> herlo: ray bradbury - something wicked this way comes 19:07:10 <nirik> anyhow, if nothing else on apprentice tasks, moving on... 19:07:14 <herlo> not much of a literary buff 19:07:18 <nirik> #topic Moving SOP docs from wiki to git 19:07:25 <skvidal> w00t 19:07:35 <smooge> csi? 19:07:36 <skvidal> nirik: so - we just need a decision - put them with the csi docs or put them in their own repo 19:07:42 <nirik> so, there are some advantages and disadvantages here. 19:07:48 <StylusEater> skvidal: put them on github? :-) 19:07:56 <nirik> but I think the advantages outweigh the disadvantages. 19:07:57 <skvidal> StylusEater: <stab> 19:08:06 <skvidal> what are the disads? 19:08:08 * nirik looks at the csi docs repo. 19:08:18 <skvidal> nirik: is csi docs repo even a repo? 19:08:20 <skvidal> or is it just a dir? 19:08:24 <smooge> it is a repo 19:08:25 <nirik> less ability for $otherpeople to correct things/contibute. 19:08:34 <skvidal> smooge: where is it housed/ 19:08:57 <smooge> Most of the stuff under it is git clone git://git.fedorahosted.org/csi.git 19:09:04 <skvidal> okay 19:09:04 <nirik> it's on hosted. 19:09:07 <skvidal> that I didn't understand 19:09:18 <skvidal> I'm not in favor of our sop docs being on hosted 19:09:29 <nirik> yeah, that drops one of the advantages. 19:09:31 <skvidal> right 19:09:41 <smooge> well it could be moved. 19:09:42 <skvidal> and I'd rather have to protect a single basket in the event of disaster 19:09:44 <skvidal> not multiple ones 19:09:45 <abadger1999> yeah 19:10:02 <nirik> it's also using publican I think... 19:10:20 <abadger1999> harder to point $otherpeople at how to do something. 19:10:32 <smooge> Correct. The CSI documents are supposed to be the policies and the SOPs are supposed to be the how to complete the policies 19:11:38 <skvidal> right 19:11:38 <skvidal> so 19:11:42 <nirik> so, if we had a repo on infrastructure, could we have it allow the same groups as the wiki does to edit? cla_done+1 or whatever. 19:11:42 <smooge> The main thing I wanted to was to make sure that we keep both in sync 19:11:43 <abadger1999> think we're agreed.... git repo on lockbox separate from the other git repos males more sense. 19:11:51 <smooge> yes I agree on that 19:12:07 <abadger1999> *makes sheesh, the typos today. 19:12:10 <skvidal> in the event of a disaster 19:12:15 <skvidal> I don't need to see our policies 19:12:18 <nirik> we probibly need to give the CSI docs a good lookover/edit/cleanup someday. 19:12:20 <skvidal> I will need to see our SOPs 19:12:24 <skvidal> cleaning up CSI makes sense 19:12:35 <skvidal> but I wouldn't want to clean it up at the detriment of SOPs being current 19:12:44 <herlo> yeah, that was my question nirik, what is the format going to be, publican? or something else? 19:12:49 <skvidal> herlo: txt 19:12:50 <skvidal> txt file 19:13:02 <nirik> yeah, text would be fine with me. 19:13:06 <skvidal> any system which involves effort to maintain == doom 19:13:09 <skvidal> b/c people will avoid it 19:13:15 <herlo> somethign like rst would be nice 19:13:17 <skvidal> and the SOPs don't have anything that is not text-able 19:13:23 <herlo> and it still is basically text 19:13:30 <nirik> it doesn't need to be pretty, just contain the data. Hopefully so you can cut and paste things. 19:13:35 <herlo> but I supppose we could make that later... 19:13:40 <herlo> yeah 19:13:53 <smooge> I would prefer the SOP's to be in text 19:14:03 <skvidal> nirik: paste++++ 19:14:32 <nirik> so, any objections to that plan? If not, we need to plan the move, convert docs and redirect as we go... I could write up a plan for it. 19:15:03 <skvidal> nirik: I can take the action item to make the repo and get the pushing happening to a path in infra/web 19:15:10 <skvidal> err /srv/web/infra/ 19:15:32 <nirik> skvidal: ok. Thanks. Perhaps we need a 'new infra git repo with hooks' SOP. ;) 19:15:54 <skvidal> nirik: I genericized the hook code 19:16:00 <skvidal> so I don't have to do it a billion times 19:16:04 <nirik> #action skvidal to make new repo 19:16:04 <skvidal> when I setup infra-hosts 19:16:11 <nirik> #action nirik to write up migration plan 19:16:23 <abadger1999> I'm still lamenting lack of TOC and hyperlinks but.... getting it off the wiki is pretty important. 19:16:47 <skvidal> abadger1999: suggestions welcome - but not publican 19:16:52 <nirik> we could make an index.html? ;) 19:16:52 <skvidal> abadger1999: that's A LOT of infrastructure for links 19:16:52 <abadger1999> *shudders* 19:17:36 <nirik> #action nirik will investigate updating CSI docs, or see what we can do to update them. 19:17:41 <abadger1999> I wouldn't wish publkican on this in a million years. 19:18:06 * CodeBlock is back, sorry, had to talk to my boss about a $dayjob project. 19:18:09 <smooge> having done this in the long ago past.. you use a minimal markup at the top to say things like: Title:, Reason: Keywords, and then use a quick txt2html wrapper which makes html files with just <pre></pre>and index.html to link them 19:18:30 * nirik dealt with docbook when he did his howto, would really prefer to avoid that complexity (and publican is another layer on top of that) 19:18:59 <abadger1999> smooge: Yeah, something more like that is what I was envisioning. 19:19:05 <skvidal> nirik: docbook---- 19:19:15 <skvidal> nirik: I've done it there, too - with the nfs-howto - it made me cranky 19:19:17 <abadger1999> smooge: Maybe parse some sort of internal headers as well. 19:19:20 <smooge> basically a checkin rebuilds the stuff and we go 19:19:25 <skvidal> abadger1999: I suspect someone has this 19:19:29 <skvidal> abadger1999: and it is lightweight 19:19:30 <skvidal> and simple 19:19:39 <skvidal> abadger1999: CodeBlock mentioned something 19:19:41 <skvidal> markdown? 19:19:46 <nirik> we can look at implementation details out of band? 19:19:57 <ianweller> markdown is an *okay* language, it's not great 19:20:03 <skvidal> nirik: nod 19:20:04 <smooge> skvidal, only in their 0.1 versions.. then people start asking for features and it becomes an XML/SGML/publican thing in the 0.2 version 19:20:20 <abadger1999> +1 19:20:25 <smooge> yeah offline after meeting 19:20:35 <nirik> I think possibly just adding a link/description to a index.html when you add a SOP could be enough... 19:20:50 <ianweller> nirik: directory listing? 19:21:10 <skvidal> post-meeting 19:21:12 <nirik> ie, "GomGabbar SOP - use this when you want to install a new Gom Gabbar device" 19:21:14 <skvidal> in fedora-admin 19:21:15 <nirik> anyhow, yeah. 19:21:29 * nirik cares not what colour the bike shed is. 19:21:45 <nirik> anything else on this? or shall we move on? 19:21:45 * skvidal is partial to blue-gray 19:21:47 <CodeBlock> ianweller: I like Markdown because when it's not parsed, stuff still stands out/looks good/is distinguishable in plaintext. As in headers are underlined (with - or =) etc. 19:21:58 * CodeBlock shuts up so we can move on :P 19:22:19 <nirik> #topic QA network setup 19:22:34 <nirik> ok, so we talked about this some last week... here's the conclusion I came to: 19:22:56 <nirik> monitoring - use our nagios for monitoring, and have alerts go to sysadmin-qa folks. 19:23:30 <nirik> config management - try out bcfg2 there. This means removing virthost-comm01 and bastion-comm01 from our puppet and adding them into bcfg2 there. 19:24:06 <nirik> I'm undecided if they should have a seperate lockbox-comm01 for bcfg2 or not... I guess so to be carefull. 19:24:17 <nirik> so, if anyone wants to help with that setup, please do. ;) 19:24:44 <skvidal> ok 19:24:55 <smooge> ok will do so 19:25:02 <nirik> once we get it setup, we can re-eval down the road... if bcfg2 isn't working we can switch them out. 19:25:16 <smooge> cfengime 19:25:16 <nirik> they will still need to use our setup for a few things... like repos I suspect. 19:25:48 <nirik> also, in the qa space a few more things down the road: 19:26:25 <nirik> it would be good if we could add some fedora ks files if we want them to use our kickstarts (which I think is probibly easiest, since they have to use our repos anyhow) 19:27:11 <nirik> probibly before too long will be adding some secondary arch signing instances there... which will let us test out the new sigul/setup 19:27:30 <skvidal> heh 19:27:32 <nirik> thats all I had off hand on qa network stuffs. Any questions/concerns/ideas? 19:27:36 * skvidal read that as 'sighing instances' 19:27:49 <nirik> yeah, basically. ;) 19:28:39 <nirik> ok, moving on. 19:29:02 <nirik> #topic Upcoming Tasks/Items 19:29:14 <nirik> So, monday morning we have some reboots... 19:29:29 <nirik> tuesday is the start of f16alpha freeze. 19:30:11 <nirik> we have some new machines hopefully racked or being racked, so they will need installing and adding to monitoring. 19:30:46 <nirik> Any other upcoming items folks would like to note/schedule/ask about/plan for? 19:32:01 <nirik> ok. moving on then. 19:32:09 <nirik> #topic Meeting tagged tickets 19:32:24 <nirik> https://fedorahosted.org/fedora-infrastructure/query?status=new&status=a... 19:32:30 <nirik> any here folks want to discuss? 19:32:49 * skvidal looks 19:33:06 <smooge> not me 19:33:38 <smooge> 2501 will need us to get hardware up at ibiblio working I believe 19:33:44 <nirik> There's some old stuff there I might close or remove meeting from. 19:33:57 <skvidal> smooge: the hw is there now 19:34:02 <skvidal> when I hear from reuning 19:34:05 <skvidal> I'll go over and hitch it up 19:34:10 <smooge> ok cool 19:34:15 <nirik> well, hosted needs a plan. I might try and tackle that in my copius free time.... 19:34:22 <skvidal> and you shall surely here my cries and curses when the ipv6 stuff doesn't work 19:34:34 <skvidal> nirik: so I had a thought 19:34:35 <skvidal> on hosted 19:34:39 <skvidal> that you may well hate 19:34:43 <skvidal> but I wanted to bring it up 19:34:48 * nirik gets ready on the hate button. 19:35:06 <skvidal> what if we did it piece-meal? 19:35:17 <skvidal> ie: could we setup a new infrastructure that let us do a project at a time 19:35:36 <smooge> I thought that was what we were going to do 19:35:37 <nirik> yeah, I did think of that too. 19:35:46 <skvidal> ah, hmm 19:35:47 <nirik> I don't hate it, but we still need a plan... 19:35:50 <smooge> put a proxy in front and then move through that 19:35:59 <nirik> ie, how many instances, seperated/connected how, etc 19:36:22 <skvidal> nirik: would hosted be an example of service that is well suited to having 'in the cloud'? 19:36:23 <nirik> and preferably a way to make it spread out more. 19:36:49 <nirik> skvidal: I'm not sure. Possibly... it does get hit pretty hard... so it would use a lot of resources. 19:37:03 <skvidal> nirik: which is sorta the point... 19:37:20 <skvidal> nirik: did we ever get an answer on cloud-money? 19:37:35 <nirik> skvidal: no. sadly. 19:37:45 <skvidal> nirik: okay, so I didn't just miss that meeting 19:37:46 <skvidal> okay 19:37:47 <skvidal> thx 19:38:07 <nirik> but any plan could look at how to split it out, and if we have cloud, we could use cloud for some or part of it if it makes sense. 19:38:48 <skvidal> okay 19:38:53 <skvidal> so really this needs some focus 19:38:56 <nirik> yes. 19:39:03 <skvidal> I suspect that's why it's not gotten very far ;) 19:39:08 <skvidal> or rather :-\ 19:39:27 <nirik> I think we all agree it would be good to get it updated/upgraded, but we also want to try and make it less SPOF and such at the same time... 19:39:46 <nirik> anyhow, I can try and at least whip up some plan for people to be inspired to counterpropose. ;) 19:39:46 <skvidal> nod 19:40:02 <skvidal> nirik: not that we have the time right now 19:40:05 <skvidal> nor (probably) the money 19:40:18 <skvidal> but hosted migration sure seems like we'd benefit from a FAD 19:40:21 <nirik> it may be that we should be less far reaching... just plan for moving it the way it is now, and do a longer term thing later. 19:40:34 <nirik> yeah, that could be the case... 19:40:35 <skvidal> nirik: which will, likely, be kicked down the road forever :( 19:41:48 <skvidal> okay 19:41:50 <skvidal> what else? 19:41:51 <nirik> ok, moving on. 19:41:55 <nirik> #topic Open Floor 19:42:00 <nirik> anything for open floor? 19:42:27 <herlo> oh, I do 19:42:37 <nirik> herlo: fire away 19:42:39 <herlo> I had a nice little bug for fpaste-server 19:42:48 <herlo> with django-tracking, it needed to be updated 19:42:58 <herlo> and I'm going to be pushing that back some to accommodate 19:43:04 <herlo> but I needed to ask a couple questions regarding it 19:43:18 <nirik> sure, whats the questions? 19:43:19 <herlo> one is, do we plan to host it on its own vm? or will it live with other services? 19:44:03 <nirik> excellent question. :) I think this is exactly the sort of thing we should be figuring out when a new resource is in the 'dev' stage. ;) 19:44:08 <herlo> I've currently got the package deploying an fpaste.conf which I need to alter to better accommodate vhosts 19:44:51 <herlo> I think it could run with other vhosts as I don't think the load is going to be too high at first. 19:45:32 <nirik> so, there's a spectrum here: on one side is totally seperate. it's own instance with it's own db and own webserver. On the other end is in our proxy/app mix. It's hit via proxy and uses varnish/haproxy, runs on the app servers and talks to a single backend db. 19:46:04 <nirik> there's also some middle ground where it could be using proxy/caching, but have it's own instance and db 19:46:16 <herlo> which if we did the latter, would need to be similarly setup somewhere along the way, right? 19:46:25 <herlo> would that be in dev or rather in staging? 19:46:50 * herlo thinks most of this convo can go offline, but just wanted to bring up these thoughts 19:47:00 <abadger1999> Are our proxies a limited resource? 19:47:16 <herlo> especially since we're working on the SOPs for dev and staging rollout 19:47:28 <nirik> dev => no proxy or other setup, stg => setup like it would be in prod 19:47:33 <herlo> abadger1999: a good question, wish I knew 19:47:35 <nirik> abadger1999: I don't think so... 19:48:08 <herlo> nirik: k, I'll work with it that way then... 19:48:10 <herlo> thanks 19:48:18 <abadger1999> nirik: I get a good feling about the middle way, then, treat the app as a separate resource but the infrastructure around it as shared. 19:48:49 <abadger1999> so that we can upgrade the host/db for one app independently of the host for a different app. 19:48:50 <nirik> yeah, adding more to a single db is something I wish to avoid... more eggs in one basket. 19:49:17 <abadger1999> I'm not sure what makes the most sense from a sysadmin/money perspective though. 19:49:46 <nirik> I guess if we had better db replication it might be less anoying. 19:51:12 <nirik> I guess it also depends on load/how popular something becomes. 19:51:44 <nirik> if it's really popular and we need more app resources, we could also move an app from it's own instance out to the app* machines to spread that out... 19:51:49 <abadger1999> I'm not sure if separate db's gains us as much as separate app servers... I think it would distribute load and allow tweaking individual dbs for different types of queries but I'm not sure we have those issues. 19:52:10 <smooge> I would prefer middle road proxies -> app-fpasteXX -> db-fpasteXX 19:52:14 <abadger1999> it's a the db is a SPOF for an app whether it's in a shared db or separate. 19:52:22 <abadger1999> s/it's a// 19:52:35 <smooge> yeah but instead of 20 apps going down for an hour.. we may have 1. 19:52:50 <abadger1999> smooge: All depends.... 19:52:57 <smooge> right now when we reboot the db servers, most of everything Fedora is offline 19:53:02 <smooge> and if the db doesn't come back 19:53:30 <abadger1999> why did it go down? postgres update or hardware? Did it take out fas? etc etc. 19:53:40 <herlo> so I basically setup my app as normal and then we add it to the lb structure? That sound about right? 19:54:04 <nirik> herlo: yeah, that work takes place in stg step... but do be thinking about it now, I think thats good. 19:54:07 <herlo> there isn't a lot to fpaste-server. 19:54:09 <smooge> usually it goes down because we need to reboot or the hardware itself needs to reboot and then it all waits til it comes back 19:54:11 <abadger1999> smooge: <nod>... otoh, if we have five different db serves, doesn't that mean we're five times as likely to have any single piece of hardware go bad? 19:54:39 <smooge> abadger1999, no.. murphy is kind to us there. The hardware will go bad sometime. 19:54:44 <herlo> cool, this sounds like we could do some cool stuff like sticky sessions and add an additional fpaste machine if ever needed. 19:54:45 <smooge> anywhere 19:54:47 * herlo likes 19:54:47 <abadger1999> :-) 19:55:26 <nirik> I guess memory on app servers is a limited resource... 19:55:29 <nirik> and cpu there. 19:55:29 <smooge> abadger1999, you can't go over 100% failure possibility 19:55:35 <abadger1999> hehe :-) 19:56:23 <nirik> if someone would like to add these questions into https://fedoraproject.org/wiki/Request_for_resources_SOP that would be great. ;) 19:56:28 <abadger1999> nirik: Hmm... memory is the limiting factor for dbs in my experience.... 19:56:33 <smooge> I mean I figure if we could devote time and effort we could have failover and clustering of some sort in place.. but I think that has been the equivalent of bug#1 since 2005? 19:57:20 * nirik nods. 19:57:29 <nirik> so, anything more? or shall we call it a meeting? 19:57:30 <abadger1999> smooge: Well... I think the reason it's still bug #1 is that no one's laid down what we want to solve and what limitations we're willing to live with. 19:58:19 <abadger1999> ie: We could have warm backups of postgres and masterslave of mysql right now... but if we anticipate switching to the failover machines in case of db outage we're going to have to acept that we might lose data. 19:58:52 <smooge> abadger1999, I agree. 19:58:55 <abadger1999> I think we could accept that... but no one wants to actually commit to it. 19:58:58 * nirik nods again. 19:59:06 <smooge> I commit to losing data 19:59:22 <smooge> that will make a great Compass goal 19:59:27 <abadger1999> hehe :-) 19:59:43 <skvidal> abadger1999: can we quantify the kind of loss? 20:00:06 <nirik> I had this random thought yesterday... I might start tossing out disaster scenerios to the mailing list. ;) "phx2 is gone. What do we have left? how would we recover" "serverbeach down, what do we have, how do we recover" etc. 20:00:48 <abadger1999> skvidal: I'm thinking in many cases, it would be minimal -- postgres warm backups rsync the transaction logs at a period you define. So you'd lose the data from that period. 20:00:59 <skvidal> then +1 20:01:05 <abadger1999> skvidal: We'd probably set it somewhat low... maybe 5 minutes. 20:02:13 <abadger1999> mysql master-slave can get out of sysnc. if it does and master fails in that time, we lose all the data that didn't get synced. If we fix any out-of-sync errors promptly and we don't fail when out of sync, we'd only lose a few minutes of data there as well. 20:03:22 <abadger1999> nirik: Perhaps we should do that with one of the new apps? Set up a replicated db server for it. 20:03:26 <abadger1999> See how it works. 20:03:37 <nirik> yeah, thats a possibility... 20:03:46 <nirik> one in phx2 and one elsewhere? 20:03:46 <abadger1999> nirik: The one issue I'd see is... db servers take a lot of memory to be high performance. 20:04:12 <abadger1999> And you want the two boxes to have the same specs so that if you have to switch to the other box, it can take the load. 20:04:15 <abadger1999> well... 20:04:27 <nirik> yeah. true 20:04:32 <abadger1999> network *latency* can be an issue with app to db. 20:04:46 <nirik> although sometimes slower performance is still better than down. 20:05:03 <abadger1999> I think the app servers not at phx show that when they try to talk to the db there. 20:05:31 <abadger1999> so it may make sense to have the db servers not at phx if our intent is protecting data in case phx disappears. 20:05:51 <nirik> or if there's some way to do master/master. ;) 20:05:52 <abadger1999> but it may not if our intent is to have a db server to drop into place if the db server in phx goes kaput. 20:06:10 <abadger1999> master/master tends to slow everything down from what I can tell. 20:06:18 <nirik> yeah, I would imagine so... 20:06:28 <abadger1999> Since you have to wait for the slowest master to commit the data. 20:06:42 <nirik> ok, lets keep pondering on it, and discuss more out of meeting/next week. ;) 20:06:48 <abadger1999> <nod> 20:06:59 <nirik> Thanks for coming everyone! 20:07:04 <nirik> #endmeeting