infrastructure January 2010

infrastructure@lists.fedoraproject.org

42 participants
45 discussions

which tool chain are you guys using for c(not c++) application on fedora
by yunhui song 21 Jan '10

21 Jan '10

Hi folks, I'm thinking develop service on fedora by C. I had some experience on GNU/C Makefile, gdb, Eclipse CDE. I'm using vim as my editor. What kind of tool chain are you using? Hope to get better toolset from fedora community. Thanks, Sammi

1 0

Outage Notification - 2010-01-21 11:00 UTC
by Mike McGrath 21 Jan '10

21 Jan '10

There will be an outage starting at 2010-01-21 11:00 UTC, which will last approximately X hours. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2010-01-21 11:00 UTC' Affected Services: Websites Buildsystem Unaffected Services: Bodhi CVS / Source Control Database DNS Fedora Account System Fedora Community Fedora Hosted Fedora Package Database Fedora People Fedora Talk Mail Mirror System Torrent Translation Services Ticket Link: https://fedorahosted.org/fedora-infrastructure/ticket/1939 Reason for Outage: Lost a switch blade this morning. A replacement is on the way. Contact Information: Please join #fedora-admin in irc.freenode.net or respond to this email to track the status of this outage.

1 0

host-lifecycle-policy
by Mike McGrath 20 Jan '10

20 Jan '10

It dawns on me I never actually sent this to the list for comments and we haven't officially adopted it yet. http://infrastructure.fedoraproject.org/csi/host-lifecycle-policy/en-US/htm… It's still in flux but give it a read over. For some stuff it seems like it adds more work (like post kickstart checklist) but really it just lists what needs to be done as part of bringing a host online. Backups and monitoring are things that often get forgotten about. I'm working on a very similar doc for bringing services online. -Mike

3 3

Won't be around off-hours
by Toshio Kuratomi 18 Jan '10

18 Jan '10

My wife is going to a funeral out of state for a week so I'm home with the kids. I'll still be here normal working hours but not so much outside of those. -Toshio

1 0

logging infrastructure and notes
by Seth Vidal 15 Jan '10

15 Jan '10

I did a little spelunking around our system and I have some suggestions for the logging infrastructure. We have enough hosts and complexity that log analysis will help us know when something is misconfigured or flapping in a weird way. 1. logs in /var/log/hosts on log1 are not consistently named - sometimes they are being reported with ips, sometimes with short hostname, sometimes with fqdn. It needs to be made consistent 2. we need to make sure we cleanup old logs from the above, too. 3. the structure of the log dir doesn't seem to match what we'd normally see in /var/log on any host. They are being logged as a different dir per day, which is great, but it'd be good if rsyslog was putting in the same file structure as a normal set of logs so normal log analysis tools will work on it 4. I installed pflogsumm on log1 so I could do a little postfix mail log analysis - found some issues that way too. Regularly generating these reports, especially the error reports would help us figure out what we need to improve. We are clearly sending/redelivering A LOT more mail than we're receiving so bumping our smtp process count would help. 5. Grouping the logs by type of service would also help look at group/service trending and issues. especially if an issue is only popping up on one box. Just some initial thoughts. -sv

4 9

Outage Notification - Fri Jan 15 10:30:00 UTC
by Mike McGrath 15 Jan '10

15 Jan '10

There will be an outage starting at Fri Jan 15 10:30:00 UTC, which is ongoing. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d 'Fri Jan 15 10:30:00 UTC' Affected Services: Fedora People Unaffected Services: Buildsystem CVS / Source Control Database DNS Fedora Hosted Fedora Talk Mail Mirror System Torrent Translation Services Websites Ticket Link: https://fedorahosted.org/fedora-infrastructure/ticket/1930 Reason for Outage: The xen host fedorapeople is on is currently not responding to pings. I've contacted the hosting provider but have not heard back yet. It is unclear when this host will be back online. Contact Information: Please join #fedora-admin in irc.freenode.net or respond to this email to track the status of this outage.

1 0

Meeting Log - 2010-01-14
by Ricky Zhou 14 Jan '10

14 Jan '10

20:01 < mmcgrath> .startmeeting Infrastructure 20:01 < mmcgrath> Who's here? 20:01 -!- belegdol [n=belegdol@fedora/belegdol] has joined #fedora-meeting 20:01 * wzzrd is here 20:01 * nirik hands mmcgrath a # 20:01 * sijis is here 20:01 < PhrkOnLsh> mind if i watch the show? 20:01 -!- a-k [n=akistler@2002:6390:aba9:3:20d:56ff:fe10:bb8d] has joined #fedora-meeting 20:02 < mmcgrath> PhrkOnLsh: not at all 20:02 < nirik> mmcgrath: you want #startmeeting there, not .startmeeting. ;) 20:02 < mmcgrath> #startmeeting Infrastructure 20:02 < zodbot> Meeting started Thu Jan 14 20:02:37 2010 UTC. The chair is mmcgrath. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:02 < zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 20:02 -!- zodbot changed the topic of #fedora-meeting to: (Meeting topic: Infrastructure) 20:02 < mmcgrath> damnit 20:02 < mmcgrath> Who's here? 20:02 * a-k is 20:02 * wzzrd is here 20:02 * nirik is hanging around in the back 20:02 -!- _lmr_ [n=lmr(a)187.64.216.6] has joined #fedora-meeting 20:02 * PhrkOnLsh . 20:02 * skvidal is here 20:03 < mmcgrath> Ok, lets get started 20:03 < mmcgrath> #topic Infrastructure -- Tickets 20:03 -!- zodbot changed the topic of #fedora-meeting to: Infrastructure -- Tickets (Meeting topic: Infrastructure) 20:03 < mmcgrath> Looks like no meeting notes so that's good. 20:03 < mmcgrath> I'll go over just a couple of things that happened this week worth mentioning 20:03 < mmcgrath> #topic Fedora hosted 20:03 -!- zodbot changed the topic of #fedora-meeting to: Fedora hosted (Meeting topic: Infrastructure) 20:04 < mmcgrath> hosted got it's new memory replaced. 20:04 < mmcgrath> it wasn't as smooth as it could have been but it did get done and within the outage window time. 20:04 < mmcgrath> #topic SPOF for koji and bastion 20:04 -!- zodbot changed the topic of #fedora-meeting to: SPOF for koji and bastion (Meeting topic: Infrastructure) 20:04 < mmcgrath> koj and bastion are still SPOF right now. 20:04 -!- mdomsch [n=Matt_Dom(a)cpe-70-124-57-11.austin.res.rr.com] has joined #fedora-meeting 20:04 < mmcgrath> mostly because we've not re-configured heartbeat. 20:04 < mmcgrath> but I'm also trying to re-think our vpn setup. 20:04 < mmcgrath> to be more robust with outage scenarios. 20:05 < mmcgrath> not allowing outbound udp makes that difficult. 20:05 < mmcgrath> though I suspect specific outound udp requests could be allowed. 20:05 < mmcgrath> #topic /mnt/koji 20:05 -!- zodbot changed the topic of #fedora-meeting to: /mnt/koji (Meeting topic: Infrastructure) 20:05 < dgilmore> im waiting on the try-n-buy to be approved 20:05 * ricky is here for a bit 20:05 < mmcgrath> So this one's kind of happened behind closed doors, not intentional just lots of "do you think this would work?" "how much budget?" etc, etc. 20:06 < mmcgrath> but yeah, dgilmore's been working on trying to get what will be the new /mnt/koji 20:06 < mmcgrath> and I've been working to figure out what will be the new backup of /mnt/koji. 20:06 < mmcgrath> this is all good for several reasons. 20:06 < mmcgrath> 1) it'll allow us to use all of our tapes for *everything* else and do daily, weekly and monthly backups as we should be doing. 20:06 < skvidal> mmcgrath: what's the plan for the storage for the backup? Still tape or are you switching to disks? 20:06 < mmcgrath> 2) /mnt/koji will hopefully be much faster. 20:07 < mmcgrath> skvidal: switching to disks. I'm hoping we'll be able to make snapshots of it for testing. 20:07 < mmcgrath> but dgilmore and I also talked about using bacula to backup to disk. 20:07 < mmcgrath> in the end though I think it will be a better solution because if /mnt/koji does die right now a horrible death. 20:07 < mmcgrath> it'd take a long time to get a new /mnt/koji up 20:07 < mmcgrath> whereas if we have disks to backup to I think the impact would be lessoned. 20:08 < mmcgrath> I do, however, have concerns just because it's so easy to wipe disks. 20:08 < dgilmore> mmcgrath: i agree 20:08 < mmcgrath> blowing away the /mnt/koji backup right now is a significantly more difficult task I think. 20:08 < mmcgrath> but at the end of the day I think it'll be worth it to us. 20:09 < mmcgrath> we'll be spending something like 5X what we spent for the /mnt/koji now so hopefully it'll all work out :) 20:09 < mmcgrath> ehh actually that's not quite true, probably closer to 4X 20:09 < mmcgrath> but still. 20:09 < dgilmore> and ive requested extra budget for next yera to grow it and hopefully help it more 20:09 < mmcgrath> here's to hoping it's all fast and usable and safe and backed up :) 20:09 < mmcgrath> anyone else have any questions or concerns on that? 20:10 < mmcgrath> alllrighty 20:10 < mmcgrath> #topic ssh_known_hosts 20:10 -!- zodbot changed the topic of #fedora-meeting to: ssh_known_hosts (Meeting topic: Infrastructure) 20:10 < mmcgrath> smooge: around to talk about this? 20:11 < mmcgrath> the smooge is likely busy :) 20:11 -!- walters [n=walters@nat/redhat/x-dtvgzmjuntdndbxu] has quit Remote closed the connection 20:11 < mmcgrath> After the move our ssh_known_hosts got way out of wack and instead of fixing it, he's been redoing it from scratch 20:11 < mmcgrath> and including more possible names so it should be even more useful then it was. 20:11 < smooge> here 20:11 < smooge> sorry 20:11 < smooge> one sec 20:11 < mmcgrath> smooge: no worries, just wanted a quick blurb about ssh_known_hosts so people know we're working on it. 20:12 < mdomsch> do our shells automatically use it, or do we need to do something in .ssh/config for it? 20:12 < smooge> ok sorry. I have updated ssh_known_hosts to all systems I could find in DNS and 20:12 < mmcgrath> mdomsch: they'd use it automatically but you should remove your .ssh/known_hosts 20:12 < smooge> removed ones that were no longer available 20:12 < mmcgrath> mdomsch: worst case, you'll get a conflict in known_hosts and you can just remove it. 20:12 < mdomsch> ok 20:12 -!- DraZoro1 [n=CT(a)vc-41-18-142-115.umts.vodacom.co.za] has joined #fedora-meeting 20:13 < smooge> I have tested it on app01.stg and was able to go to hosts inside fedora. I have commited and pushed 20:13 < mmcgrath> but the search order is ~/.known_hosts then /etc/ssh/ssh_known_hosts 20:13 < smooge> The only things that aren't in it are IPV6 addresses 20:13 < mmcgrath> smooge: excellent, thanks. 20:13 < mmcgrath> smooge: have you updated the SOP to match the new search order? 20:13 < smooge> new search order? 20:13 < smooge> and which SOP 20:13 < dgilmore> thats the default search order 20:13 < mmcgrath> http://fedoraproject.org/wiki/Infrastructure/SOP/ssh_known_hosts 20:14 < mmcgrath> smooge: I think right now we just have short hostname, and actual IP of the host. 20:14 < smooge> ah ok will fix 20:14 < mmcgrath> in the SOP I mean, you expanded on that so it'd be good to get it in the example :) 20:14 < mmcgrath> smooge: thanks 20:14 < mmcgrath> Ok, any other questions on that? 20:14 -!- fedbot [n=supybot(a)scrye.com] has quit Remote closed the connection 20:14 < mmcgrath> zodbot: hey look, fedbot's a big quitter. 20:14 < mmcgrath> ok 20:14 < mmcgrath> #topic Search Engine 20:14 -!- zodbot changed the topic of #fedora-meeting to: Search Engine (Meeting topic: Infrastructure) 20:14 < mmcgrath> a-k: take it! 20:14 -!- fedbot [n=supybot(a)scrye.com] has joined #fedora-meeting 20:15 < a-k> The two search candidates I wanted to put in public test are Xapian and Nutch 20:16 < a-k> Xapian installed fine and crawled the wiki for 90 minutes, then died 20:16 < a-k> I think it's fixable, so I still have some hope for Xapian 20:16 < mmcgrath> a-k: has the pt instance worked out for you so far? 20:16 < a-k> For Nutch, I've got Tomcat installed, but not configured yet 20:16 < a-k> This is pt3, BTW 20:16 < a-k> I haven't checked if Tomcat's ports are open 20:16 < mmcgrath> If you need more ram or disk let me know, sometimes we're stuck but sometimes not 20:16 < a-k> Yeah, pt is working fine 20:16 < a-k> I intended to reverse proxy Tomcat through Apache, so no need for extra open ports 20:16 < mmcgrath> a-k: I've never used any of them, what are the key differences? 20:17 -!- mchua is now known as mchua_afk 20:17 < a-k> Xapian is pretty flexible and customizable, so the custom keyword requirement should be satisfied, wherever that goes 20:17 < mmcgrath> then died a good death or a bad death? 20:17 < a-k> Nutch is very pre-packaged, so not so flexible 20:17 -!- DraZoro1 [n=CT(a)vc-41-18-142-115.umts.vodacom.co.za] has left #fedora-meeting [] 20:17 -!- belegdol [n=belegdol@fedora/belegdol] has left #fedora-meeting ["Leaving"] 20:17 < a-k> Xapian didn't like a long URL, plus it may not like non-UTF8 so much 20:18 < mmcgrath> ahhh 20:18 < mmcgrath> interesting 20:18 < a-k> The URL thing I think is fixable with how I do the crawl 20:18 < mmcgrath> <nod> 20:18 < a-k> Plus, I still intend to keep looking at other candidates on my list 20:18 < mmcgrath> maybe not the non-UTF8 thing though huh? 20:18 < a-k> So these aren't necessarily the only two choices 20:18 < mmcgrath> sure 20:18 < abadger1999> Both java solutions? 20:18 * mmcgrath is happy to hear progress is being made. 20:19 < dgilmore> a-k: did we look at the one archive.org uses 20:19 < dgilmore> ? 20:19 < a-k> Yeah, UTF-8 could be the thing that kills most of the candidates, if that's going to be a real requirement 20:19 < mmcgrath> a-k: yeah what was the wiki link again? 20:19 < skvidal> why wouldn't utf-8 be a real requirement? 20:19 < a-k> dgilmore: archive.org uses arhiving, not indexing 20:19 < abadger1999> Or xapian is the C++ solution. /me was thinking lucene. 20:19 < mmcgrath> skvidal: I think non-utf-8 is the possible requirement 20:20 < a-k> .link http://fedoraproject.org/wiki/Infrastructure/Search 20:20 < a-k> #link http://fedoraproject.org/wiki/Infrastructure/Search 20:20 < mmcgrath> a-k: thanks 20:20 < dgilmore> a-k: they have a crawler http://crawler.archive.org/ 20:20 < a-k> Xapian is C, with a little Perl 20:20 < a-k> Nutch is Java, hence Tomcat 20:21 < a-k> dgilmore: I can look at archive.org again 20:21 < a-k> I think that's it, unless there are more questions 20:21 < mmcgrath> a-k: just because I know people will ask, can you make sure that those that have been eliminated have a specific "eliminated because: " section? 20:21 -!- dhillon-v10 [n=dhillon-(a)66.222.202.68.cfl.res.rr.com] has joined #fedora-meeting 20:22 < a-k> I can do that 20:22 * nirik wonders if any solutions here will tie into the mailman/lists archives? 20:22 < smooge> eliminated because: It kills kittens and makes you eat them 20:22 < nirik> to replace pipermail. 20:22 < smooge> why would a search engine replace pipermail? 20:22 -!- rishi [n=rishi@gnu-india/supporter/debarshi] has joined #fedora-meeting 20:22 < skvidal> smooge: he means a new archiver 20:23 < skvidal> nirik: I don't think any of the others are any more maintained 20:23 < mmcgrath> a-k: if you want additional help feel free to ask on the list and recruit :) This is a pretty massive project. Especially for your first one for Fedora. If you need anything feel free to ask :) 20:23 < a-k> Sure. Thanks. 20:23 < mmcgrath> anyone have anything else right now? 20:23 < nirik> sure, personally I think pipermail is ok, but there is a ticket wanting us to replace it with monharc or something. 20:24 * dgilmore is ok with pipermail 20:24 < skvidal> nirik: mhonarc is more trouble than its worth, ime 20:24 < skvidal> mmcgrath: logging? anyone interested in working on it? 20:24 * nirik nods. perhaps 3.0 will come out someday with pipermail improvements. 20:25 < wzzrd> I'm kinda working on it, fwiw 20:25 < skvidal> nirik: perhaps monkeys will fly out of my arse 20:25 < smooge> skvidal, I am interested in doing it 20:25 < wzzrd> the logging that is 20:25 < smooge> I was starting to do it after I finished DNS/NTP 20:25 < mmcgrath> skvidal: sure 20:25 < mmcgrath> #topic logging 20:25 -!- zodbot changed the topic of #fedora-meeting to: logging (Meeting topic: Infrastructure) 20:25 < smooge> wzzrd, what are you doing 20:25 < skvidal> smooge: okay 20:25 < smooge> skvidal, what did you see 20:26 < skvidal> there are a number of things to work on 20:26 < mmcgrath> So logging is a pretty large topic with lots of sub parts 20:26 < skvidal> first - cleaning up the logs we have 20:26 < smooge> skvidal, or would I be too many cooks 20:26 < mmcgrath> lets start on what I gave wzzrd a week or two back. 20:26 < skvidal> okay 20:26 < mmcgrath> which is log analysis. 20:26 < skvidal> which is the wrong place to start 20:26 < mmcgrath> wzzrd: have you had a chance to look at some of the suggestions on th elist? 20:26 < skvidal> until you have logs under control 20:26 < mmcgrath> skvidal: none of which will get done while this meeting is going on :) 20:27 < wzzrd> i've checked out epylog a bit further 20:27 < skvidal> umm - if you don't want to discuss logging, that's fine 20:27 * skvidal is sorry for making the meeting longer 20:27 < mmcgrath> skvidal: keep your pants on we're talking about logging right now, I'm just trying to do so in a way that doesn't discourage wzzrd, a new potential sysadmin member. 20:28 < mmcgrath> wzzrd: any luck with it? 20:28 < wzzrd> but i want to raise the matter of realtime parsing vs. cron-based once-a-day parsing, before i dive into something head over heels 20:28 < mmcgrath> I have very little experience with it but I know skvidal has used it as has Jeff_S and some others. 20:28 < mmcgrath> I don't think we need realtime parsing at this time. 20:28 < skvidal> wzzrd: realtime parsing and analysis are seldom done by the same tool 20:28 < mmcgrath> at least I don't think it would buys us much. 20:29 < wzzrd> epylog is nice, but i think it woulde require the logs from a group of servers going into *one* file on the loghost if you want a single report for that group 20:29 < skvidal> wzzrd: and realtime parsing is handy if there are specific triggers you know to look for - but only useful insofar as they can raise a warning in nagios 20:29 < mmcgrath> but yeah, I haven't heard anyone really argue for realtime so we can just assume non-realtime for the moment :) 20:29 < wzzrd> mmcgrath, skvidal: ok, no real-time 20:29 < wzzrd> just making sure 20:29 < mmcgrath> epylog can't look at multiple files? 20:29 < skvidal> mmcgrath: yes, it can - but it requires editingits configs 20:29 < skvidal> mmcgrath: remember what I was saying on the infrastructure list about mimicing the file structure of /var/log 20:30 < skvidal> mmcgrath: I wasn't making that up :) 20:30 < mmcgrath> skvidal: just edits though? not like some crazy bastadrization? 20:30 < skvidal> mmcgrath: significant edits 20:30 < mmcgrath> skvidal: I still have no idea what you're talking about with that? 20:30 < skvidal> look in /var/log on your laptop/desktop 20:30 < mmcgrath> you mean I should have a /var/log/messages that has all messages from all of our hosts going int o it? 20:30 < skvidal> no 20:30 < mmcgrath> so I should have a /var/log/hosts/bastion/messages? 20:30 < mmcgrath> I know what is in /var/log/ on my laptop 20:30 < skvidal> you understand there are certain files that commonly exist in /var/log 20:30 < skvidal> good 20:30 < mmcgrath> yeah? 20:31 < mmcgrath> like funcd 20:31 < mmcgrath> which won't exist on log1 20:31 < skvidal> and those files are expected to havecontent consistentwith a lot of log parsers 20:31 < wzzrd> skvidal: epylog.conf doesn't allow it afais, and I don't think it comes with a module that allows for the parsing of multiple files 20:31 < skvidal> funcd doesn't log via syslog in that way 20:31 < mmcgrath> isn't that creating just the opposite of what wzzrd is talking about though? 20:31 < skvidal> wzzrd: It really does. 20:31 -!- mether [n=Rahul(a)115.240.11.90] has joined #fedora-meeting 20:31 < skvidal> mmcgrath: no 20:31 < mmcgrath> you're talking about creating more files and he's needing less 20:31 < skvidal> no 20:31 < skvidal> I'm not 20:31 < skvidal> if y'all would let me explain 20:32 < skvidal> instead of peppering me with remarks 20:32 < skvidal> It might help 20:32 * wzzrd shuts up 20:32 < skvidal> we want logs per-host 20:32 < skvidal> but we also want logs per-by-service/group 20:32 < skvidal> so let's say all of the app servers belong to the appgroup 20:33 < skvidal> we can setup rsyslog so that if a log comes in from app01 (for example) that it gets sent to /var/log/hosts/app01/2010/01/14/ AND to /var/log/groups/app-servers/2010/01/14 20:33 < skvidal> then if we want to do log analysis for the appservers we tell epylog to look at /var/log/groups/app-servers/2010/01/14 20:34 < skvidal> if we want it to do analysis for a specific app server we tell it to look at: /var/log/hosts/app01/2010/01/14/ 20:34 < mmcgrath> epylog doesn't understand /var/log/hosts/app* ? 20:34 < skvidal> inside each of those dirs will be the syslog files normally generated by /var/log 20:35 < skvidal> mmcgrath: not when it is parsing log files - it expects the log files to be in the normal location relative to the base log path 20:35 < mmcgrath> I guess I'm not understanding how what is in my laptop in /var/log isn't what's in /var/log/hosts/app01/2010/01/14 20:35 < skvidal> okay 20:35 < skvidal> let's look at an example 20:36 < skvidal> login to bastion 20:36 < skvidal> cd /var/log 20:36 < skvidal> ls *log 20:36 < skvidal> ls *log 20:36 < skvidal> anaconda.log boot.log ha-log ldirectord.log sa-update.log yum.log 20:36 < skvidal> anaconda.syslog faillog lastlog maillog tallylog 20:37 < skvidal> the files syslog is writing are the only ones we can deal with for a remote logging server 20:37 -!- mbonnet_ [n=nmikeb@nat/redhat/x-mffdfhflbfhsdlsy] has quit "Getting off stoned server - dircproxy 1.2.0" 20:37 < skvidal> so messages, maillog, spooler, boot.log and cron 20:37 < skvidal> that's all we have access to 20:37 -!- ajax_ [n=ajackson@nat/redhat/x-dufagrlcirofytgc] has joined #fedora-meeting 20:37 < skvidal> now 20:37 -!- ldimagg__ [n=ldimaggi@nat/redhat/x-raflashvigxdpfbm] has joined #fedora-meeting 20:37 < skvidal> if you look on log1 20:37 -!- pjones [n=nnnnnnnn@fedora/pjones] has quit Read error: 54 (Connection reset by peer) 20:37 < skvidal> in 20:37 < skvidal> /var/log/hosts/bastion01/2010/01/14 20:37 < skvidal> for example 20:37 -!- halfline_ [n=rstrode@nat/redhat/x-orzrcxfjwbumvjdj] has joined #fedora-meeting 20:37 -!- ldimaggi___ [n=ldimaggi@nat/redhat/x-iylrghkpugcssyrv] has joined #fedora-meeting 20:37 < skvidal> cron.log kernel.log mail.log messages.log secure.log 20:37 < skvidal> you have all of those files 20:37 -!- ajax__ [n=ajackson@nat/redhat/x-ciqdvigcdiekghja] has joined #fedora-meeting 20:38 < skvidal> do you see how those files do not match the filenames and separation that are normally in /var/og? 20:38 -!- mbonnet_ [n=nnmikeb@nat/redhat/x-kjygxqitpvxutbom] has joined #fedora-meeting 20:38 < skvidal> ie mail.log vs maillog 20:38 < skvidal> kernel.log existing AT ALL 20:38 < skvidal> messages.log vs messages 20:38 < smooge> oi 20:38 < mmcgrath> yeah, so you're talking, mostly, about renaming the files? 20:38 < skvidal> that's what I mean by the difference 20:38 < mmcgrath> k 20:38 < skvidal> mmcgrath: and what content goes into them 20:39 < skvidal> ie: kernel.log shouldn 20:39 < skvidal> 't exist at all really 20:39 < skvidal> it's content should be in messages 20:39 < wzzrd> skvidal: crap you were right, i was looking in the wrong place... i didn't quite grasp how epylog's internals worked yet, i suppose... 20:39 < mmcgrath> k, I don't see any problem with that. 20:39 < skvidal> that's what I mean about fixing the structure of our remote logs 20:39 < skvidal> then once we do that 20:39 -!- pjones- [n=pjones@fedora/pjones] has joined #fedora-meeting 20:39 < skvidal> and we log by 'type of server/service' 20:39 < mmcgrath> wzzrd: k, you want to continue working and learning epylog? 20:40 < skvidal> then we can run generic log tools like epylog and generate lovely results 20:40 -!- halfline [n=rstrode@nat/redhat/x-bhhgxvexpbwoodqd] has quit Read error: 60 (Operation timed out) 20:40 < wzzrd> mmcgrath: sure, eager to help out 20:40 < skvidal> w/o having to beat our brains out modifying epylog to access logs we don't want 20:40 < mmcgrath> skvidal: so in theory the amount of logs we store is going to about double. Which do you think we should keep longer? the host level logs or the service level logs? 20:40 -!- pjones- is now known as pjones 20:40 < skvidal> mmcgrath: it's not going to double 20:40 < skvidal> right now we've made the mistake of doing *.* from syslog.conf on our logclients 20:41 < skvidal> instead of trimming the crap out 20:41 < skvidal> no one needs spooler.debug sent remotely 20:41 < mmcgrath> how do you know what to include and what not to? 20:41 < skvidal> years of experience? 20:41 < skvidal> :) 20:41 < skvidal> seriously- you keep warning and above 20:41 < skvidal> and drop a lot of the info and debug crap 20:41 < mmcgrath> skvidal: but still, lets say we only sent what warning and above right this second. 20:41 < skvidal> we can get rid of a lot of crap that's not helpful 20:41 < mmcgrath> when we start storing services too, we're storing all logs twice right? 20:42 < skvidal> <shrug> sure but it's just not that much content 20:42 < mmcgrath> skvidal: so, for example, we wouldn't be sending mail logs? 20:42 < skvidal> I think we should send maillogs 20:42 -!- lmacken_ [n=lmacken@nat/redhat/x-mwvftkmgxvqfjsea] has joined #fedora-meeting 20:42 < skvidal> unless you want to do mailog analysis ON the mailservers 20:42 < skvidal> which seems like a bad use of their cpu time 20:42 < mmcgrath> I've never setup a central logger that didn't store everything, skvidal do you happen to want to take lead on trimming that stuff down? 20:43 < dgilmore> i should just stop getting email and there would be alot less log data 20:43 < mmcgrath> <nod> I'd prefer to keep log analysis on the logger. 20:43 < mmcgrath> dgilmore: that's very true :) 20:43 < skvidal> mmcgrath: sure 20:43 < skvidal> mmcgrath: smooge you wanna work w/me on this? 20:43 < smooge> I am happy to. I love log analysis 20:43 < mmcgrath> skvidal: yup yup. 20:43 < skvidal> smooge: I'm glad I'm not alone :) 20:43 < mmcgrath> so on this same topic... there's still one thing I'd like to get converted. 20:44 < mmcgrath> most of our hosts are still not using rsyslog 20:44 < mmcgrath> I'd like to convert them to rsyslog, some have been but not most. 20:44 < smooge> mmcgrath, on my list of things to fix after ntpd 20:44 * mmcgrath is just mentioning that. 20:44 < skvidal> mmcgrath: for rhel5? 20:44 < mmcgrath> it could be as easy as yum install rsyslgo 20:44 < skvidal> did rhel5 switch to rsyslog? 20:44 < mmcgrath> skvidal: yeah 20:44 < skvidal> okie doke 20:44 < smooge> mmcgrath, skvidal it is basically 5 commands 20:44 < mmcgrath> skvidal: it does for fresh installs, but if you updated, it didn't do a replace. 20:44 < smooge> 1) yum install rsylog 20:44 < mmcgrath> as of 5.3 I think. 20:44 < skvidal> smooge: no problem - I just hadn't heard the switch was official in rhel5 20:45 < smooge> its an alternate 20:45 < smooge> syslogd is still prefered because of age 20:45 < smooge> oh wait.. I missed the new isntall part 20:46 < mmcgrath> smooge: is this something that's going to be possible / easy in puppet or are we goign to have to get func involved? 20:46 < smooge> func migth be easiest 20:46 * mmcgrath thinks perhaps he's been overthinking it. 20:46 < mmcgrath> smooge: what about new installs? 20:46 < mmcgrath> well, we can figure that after the meeting I guess :) 20:46 < smooge> yeah.. 20:47 < mmcgrath> wzzrd: any other questions on your side? 20:47 < mmcgrath> does anyone have any thoughts about exactly what we're looking for in these reports? 20:47 < wzzrd> well, im not sure whether this ok to ask 20:48 < mmcgrath> you can ask whatever you want. If it's for the root password though we probably won't answer. 20:48 < wzzrd> i think it would be easy to have some sort of mentor, 20:48 < wzzrd> you know, to ask some questions to 20:48 < skvidal> mmcgrath: for mail- I'm looking for errors and to make sure we don't have too much overrun/disk/cpu issues, for the rest of systems I'm looking to start getting a baseline on what 'normal' looks like and then fixing up problems 20:48 < skvidal> wzzrd: ask me 20:48 < wzzrd> skvidal: great, thanks! 20:48 < skvidal> I'm around often and I know a fair bit about the epylog code base 20:48 -!- ajax [n=ajackson@nat/redhat/x-doivlmfizynhhosw] has quit Read error: 60 (Operation timed out) 20:49 < skvidal> and I know the author of epylog personally and am willing to annoy him 20:49 < wzzrd> skvidal: you seem to be pretty well informed in this logging business :) 20:49 -!- tomspur [n=Tom(a)p54A54B74.dip.t-dialin.net] has joined #fedora-meeting 20:49 < wzzrd> lol 20:49 < mmcgrath> wzzrd: yeah for epylog ask skvidal I have no experience with it. If you have questions about fedora or how we're doing something just ask anyone in #fedora-admin, skvidal smooge and I are almost always in there. 20:49 < wzzrd> ok appreciate it 20:49 -!- ldimaggi_ [n=ldimaggi@nat/redhat/x-gjpbecmbtnvijmjl] has quit Read error: 60 (Operation timed out) 20:49 < dgilmore> wzzrd: skvidal, mmcgrath, myself, ricky, smooge and others will be more than happy to answer questions 20:50 < mmcgrath> Ok, anyone hav any other questions on logging? 20:50 < smooge> not me 20:50 < mmcgrath> ok, with that I'll open the floor for anything and everything 20:51 < mmcgrath> #topic Infrastructuer -- Open FLoor 20:51 -!- zodbot changed the topic of #fedora-meeting to: Infrastructuer -- Open FLoor (Meeting topic: Infrastructure) 20:51 < mmcgrath> jds2001: you around? 20:51 < mmcgrath> our newest -main member has been pretty quiet. 20:51 < dgilmore> mmcgrath: i wore him out on Saturday 20:51 < mmcgrath> hehe 20:51 < mmcgrath> oh! i know one thing. 20:51 < skvidal> dgilmore: umm - that sounds 20:51 < skvidal> umm 20:51 < skvidal> wrong 20:52 < mmcgrath> I'm still in the process of getting our secondary,alt,archive stuff to download.fedora.redhat.com 20:52 < dgilmore> skvidal: sure. 20:52 < skvidal> :) 20:52 < dgilmore> skvidal: i made him work alot on saturday while migrating the lists 20:52 < dgilmore> ;) 20:52 < mmcgrath> After the move we have root squashed for all /pub content (which is good) 20:52 < dgilmore> that better 20:52 < mmcgrath> but now I can't do wnything with my dirs that are there becaues they're root owned. 20:53 < ricky> Heh, yow :-) 20:53 < dgilmore> opps 20:53 < mmcgrath> yeah 20:53 -!- lmacken [n=lmacken@fedora/lmacken] has quit Read error: 113 (No route to host) 20:53 < mmcgrath> I do still have several concerns. 20:53 < mmcgrath> same as smooge 20:53 < mmcgrath> will the disks be able to keep up? 20:53 < mmcgrath> will the snap-mirror process work correctly with the new load? 20:54 < mmcgrath> Ok, anyone have anything else? 20:54 < mmcgrath> If not we'll close the meeting in 30 20:54 -!- ajax_ [n=ajackson@nat/redhat/x-dufagrlcirofytgc] has quit Connection timed out 20:54 < smooge> concerns? I have no concerns.. I have reality 20:54 < smooge> cold clear that it probably wont keep up 20:54 < mmcgrath> smooge: yeah, I have a feeling we're in for a ride here. 20:54 -!- ldimagg__ [n=ldimaggi@nat/redhat/x-raflashvigxdpfbm] has quit Read error: 110 (Connection timed out) 20:55 < mmcgrath> but who knows, we might get surprised :) 20:55 < nirik> mmcgrath: will this change what I have to do to push the nightly live composes? 20:55 < smooge> yeah.. I am buying lotto tickets just in case 20:55 < mmcgrath> nirik: it'll just change were you write to 20:55 < mmcgrath> nirik: hopefully for the better because it'll be all mirrored and stuff 20:56 < nirik> cool. 20:56 < mmcgrath> ok, with that! 20:56 < mmcgrath> #meetingend 20:56 < mmcgrath> #endmeeting 20:56 -!- zodbot changed the topic of #fedora-meeting to: Channel is used by various Fedora groups and committees for their regular meetings | Note that meetings often get logged | For questions about using Fedora please ask in #fedora | See http://fedoraproject.org/wiki/Meeting_channel for meeting schedule 20:56 < zodbot> Meeting ended Thu Jan 14 20:56:53 2010 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . 20:56 < zodbot> Minutes: http://meetbot.fedoraproject.org/fedora-meeting/2010-01-14/fedora-meeting.2… 20:56 < zodbot> Minutes (text): http://meetbot.fedoraproject.org/fedora-meeting/2010-01-14/fedora-meeting.2… 20:56 < mmcgrath> damnit 20:56 < zodbot> Log: http://meetbot.fedoraproject.org/fedora-meeting/2010-01-14/fedora-meeting.2… 20:56 < mmcgrath> Thanks for coming everyone!

1 0

Old IP names
by Stephen John Smoogen 13 Jan '10

13 Jan '10

I was looking at IPs and found that rawhide1.fedoraproject.org and smtp-mm2.fedoraproject.org share the same IP address. Is this ok? -- Stephen J Smoogen. Ah, but a man's reach should exceed his grasp. Or what's a heaven for? -- Robert Browning

2 1

Log management
by Maxim Burgerhout 13 Jan '10

13 Jan '10

After last weeks meeting, Mike asked me to look into a solution for log management, reporting and alerting. As the new guy, I happily jumped in and so here's a short status report on what I found and what some of the options are. I would appreciate any input. First of all, there's a lot of different projects trying to deliver this functionality. A big part of them though, are either dead, commercial and / or mainly aimed at parsing Apache logfiles. I am assuming we want an open source tool, so, skipping the commercial products like Splunk and skipping the dead projects, it boils down to these as most promising. If I turn out to be missing some great project, please tell me and I'll look into it. 1. octopussy (http://www.8pussy.org/dokuwiki/doku.php) Log aggregation tool, providing a one stop web UI to view all logs for a bunch of servers, complete with alerting to email, jabber or nagios, creation of graphs etc. Imports data into a MySQL database. The downsides are that it's developer base is pretty narrow, it is relatively complex to configure and it is Debian centric in both documentation and available packaging. On the other hand, octopussy is what comes closest to Splunk in the open source world that I know of. Sadly, all other Splunk-like apps are commercial. Last release: December 2009, last commit a couple of days ago. 2. logreport / lire (http://www.logreport.org/) Log parser that runs from cron or manually; parses logs from many different applications and generates html, txt or pdf that can optionally be mailed to people. Slightly odd curses configuration frontend. Works by importing log files into what is called the 'dlf store' and renders periodic reports from that data. Can receive logfiles over mail. Downside is that development is pretty slow, even if it is backed by a foundation (about one release per year; last one was in March 2009). Low mailinglist traffic. Does not do alerting and real time parsing. DLF has sqlite as a backend, afaict, and I'm not sure how well that scales in the long run. Last release: March 2009, no activity on mailinglist and CVS since then. 3. epylog (https://fedorahosted.org/epylog/) Has some fans in the Infra group already, it seems. Modular log parser, run from cron. Stores offsets for already parsed files, to the next run can start where the previous one left off. Generates reports in html in a configurable location or sends them out per mail. Custom report delivery methods can be configured. Does not do alerting and real time parsing. Hosted on Fedorahosted, written in Python (the other two apps are mainly in Perl). Very easy to write custom modules for. Last release: none recently, but subversion is active. Of the projects, only epylog is packaged for Fedora already and Octopussy is the only one that does realtime log parsing and alerting. What should be the next step in this? I probably need to do some more testing, but before I do that, let me know what you think is important in a log monitoring and reporting application? Any specific tests you would like to see done for these apps? Let me know what you think. Maxim Burgerhout maxim(a)wzzrd.com ---------------- GPG Fingerprint EB11 5E56 E648 9D99 E8EF 05FB C513 6FD4 1302 B48A

5 6

Jon Stanley
by Mike McGrath 12 Jan '10

12 Jan '10

I am happy to announce that Jon Stanley, proud cubs fan[1], has been promoted to sysadmin-main. He's been working with Fedora for years and he's been working in several infrastructure groups for a long time. Most recently of which was the mail list migration. So welcome to syadmin-main Jon! -Mike [1] I actually think he's a cardinals fan, I could be wrong. No, I think I am wrong, yeah he's a cardinals fan :)

4 3

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure January 2010