Last week when we were talking about spawning rdiff-backup to backup our systems, we diverged into discussing app/apache logs and the somewhat complicated system we currently have for grabbing those logs.
Right now we have a list of hosts on log02 that it should grab logs from. Those hosts need to have rsyncd running on them to allow access from log02 to fetch the /var/log/httpd/ path from them.
That requires 2 things to be coupled and it is a bit awkward if you set up a host that is tricky to access from log02 or isn't on the vpn.
In general I also am not in love with having to have rsyncd listening on systems - even if it is ip-restricted.
So the thought was we could do something like this on log02:
1. setup an ssh key on log02 that can run rsync to /var/log/httpd on all hosts 2. make any host that needs to have its logs retrieved be marked in the ansible inventory host/group vars 3. git clone public-ansible-repo onto log02 4. use group_by to construct a group of the hosts which can then be retrieved using rsync.
The sole reason for using ansible here is so we can keep the log sync info in our inventory and to parallelize the retrieval of logs.
This is more or less identical to what we talked about for backups using rdiff-backup.
When we were discussing this Luke mentioned then using tbgrep(https://pypi.python.org/pypi/tbgrep) to search the resulting files and compile a set of tracebacks our apps are dumping out.
If we have all the logs on log02 generating a report like this would be pleasantly kept away from the rest of our hosts and could give us reasonably useful reports of brokenness.
I'd love some feed back on if this is all crazy or not :) -sv
On Tue, 25 Jun 2013 15:16:04 -0400 seth vidal skvidal@fedoraproject.org wrote:
Last week when we were talking about spawning rdiff-backup to backup our systems, we diverged into discussing app/apache logs and the somewhat complicated system we currently have for grabbing those logs.
Right now we have a list of hosts on log02 that it should grab logs from. Those hosts need to have rsyncd running on them to allow access from log02 to fetch the /var/log/httpd/ path from them.
That requires 2 things to be coupled and it is a bit awkward if you set up a host that is tricky to access from log02 or isn't on the vpn.
In general I also am not in love with having to have rsyncd listening on systems - even if it is ip-restricted.
So the thought was we could do something like this on log02:
- setup an ssh key on log02 that can run rsync to /var/log/httpd on
all hosts 2. make any host that needs to have its logs retrieved be marked in the ansible inventory host/group vars 3. git clone public-ansible-repo onto log02 4. use group_by to construct a group of the hosts which can then be retrieved using rsync.
The sole reason for using ansible here is so we can keep the log sync info in our inventory and to parallelize the retrieval of logs.
This is more or less identical to what we talked about for backups using rdiff-backup.
Yeah, I think it's worth trying out... we may need to wait until we have more stuff moved over to ansible just to try and minimize confusion.
When we were discussing this Luke mentioned then using tbgrep(https://pypi.python.org/pypi/tbgrep) to search the resulting files and compile a set of tracebacks our apps are dumping out.
Sounds good to me. We might also look at any stats we want to pull out or other errors we want to note. We could do these similar to the epylog runs? generate some kind of report link and mail it to interested parties?
If we have all the logs on log02 generating a report like this would be pleasantly kept away from the rest of our hosts and could give us reasonably useful reports of brokenness.
Yeah, I agree...
I'd love some feed back on if this is all crazy or not :)
I don't think it's crazy at all.
kevin
On 25 June 2013 13:16, seth vidal skvidal@fedoraproject.org wrote:
Last week when we were talking about spawning rdiff-backup to backup our systems, we diverged into discussing app/apache logs and the somewhat complicated system we currently have for grabbing those logs.
Right now we have a list of hosts on log02 that it should grab logs from. Those hosts need to have rsyncd running on them to allow access from log02 to fetch the /var/log/httpd/ path from them.
That requires 2 things to be coupled and it is a bit awkward if you set up a host that is tricky to access from log02 or isn't on the vpn.
In general I also am not in love with having to have rsyncd listening on systems - even if it is ip-restricted.
So the thought was we could do something like this on log02:
- setup an ssh key on log02 that can run rsync to /var/log/httpd on
all hosts 2. make any host that needs to have its logs retrieved be marked in the ansible inventory host/group vars 3. git clone public-ansible-repo onto log02 4. use group_by to construct a group of the hosts which can then be retrieved using rsync.
The sole reason for using ansible here is so we can keep the log sync info in our inventory and to parallelize the retrieval of logs.
This is more or less identical to what we talked about for backups using rdiff-backup.
My question is will a person who is on log02 be able to ssh into every rsyncable host as root like they can do so from lockbox. or will we be using a sub-user who can be ssh'd from log02 to get the log files? I am just wanting to keep the number of systems we need to really worry about to a minimum so we aren't ending up with whackamole later.
On Thu, 27 Jun 2013 13:12:49 -0600 Stephen John Smoogen smooge@gmail.com wrote:
On 25 June 2013 13:16, seth vidal skvidal@fedoraproject.org wrote:
Last week when we were talking about spawning rdiff-backup to backup our systems, we diverged into discussing app/apache logs and the somewhat complicated system we currently have for grabbing those logs.
Right now we have a list of hosts on log02 that it should grab logs from. Those hosts need to have rsyncd running on them to allow access from log02 to fetch the /var/log/httpd/ path from them.
That requires 2 things to be coupled and it is a bit awkward if you set up a host that is tricky to access from log02 or isn't on the vpn.
In general I also am not in love with having to have rsyncd listening on systems - even if it is ip-restricted.
So the thought was we could do something like this on log02:
- setup an ssh key on log02 that can run rsync to /var/log/httpd on
all hosts 2. make any host that needs to have its logs retrieved be marked in the ansible inventory host/group vars 3. git clone public-ansible-repo onto log02 4. use group_by to construct a group of the hosts which can then be retrieved using rsync.
The sole reason for using ansible here is so we can keep the log sync info in our inventory and to parallelize the retrieval of logs.
This is more or less identical to what we talked about for backups using rdiff-backup.
My question is will a person who is on log02 be able to ssh into every rsyncable host as root like they can do so from lockbox. or will we be using a sub-user who can be ssh'd from log02 to get the log files? I am just wanting to keep the number of systems we need to really worry about to a minimum so we aren't ending up with whackamole later.
1. we could do a separate user - we just have to make sure /var/log/httpd stays 'open' to that user - which is actually quite tricky in the face of apache updated rpms
2. we could also just keep using rsync - but over ssh and restrict that particular ssh key to only running rsync and only from one path.
-sv
infrastructure@lists.fedoraproject.org