lumberjack-developers September 2012

lumberjack-developers@lists.fedorahosted.org

4 participants
4 discussions

Announcing ceelog-0.1 - comments wanted
by Miloslav Trmač 05 Oct '12

05 Oct '12

Hello, I have just published a 0.1 of ceelog/libceelog at https://fedorahosted.org/ceelog/ : Tarball at https://fedorahosted.org/releases/c/e/ceelog/ceelog-0.1.tar.xz git http://git.fedorahosted.org/cgit/ceelog.git/. A full README is appended below. Comments (on project name, API design, command-line tool wishlist), testing, review, contributions of any kind are very welcome - on the list, in trac, to me privately, in any way you find convenient. Mirek About ===== The ceelog project provides libceelog, a library for receiving, filtering and searching a stream or log of CEE/Lumberjack syslog records, and an associated command-line tool, named ceelog. The goal is to abstract the user from the backend storage (files, some kind of local indexed storage, a remote database) and to provide efficient log processing tools that can be used in applications and scripts for automated log processing. The project's home page is at https://fedorahosted.org/ceelog/ . To get you started ================== The ceelog(1) tool reads the "default" event source and outputs events matching a filter. Currently, the "default" event source is hardcoded to /var/log/messages. Example filters: * A regexp (matches the unstructured event text, or the "msg" field for CEE/Lumberjack structured events) ceelog '/DHCP/' * A field comparison (matches a CEE/Lumberjack field) ceelog 'uid == 0' ceelog 'uid != 0' ceelog 'trusted!uid == 0' ceelog 'username ~ /^guest-/' ceelog 'username !~ /^guest-/' * A combination of the above ceelog 'trusted!uid == 0 && username ~ /^guest-/' See the source code in src/ceelog.c for an example of a subset of the API. Roadmap ======= * Add "Live log file" input source that can handle messages being appended and log rotation. * Document the filter expression format. * Document ceelog(1). * Get as close to 100% test coverage as possible. * Support best-effort saving/restoring the current position in a source. * Add better support for JSON types. * Implement MongoDB input source. * Support searching directly in the input source (e.g. to evaluate the filter server-side). * ceelog(1) improvements: - Input processing (e.g. output only the last N recent events, block for more incoming events) - Output formatting (e.g. only output some structured fields) - Statistics/table output (group matching events by one field, output counts) Bugs ==== Please consider reporting the bug to your distribution's bug tracking system. Otherwise, please report bugs at https://fedorahosted.org/volume_key/ . Bug reports with patches are especially welcome.

3 3

@cee and MSG content
by Rainer Gerhards 12 Sep '12

12 Sep '12

Hi all, the CEE spec from early this year says that a syslog message with @cee MUST contain JSON content only, and not a message. Take for example this syslog payload: @cee:{"f":"1"} some text As of the spec, this is not CEE-enhanced syslog and as such NO JSON object is to be generated. Instead, this would need to be encoded as @cee:{"f:":"1", "msg":"some text"} To create a proper CEE-enhanced message. I wonder what our position in lumberjack is in regard to this. My impression is that the "invalid" format (the first sample) will be seen in practice. Comments? Rainer

2 1

should trusted properties be in a subobject?
by David Lang 11 Sep '12

11 Sep '12

should the trusted properties be grouped into a subobject (i.e. trusted!pid, trusted!uid, etc)? I think that the answer is yes for the following reasons 1. there may be multiple PID objects. For examplthe syslogtag includes a PID. It's useful to have both what the application claims and what the system claims and be able to compare the two. 2. when relaying from one machine to another, the trusted properties need to be handled specially depending on the degree of trust that the transport protocol provides (and that the sysadmin places on that trust) 3. there needs to be some protection to prevent people from generating logs that have the trusted properties in them. This is easier if these are in one object and you can protect that object than if they are a bunch of individual properties that have to be protected seperately. 4. If there is a trusted subobject, it's easier to extend. For example, when logs are being relayed from one system to another, the recieving system can add items into the trusted subobject to provide information about the connection. For example, it could add certificate information or the remote IP address of the sending machine. I would define this trusted object as something like: Fields in the Trusted object should be generated by the software recieving the log messages and MUST not be able to be affected by anything that the application generating the log puts into the log message. Applications recieving relayed log messages SHOULD update the Trusted object to identify if the data should still be trusted. Applications recieving relayed log messages MAY update the Trusted object with additional information indicating why the fields should still be trusted. Thoughts? David Lang

3 3

lumberjack schema stability
by Rainer Gerhards 10 Sep '12

10 Sep '12

Hi all, do we consider the current lumberjack XML schema as published on the web site as stable? If so, I think it would make sense to version it and place a "stable" tag onto it (gives implementers peace of mind, at least I think so). With it being officially stable, we can begin to check interop. With versioning, we can catch up with CEE changes. Comments are appreciated. Thanks, Rainer

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

lumberjack-developers September 2012