lumberjack-developers April 2012

lumberjack-developers@lists.fedorahosted.org

11 participants
13 discussions

New Syntax Spec and updated Field list
by William Heinbockel 03 May '12

03 May '12

Folks, I'm prepping the CEE 1.0beta release and as part of that, I've tried to summarize some of the discussions surrounding the field naming and syntaxes. The plan for CEE to basically adopt this Lumberjack syntax. An update field list, based on David's suggestions and borrowing from Syslog & libumberlog, is posted here: https://fedorahosted.org/lumberjack/wiki/FieldList#UnifiedFieldList The Syntax spec is mostly complete, but is still a bit rough around the edges: https://fedorahosted.org/lumberjack/wiki/SyntaxFormats What I did was take the JSON field types and add datetime, ipv4, and ipv6 types. Then made a basic conversion from JSON --> XML and tried to align the libumberlog syntaxes with Keith's XML. I also added some notes about the "structured" vs. "inline" field naming. Basically, most of this should line up pretty well with the discussions we've been having. The only real delta is the difference between Keith's XML and the proposed syntax -- not everything aligns, but it is fairly close. Keith: I will contact you directly with my schema proposals.

7 17

Syslog priority & facility
by William Heinbockel 27 Apr '12

27 Apr '12

What are people's opinions on the Syslog priority and facility being supported in the field list? I see that they are currently in libumberlog and am not sure that they should be there. As used, the *priority* and *facility* have no real meaning outside of the Syslog protocol. Priority has some relation to severity, so I can of understand supporting one/both of those -- though I think a numeric value instead of a string might make more sense. Facility however seems very specific to Syslog and we might want to think of a way to "modernize" the concept when putting the information into structured data (if the concept belongs there at all) After all, when you encode the event data into structured text, the priority/facility are still present in the Syslog header. Carrying them over into the event data as is makes little sense being that they are specific to the Syslog protocol. Thoughts?

4 3

Re: [lumberjack] [ANN] libumberlog 0.2.0 released!
by Rainer Gerhards 20 Apr '12

20 Apr '12

Thx for all your hard work and congrats for what has been achieved already :-) Rainer Peter Czanik <czanik(a)balabit.hu> hat geschrieben:On 04/16/2012 09:19 PM, Gergely Nagy wrote: > Hi! > > I'll skip the formal release announcement this time, as the new stuff > the 0.2.0 release brings are fairly small: there was another release > inbetween, 0.1.2, which fixed a silly mistake I made by not copying > va_lists. > > The 0.2.0 release drops the json-c dependency, all the library needs now > is libc. This has multiple advantages, ranging from not depending on > anything outside of /lib on a default install, through noticably faster > speed, to a more compact JSON payload - one with no unnecessary > whitespace. It is now available in FreeBSD ports: http://www.freshports.org/sysutils/libumberlog/ But as it does not have ld.so.preload, and even LD_PRELOAD has more limitations, it's not that much useful, as on Linux. The good news is, that it is going to be fixed (loader & libmap.conf extended). Libumberlog is the perfect use case to add this functionality to FreeBSD :-) It is also part of devel:libraries:c_c++ development project on openSUSE, which is the stepping stone to the distribution. I still need to fix some packaging bugs before submitting it to openSUSE factory. Not yet tested on Fedora, but as it's a very simple spec file, it should also work there with minor modifications. The one at https://build.opensuse.org/package/show?package=libumberlog&project=devel%3… still installs to /usr/lib, which is problematic, if used in /etc/ld.so.preload and /usr is a separate partition. The one in https://build.opensuse.org/package/show?package=libumberlog&project=home%3A… installs to /lib, but I still need to fix the 64bit package. Once it's fixed, I can push to devel:libraries:c_c++ and to openSUSE factory. Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ _______________________________________________ lumberjack-developers mailing list lumberjack-developers(a)lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers

3 2

[ANN] libumberlog 0.2.0 released!
by Gergely Nagy 20 Apr '12

20 Apr '12

Hi! I'll skip the formal release announcement this time, as the new stuff the 0.2.0 release brings are fairly small: there was another release inbetween, 0.1.2, which fixed a silly mistake I made by not copying va_lists. The 0.2.0 release drops the json-c dependency, all the library needs now is libc. This has multiple advantages, ranging from not depending on anything outside of /lib on a default install, through noticably faster speed, to a more compact JSON payload - one with no unnecessary whitespace. Functionality remains the same, but the dependencies are lighter, and the output tighter. For the test suite though, one still needs json-c, but it's completely optional, and not required for the runtime library. As usual, it is available on github, both via git[1], and as a tarball[2]. [1]: git://github.com/algernon/libumberlog.git [2]: https://github.com/downloads/algernon/libumberlog/libumberlog-0.2.0.tar.gz -- |8]

2 1

libumberlog openSUSE package
by Peter Czanik 11 Apr '12

11 Apr '12

Hello, I have now libumberlog packaged for openSUSE/SLES (tested only on openSUSE 12.1). It is available at https://build.opensuse.org/package/show?package=libumberlog&project=home%3A… For SLES and 11.4 one also needs to install json-c from my repository, as these are not available in those distributions. It could probably also used as a basis for a Fedora package. There is also a FreeBSD port in the works, most likely published next week. Or if interested, I can send my current version by e-mail. Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1

Re: [lumberjack] field names and taxonomy
by William Heinbockel 04 Apr '12

04 Apr '12

I just updated the Lumberjack Field List on the wiki to start unifying the field names based on the nxlog, libumberlog, and some of the auditd fields. I tried to keep the list minimal and have at least enough support for syslog compatibility. <https://fedorahosted.org/lumberjack/wiki/FieldList> > On Mon, Apr 2, 2012 at 11:51 AM, Jan Zelený <jzeleny(a)redhat.com> wrote: >>> >> What we should figure out is whether it is better to prefer one field >>> >> that might have a range of values depending on the producer, or we want >>> >> to separate these across multiple fields. >>> > >>> > I thought the whole point of defining common set of names is to unify >>> > what is produced by producers, isn't it? If yes then I don't think we >>> > want a range of multiple fields. >>> >>> Correct. >>> >>> But there are two ways to approach the problem, either have fields be >>> unique to the concept or unique to the (concept, value type). >> >> Ah, ok. Thanks for clarification >> >>> For example, I could argue to have something like >>> >>> { "dst":["10.10.0.2","localhost.localdomain"] } >>> >>> instead of >>> >>> { "dst_ipv4":"10.10.0.2", "dst_host":"localhost.localdomain" } >>> >>> While the latter option might be more intuitive, the former is more >>> flexible. This loops back to practice of whether the datatype should >>> be associated with an array or the array values. >>> >>> >From an end-user standpoint, I would push for the latter >> >> I agree with the latter approach. Among other things, it will probably be >> easier to read and less error-prone when it comes to optional values. (for >> example only hostname being set for dst) >> >> Jan

2 7

Field naming structure
by William Heinbockel 03 Apr '12

03 Apr '12

Before we dive too deep into which fields to support and how to name them, we should first discuss the structure and organization of the names. I propose we roughly follow the proposal from Balazs: 1. Core Fields -- flat names that are defined by lumberjack/CEE. These should encompass the minimal set of fields that are commonly used. 2. Extended Fields -- additional fields are either tied to a specific product/application or a functional product group (anti-virus, firewalls, etc.) We should focus on scoping and defining #1, probably combining the libumberlog, patterndb, liblognorm, RFC5424/IANA fields. Maybe the best place to start is with the `auditd` list <http://people.redhat.com/sgrubb/audit/audit-parse.txt> and abstract linux-specific names to be more generally applicable. For #2, we need to figure out how these work. I vote that the structure should infer the namespace and the namespace should either be unique to the vendor/product or developed by the community. {"red_hat":{"auditd":{"time":"..."}}} would work, or collapsed to {"red_hat|auditd|time":"..."} (feel free to substitute divisive char: !/.:|) The functional product categories should be related to an organizational entity -- lumberjack, cee, IANA, etc. We need some way to distinguish vendor/product namespaces from organization/functional ones.

4 12

Re: [lumberjack] [ANN] libumberlog 0.1.1 released!
by Rainer Gerhards 03 Apr '12

03 Apr '12

Congrats, very useful :-) Rainer Gergely Nagy <algernon(a)balabit.hu> hat geschrieben:--------------------------------------------------------------- PACKAGE : libumberlog VERSION : 0.1.1 SUMMARY : First (public) stable release DATE : 2012 April 2 HOMEPAGE : http://algernon.github.com/libumberlog/ --------------------------------------------------------------- DESCRIPTION: The libumberlog library is a a thin layer over the traditional syslog() function that turns your legacy syslog messages into structured logs, so that the message part becomes a JSON string (see the example below), with a couple of extra fields added by default: a high-precision timestamp, uid, gid, and pid, to name a few. The library can be used as a system-wide LD_PRELOAD-ed library, turning every syslog() message into the improved format, or it can be LD_PRELOAD-ed on a case-by-case basis, or even linked to, in which case it provides a few other functions, mentioned below. EXAMPLE: Mar 24 12:01:34 localhost sshd[12590]: @cee:{ "msg": "Accepted publickey for algernon from 127.0.0.1 port 55519 ssh2", "pid": "12590", "facility": "auth", "priority": "info", "program": "sshd", "uid": "0", "gid": "0", "host": "hadhodrond", "timestamp": "2012-03-24T12:01:34.236987887+0100" } FEATURES: The library supports LD_PRELOAD (and has been used in such capacity on production systems), turning legacy messages into something little bit more structured, yet, backwards compatible. It also introduces a few new functions, namely ul_format() and ul_syslog(), that allow one to make use of structured logging even better, by allowing them to include arbitrary key-value pairs in the message. Credits ======= The library has been written by Gergely Nagy <algernon(a)balabit.hu>, with invaluable feedback from members of Project Lumberjack, an open-source initiative to update and enhance the event log architecture. DOWNLOADS: The source is available from the git repository at github: git://github.com/algernon/libumberlog.git A source tarball has also been released, and is available via github aswell, at: https://github.com/downloads/algernon/libumberlog/libumberlog-0.1.1.tar.gz Documentation and more information about the library is available on its homepage at http://algernon.github.com/libumberlog/. For more information about Project Lumberjack, please see their homepage at https://fedorahosted.org/lumberjack/, and the mailing list at https://fedorahosted.org/mailman/listinfo/lumberjack-developers -- |8] _______________________________________________ lumberjack-developers mailing list lumberjack-developers(a)lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers

4 4

[ANN] libumberlog 0.1.1 released!
by Gergely Nagy 02 Apr '12

02 Apr '12

--------------------------------------------------------------- PACKAGE : libumberlog VERSION : 0.1.1 SUMMARY : First (public) stable release DATE : 2012 April 2 HOMEPAGE : http://algernon.github.com/libumberlog/ --------------------------------------------------------------- DESCRIPTION: The libumberlog library is a a thin layer over the traditional syslog() function that turns your legacy syslog messages into structured logs, so that the message part becomes a JSON string (see the example below), with a couple of extra fields added by default: a high-precision timestamp, uid, gid, and pid, to name a few. The library can be used as a system-wide LD_PRELOAD-ed library, turning every syslog() message into the improved format, or it can be LD_PRELOAD-ed on a case-by-case basis, or even linked to, in which case it provides a few other functions, mentioned below. EXAMPLE: Mar 24 12:01:34 localhost sshd[12590]: @cee:{ "msg": "Accepted publickey for algernon from 127.0.0.1 port 55519 ssh2", "pid": "12590", "facility": "auth", "priority": "info", "program": "sshd", "uid": "0", "gid": "0", "host": "hadhodrond", "timestamp": "2012-03-24T12:01:34.236987887+0100" } FEATURES: The library supports LD_PRELOAD (and has been used in such capacity on production systems), turning legacy messages into something little bit more structured, yet, backwards compatible. It also introduces a few new functions, namely ul_format() and ul_syslog(), that allow one to make use of structured logging even better, by allowing them to include arbitrary key-value pairs in the message. Credits ======= The library has been written by Gergely Nagy <algernon(a)balabit.hu>, with invaluable feedback from members of Project Lumberjack, an open-source initiative to update and enhance the event log architecture. DOWNLOADS: The source is available from the git repository at github: git://github.com/algernon/libumberlog.git A source tarball has also been released, and is available via github aswell, at: https://github.com/downloads/algernon/libumberlog/libumberlog-0.1.1.tar.gz Documentation and more information about the library is available on its homepage at http://algernon.github.com/libumberlog/. For more information about Project Lumberjack, please see their homepage at https://fedorahosted.org/lumberjack/, and the mailing list at https://fedorahosted.org/mailman/listinfo/lumberjack-developers -- |8]

2 1

libumberlog packaged for archlinux
by Brian Knox 02 Apr '12

02 Apr '12

For anyone who might use archlinux for development work - just a quick heads up I tossed a PKGBUILD up on the arch user repo that will grab libumberlog from git and package it: https://aur.archlinux.org/packages.php?ID=58140 Brian

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

lumberjack-developers April 2012