= Proposed Self Contained Change: Remote Journal Logging = https://fedoraproject.org/wiki/Changes/Remote_Journal_Logging
Change owner(s): Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl
Systemd journal can be configured to forward events to a remote server. Entries are forwarded including full metadata, and are stored in normal journal files, identically to locally generated logs.
== Detailed Description == Systemd's journal currently provides a replacement for most functionality offered by traditional syslog daemons, with two notable exceptions: arbitrary filtering of messages and forwarding of messages over the network. This Change targets the latter.
The high-level goal is to have a mechanism where journal logging can be extended to keep a copy of logs on a remote server, without requiring any maintenance, done fairly efficiently and in a secure way.
Two new daemons are added as part of the systemd package: * on the receiver side systemd-journal-remote accepts messages in the Journal Export Format [1]. The export format is a simple serialization of journal entries, supporting both text and binary fields. This means that the messages are transferred intact, apart from the "cursors", which specify the location in the journal file. Received entries are stored in local journal files underneath /var/log/journal. Those files are subject to normal journald rules for rotation, and the older ones will be removed as necessary to stay within disk usage limits. Once entries have been written to the journal file, they can be read using journalctl and the journal APIs, and are available to all clients, e.g. Gnome Logs [2].
* on the sender side systemd-journal-upload is a journal client, which exports all available journal messages and uploads them over the network. The (local) cursor of the last message successfully forwarded is stored on disk, so when systemd-journal-upload is restarted (possibly after a reboot of the machine), it will send all recent messages found in the journal and then new ones as they arrive.
The communication between the two daemons is done over standard HTTPS, following rather simple rules, so it is possible to create alternate implementations without much work. For example, curl can be easily used to upload journal entries from a text file containing entries in the export format. Basically, the data are sent in an HTTP POST to /upload with Content- Type: application/vnd.fdo.journal. When doing "live" forwarding, the size of the transfer cannot be known in advance, so Transfer-Encoding: chunked is used. All communication is encrypted, and the identity of both sides is verified by checking for appropriate signatures on the certificates.
== Scope == * Proposal owners: The code in systemd for systemd-journal-remote and systemd-journal-upload will have to be added. The first is done, and has already been released in the latest Rawhide systemd package. The second is mostly done, and will be submitted upstream soon. Necessary units will have to be created, and a location with suitable permissions will have to be created so that systemd- journal-remote can run unprivileged. This means that systemd-journal-remote should probably be packaged as a separate subpackage, similarly to systemd- journal-gatewayd. systemd-journal-upload uses libcurl, and thus should be also packaged as a separate subpackage to avoid introducing a new dependency for systemd.
Suitable defaults for timeouts for transfers and maximum accepted entry sizes have to be chosen.
A port will have to be picked: systemd-journal-gatewayd uses 19531, so systemd-journal-remote should probably use 19532.
Two new users will have to be created when the packages are installed.
* Other developers: N/A (not a System Wide Change) * Release engineering: N/A (not a System Wide Change) * Policies and guidelines: N/A (not a System Wide Change)
[1] http://www.freedesktop.org/wiki/Software/systemd/export/ [2] https://wiki.gnome.org/Apps/Logs _______________________________________________ devel-announce mailing list devel-announce@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel-announce
On Mon, 14 Apr 2014, Jaroslav Reznik wrote:
= Proposed Self Contained Change: Remote Journal Logging = https://fedoraproject.org/wiki/Changes/Remote_Journal_Logging
Change owner(s): Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl
Systemd journal can be configured to forward events to a remote server. Entries are forwarded including full metadata, and are stored in normal journal files, identically to locally generated logs.
== Detailed Description == Systemd's journal currently provides a replacement for most functionality offered by traditional syslog daemons, with two notable exceptions: arbitrary filtering of messages and forwarding of messages over the network. This Change targets the latter.
The high-level goal is to have a mechanism where journal logging can be extended to keep a copy of logs on a remote server, without requiring any maintenance, done fairly efficiently and in a secure way.
Two new daemons are added as part of the systemd package:
- on the receiver side systemd-journal-remote accepts messages in the Journal
Export Format [1]. The export format is a simple serialization of journal entries, supporting both text and binary fields. This means that the messages are transferred intact, apart from the "cursors", which specify the location in the journal file. Received entries are stored in local journal files underneath /var/log/journal. Those files are subject to normal journald rules for rotation, and the older ones will be removed as necessary to stay within disk usage limits. Once entries have been written to the journal file, they can be read using journalctl and the journal APIs, and are available to all clients, e.g. Gnome Logs [2].
- on the sender side systemd-journal-upload is a journal client, which exports
all available journal messages and uploads them over the network. The (local) cursor of the last message successfully forwarded is stored on disk, so when systemd-journal-upload is restarted (possibly after a reboot of the machine), it will send all recent messages found in the journal and then new ones as they arrive.
The communication between the two daemons is done over standard HTTPS, following rather simple rules, so it is possible to create alternate implementations without much work. For example, curl can be easily used to upload journal entries from a text file containing entries in the export format. Basically, the data are sent in an HTTP POST to /upload with Content- Type: application/vnd.fdo.journal. When doing "live" forwarding, the size of the transfer cannot be known in advance, so Transfer-Encoding: chunked is used. All communication is encrypted, and the identity of both sides is verified by checking for appropriate signatures on the certificates.
How certificates are managed for sender and receiver parts? Who generates them? Do you require explicit placement of the certificates prior to enabling the service?
On Mon, Apr 14, 2014 at 05:19:17PM +0300, Alexander Bokovoy wrote:
How certificates are managed for sender and receiver parts?
By some external means... This could be automated, e.g. using certmaster, but I don't want to tie to a specific certificate distribution implementation.
Who generates them? Do you require explicit placement of the certificates prior to enabling the service?
Yes. I want to push towards having the certificates in place in the default location, although it is of course possible to specify an alternate location through the config file, or even turn off certificate checking, but the defaults are supposed to be secure.
Zbyszek
On Mon, 14 Apr 2014, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Apr 14, 2014 at 05:19:17PM +0300, Alexander Bokovoy wrote:
How certificates are managed for sender and receiver parts?
By some external means... This could be automated, e.g. using certmaster, but I don't want to tie to a specific certificate distribution implementation.
Ok. I was worried you'll do the opposite. ;)
Who generates them? Do you require explicit placement of the certificates prior to enabling the service?
Yes. I want to push towards having the certificates in place in the default location, although it is of course possible to specify an alternate location through the config file, or even turn off certificate checking, but the defaults are supposed to be secure.
Agreed.
To be clear here, all this is implemented in the two daemons right?
When you say it uses https, thats natively done in the daemons, they don't need apache or some other https implementor in the way?
Which ssl stack does this use? nss? openssl? gnutls? something else?
kevin
On Tue, Apr 15, 2014 at 09:25:39AM -0600, Kevin Fenzi wrote:
To be clear here, all this is implemented in the two daemons right?
When you say it uses https, thats natively done in the daemons, they don't need apache or some other https implementor in the way?
Yes, it's implemented in two daemons, using libmicrohttpd (for receiving) and libcurl (for sending).
Which ssl stack does this use? nss? openssl? gnutls? something else?
libmicrohttpd is linked with gnutls, and libcurl is linked with nss. It's a bit unfortunate, but hard to work around. OTOH, it is likely that only one of those daemons would be running on a given machine, so the fact that they use different cryptostacks might not be that important.
The two daemons can be fairly well locked down, since they don't need any prileges apart from access to the journal files. So they'll run as separate users, the receiver with PrivateNetwork=yes. I also plan to add some seccomp filters later on.
Zbyszek
Jaroslav Reznik (jreznik@redhat.com) said:
= Proposed Self Contained Change: Remote Journal Logging = https://fedoraproject.org/wiki/Changes/Remote_Journal_Logging
Change owner(s): Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl
Systemd journal can be configured to forward events to a remote server. Entries are forwarded including full metadata, and are stored in normal journal files, identically to locally generated logs.
What's the future of gatewayd if this becomes more widely used?
Bill
On Mon, Apr 14, 2014 at 04:20:16PM -0400, Bill Nottingham wrote:
Jaroslav Reznik (jreznik@redhat.com) said:
= Proposed Self Contained Change: Remote Journal Logging = https://fedoraproject.org/wiki/Changes/Remote_Journal_Logging
Change owner(s): Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl
Systemd journal can be configured to forward events to a remote server. Entries are forwarded including full metadata, and are stored in normal journal files, identically to locally generated logs.
What's the future of gatewayd if this becomes more widely used?
gatewayd works in pull mode. Here I'm proposing a push model, where the "client" (i.e. machine generating the logs) pushes logs to the server at the time of its own chosing. gatewayd is probably better for some use cases, this for others.
Zbyszek
Zbigniew Jędrzejewski-Szmek (zbyszek@in.waw.pl) said:
On Mon, Apr 14, 2014 at 04:20:16PM -0400, Bill Nottingham wrote:
Jaroslav Reznik (jreznik@redhat.com) said:
= Proposed Self Contained Change: Remote Journal Logging = https://fedoraproject.org/wiki/Changes/Remote_Journal_Logging
Change owner(s): Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl
Systemd journal can be configured to forward events to a remote server. Entries are forwarded including full metadata, and are stored in normal journal files, identically to locally generated logs.
What's the future of gatewayd if this becomes more widely used?
gatewayd works in pull mode. Here I'm proposing a push model, where the "client" (i.e. machine generating the logs) pushes logs to the server at the time of its own chosing. gatewayd is probably better for some use cases, this for others.
I understand the pull vs push distinction ... I'm just not clear why pull would ever be a model you'd want to use. (vs something like a local cockpit agent.)
Bill
On Wed, Apr 16, 2014 at 12:46:15PM -0400, Bill Nottingham wrote:
Zbigniew Jędrzejewski-Szmek (zbyszek@in.waw.pl) said:
On Mon, Apr 14, 2014 at 04:20:16PM -0400, Bill Nottingham wrote:
Jaroslav Reznik (jreznik@redhat.com) said:
= Proposed Self Contained Change: Remote Journal Logging = https://fedoraproject.org/wiki/Changes/Remote_Journal_Logging
Change owner(s): Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl
Systemd journal can be configured to forward events to a remote server. Entries are forwarded including full metadata, and are stored in normal journal files, identically to locally generated logs.
What's the future of gatewayd if this becomes more widely used?
gatewayd works in pull mode. Here I'm proposing a push model, where the "client" (i.e. machine generating the logs) pushes logs to the server at the time of its own chosing. gatewayd is probably better for some use cases, this for others.
I understand the pull vs push distinction ... I'm just not clear why pull would ever be a model you'd want to use. (vs something like a local cockpit agent.)
Push mode feels more useful to me too, but there's some rationale for the pull mode. It leaves the schedule/speed/format in the control of the puller. So e.g. if there's just one "client" gathering logs from thousands of "servers", it might be overwhelmed in push mode, but work fine in pull mode.
Zbyszek
On Wed, 16.04.14 12:46, Bill Nottingham (notting@splat.cc) wrote:
Zbigniew Jędrzejewski-Szmek (zbyszek@in.waw.pl) said:
On Mon, Apr 14, 2014 at 04:20:16PM -0400, Bill Nottingham wrote:
Jaroslav Reznik (jreznik@redhat.com) said:
= Proposed Self Contained Change: Remote Journal Logging = https://fedoraproject.org/wiki/Changes/Remote_Journal_Logging
Change owner(s): Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl
Systemd journal can be configured to forward events to a remote server. Entries are forwarded including full metadata, and are stored in normal journal files, identically to locally generated logs.
What's the future of gatewayd if this becomes more widely used?
gatewayd works in pull mode. Here I'm proposing a push model, where the "client" (i.e. machine generating the logs) pushes logs to the server at the time of its own chosing. gatewayd is probably better for some use cases, this for others.
I understand the pull vs push distinction ... I'm just not clear why pull would ever be a model you'd want to use. (vs something like a local cockpit agent.)
Pull is the only model that scales, since the centralized log infrastructure can schedule when it pulls from where and thus do this according to available resources. THe push model is prone to logging bursts overwhelming log servers if you scale your network up.
I am pretty sure that a pull model should be the default for everything we do, and push only be done where realtimish behaviour is desired to do live debugging or suchlike.
I am pretty sure the push model concept is one of the major weaknesses of the BSD syslog protocol.
Lennart
On Tue, 2014-04-22 at 06:34 +0200, Lennart Poettering wrote:
On Wed, 16.04.14 12:46, Bill Nottingham (notting@splat.cc) wrote:
Zbigniew Jędrzejewski-Szmek (zbyszek@in.waw.pl) said:
On Mon, Apr 14, 2014 at 04:20:16PM -0400, Bill Nottingham wrote:
Jaroslav Reznik (jreznik@redhat.com) said:
= Proposed Self Contained Change: Remote Journal Logging = https://fedoraproject.org/wiki/Changes/Remote_Journal_Logging
Change owner(s): Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl
Systemd journal can be configured to forward events to a remote server. Entries are forwarded including full metadata, and are stored in normal journal files, identically to locally generated logs.
What's the future of gatewayd if this becomes more widely used?
gatewayd works in pull mode. Here I'm proposing a push model, where the "client" (i.e. machine generating the logs) pushes logs to the server at the time of its own chosing. gatewayd is probably better for some use cases, this for others.
I understand the pull vs push distinction ... I'm just not clear why pull would ever be a model you'd want to use. (vs something like a local cockpit agent.)
Pull is the only model that scales, since the centralized log infrastructure can schedule when it pulls from where and thus do this according to available resources. THe push model is prone to logging bursts overwhelming log servers if you scale your network up.
I am pretty sure that a pull model should be the default for everything we do, and push only be done where realtimish behaviour is desired to do live debugging or suchlike.
I am pretty sure the push model concept is one of the major weaknesses of the BSD syslog protocol.
Except that the server may not need direct access to the clients (in NATted LANs for examples), so sometimes push is all you can count on, make sure you can think how to properly rate limit, give feedback to clients if necessary. A good protocol would allow to send a first small packet that establish a connection and a reply that can "push back" on the client w/o requiring huge bandwidth to be spent.
Simo.
2014-04-22 15:10 GMT+02:00 Simo Sorce simo@redhat.com:
A good protocol would allow to send a first small packet that establish a connection and a reply that can "push back" on the client w/o requiring huge bandwidth to be spent.
Isn't that an inherent capability of TCP? If it is not automatic, maybe it would only be up to the log receiver to appropriately decrease the receive buffers on their incoming sockets? Mirek
On Tue, 2014-04-22 at 19:04 +0200, Miloslav Trmač wrote:
2014-04-22 15:10 GMT+02:00 Simo Sorce simo@redhat.com:
A good protocol would allow to send a first small packet that establish a connection and a reply that can "push back" on the client w/o requiring huge bandwidth to be spent.
Isn't that an inherent capability of TCP?
Sure, that's one way when bandwidth is the issue. There are other issues though, and just closing the connection on the client if you do not want any traffic is a bit blunt. It also does not give the client any idea when it is ok to retry.
If it is not automatic, maybe it would only be up to the log receiver to appropriately decrease the receive buffers on their incoming sockets?
This is not a problem I care, the kernel already does it quite well.
Simo.
2014-04-22 20:19 GMT+02:00 Simo Sorce simo@redhat.com:
On Tue, 2014-04-22 at 19:04 +0200, Miloslav Trmač wrote:
2014-04-22 15:10 GMT+02:00 Simo Sorce simo@redhat.com:
A good protocol would allow to send a first small packet that establish a connection and a reply that can "push back" on the client w/o requiring huge bandwidth to be spent.
Isn't that an inherent capability of TCP?
Sure, that's one way when bandwidth is the issue. There are other issues though, and just closing the connection on the client if you do not want any traffic is a bit blunt. It also does not give the client any idea when it is ok to retry.
Not like that—just stop reading from the socket, causing the server to advertise a zero-length window to the client. The client will then know that writes are blocking / not being processed. And when the server has more capacity, free up the buffers and the kernel will send a window update to the client automatically. Mirek
On Tue, 2014-04-22 at 20:58 +0200, Miloslav Trmač wrote:
2014-04-22 20:19 GMT+02:00 Simo Sorce simo@redhat.com:
On Tue, 2014-04-22 at 19:04 +0200, Miloslav Trmač wrote:
2014-04-22 15:10 GMT+02:00 Simo Sorce simo@redhat.com:
A good protocol would allow to send a first small packet that establish a connection and a reply that can "push back" on the client w/o requiring huge bandwidth to be spent.
Isn't that an inherent capability of TCP?
Sure, that's one way when bandwidth is the issue. There are other issues though, and just closing the connection on the client if you do not want any traffic is a bit blunt. It also does not give the client any idea when it is ok to retry.
Not like that—just stop reading from the socket, causing the server to advertise a zero-length window to the client. The client will then know that writes are blocking / not being processed. And when the server has more capacity, free up the buffers and the kernel will send a window update to the client automatically.
And you keep resources tied this way, it is better to tell the client to come back later, maybe give it a time when it is ok to come back.
The client can decide what to do, whether to save logs or dump them, or retry earlier anyway if it is running out of buffer space and some such.
But that is not for me to decide, so I'll stop here.
Simo.
On Tue, Apr 22, 2014 at 03:32:26PM -0400, Simo Sorce wrote:
On Tue, 2014-04-22 at 20:58 +0200, Miloslav Trmač wrote:
2014-04-22 20:19 GMT+02:00 Simo Sorce simo@redhat.com:
On Tue, 2014-04-22 at 19:04 +0200, Miloslav Trmač wrote:
2014-04-22 15:10 GMT+02:00 Simo Sorce simo@redhat.com:
A good protocol would allow to send a first small packet that establish a connection and a reply that can "push back" on the client w/o requiring huge bandwidth to be spent.
Isn't that an inherent capability of TCP?
Sure, that's one way when bandwidth is the issue. There are other issues though, and just closing the connection on the client if you do not want any traffic is a bit blunt. It also does not give the client any idea when it is ok to retry.
Not like that—just stop reading from the socket, causing the server to advertise a zero-length window to the client. The client will then know that writes are blocking / not being processed. And when the server has more capacity, free up the buffers and the kernel will send a window update to the client automatically.
And you keep resources tied this way, it is better to tell the client to come back later, maybe give it a time when it is ok to come back.
The amount of resources should be rather insignificant... A few hundred kilobytes of RSS on the uploader side, and an open socket and few hundred bytes of local state on the receiver side. Since it's just one uploader per client, and let's say 1kb per client on the receiver side, this shouldn't be an issue. The advantage is the simplicify of this solution.
Zbyszek
On Tue, 22.04.14 09:10, Simo Sorce (simo@redhat.com) wrote:
I am pretty sure that a pull model should be the default for everything we do, and push only be done where realtimish behaviour is desired to do live debugging or suchlike.
I am pretty sure the push model concept is one of the major weaknesses of the BSD syslog protocol.
Except that the server may not need direct access to the clients (in NATted LANs for examples), so sometimes push is all you can count on, make sure you can think how to properly rate limit, give feedback to clients if necessary. A good protocol would allow to send a first small packet that establish a connection and a reply that can "push back" on the client w/o requiring huge bandwidth to be spent.
Well, you can always turn the NAT problem around. Sometimes it's the log server behind the NAT that is the problem, sometimes it it is the log client behind the NAT that is the pronlem. If you consider push vs. pull then you simply reverse which one is the bigger issue.
Note that the journal protocol is HTTP, so it's probably as proxy and NAT-friendly as it gets.
Lennart
On Wed, 2014-04-23 at 05:36 +0200, Lennart Poettering wrote:
On Tue, 22.04.14 09:10, Simo Sorce (simo@redhat.com) wrote:
I am pretty sure that a pull model should be the default for everything we do, and push only be done where realtimish behaviour is desired to do live debugging or suchlike.
I am pretty sure the push model concept is one of the major weaknesses of the BSD syslog protocol.
Except that the server may not need direct access to the clients (in NATted LANs for examples), so sometimes push is all you can count on, make sure you can think how to properly rate limit, give feedback to clients if necessary. A good protocol would allow to send a first small packet that establish a connection and a reply that can "push back" on the client w/o requiring huge bandwidth to be spent.
Well, you can always turn the NAT problem around. Sometimes it's the log server behind the NAT that is the problem, sometimes it it is the log client behind the NAT that is the pronlem. If you consider push vs. pull then you simply reverse which one is the bigger issue.
Nope, 1 server means you can do port forwarding on the NAT to the specific server, all clients connect to the same NAT address port and their connection is forwarded to the server, because it is 1.
The reverse would require to manually map 1 port per client.
Big difference.
Note that the journal protocol is HTTP, so it's probably as proxy and NAT-friendly as it gets.
I already commented how bad an idea it is to use HTTP in the other thread.
Simo.
On Tue, Apr 22, 2014 at 06:34:48AM +0200, Lennart Poettering wrote:
Pull is the only model that scales, since the centralized log infrastructure can schedule when it pulls from where and thus do this according to available resources. THe push model is prone to logging bursts overwhelming log servers if you scale your network up.
This terminology is (excitingly?) reversed from push vs. pull in configuration management, where either clients pull from a server or have commands pushed to them. Maybe we could use different terms for logging instead, like "send" and "fetch"?
Or maybe it's just me that's easily confused.
I am pretty sure that a pull model should be the default for everything we do, and push only be done where realtimish behaviour is desired to do live debugging or suchlike. I am pretty sure the push model concept is one of the major weaknesses of the BSD syslog protocol.
There are advantages and disadvantages to both. Disadvantages of pull (fetch) include:
- If a client system is broken into, logs may be removed before they're fetched. (Of course, this is also a risk with a "batched" send/push scheme.) - Client system needs to hold logs until asked -- maybe a long time. Traditionally some devices don't hold logs at all and _just_ send them remotely. - Usually pull/fetch designs require more configuration, with the server needing a list of clients to check and the clients configured to allow connections from just that server in.
But it's definitely better for scaling, and you don't have to worry about 100% log server uptime, and the log server itself can be kept more secure.
And on the neutral side, the network design may make it hard to reach clients; they might be behind NAT/PNAT from the log server's point of view. That's not necessarily an inherent pro or con, but may make push/send fit better into a certain environment.
On Tue, Apr 22, 2014 at 06:34:48AM +0200, Lennart Poettering wrote:
On Wed, 16.04.14 12:46, Bill Nottingham (notting@splat.cc) wrote:
Zbigniew Jędrzejewski-Szmek (zbyszek@in.waw.pl) said:
On Mon, Apr 14, 2014 at 04:20:16PM -0400, Bill Nottingham wrote:
Jaroslav Reznik (jreznik@redhat.com) said:
= Proposed Self Contained Change: Remote Journal Logging = https://fedoraproject.org/wiki/Changes/Remote_Journal_Logging
Change owner(s): Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl
Systemd journal can be configured to forward events to a remote server. Entries are forwarded including full metadata, and are stored in normal journal files, identically to locally generated logs.
What's the future of gatewayd if this becomes more widely used?
gatewayd works in pull mode. Here I'm proposing a push model, where the "client" (i.e. machine generating the logs) pushes logs to the server at the time of its own chosing. gatewayd is probably better for some use cases, this for others.
I understand the pull vs push distinction ... I'm just not clear why pull would ever be a model you'd want to use. (vs something like a local cockpit agent.)
Pull is the only model that scales, since the centralized log infrastructure can schedule when it pulls from where and thus do this according to available resources. THe push model is prone to logging bursts overwhelming log servers if you scale your network up.
How many clients would need to connect simultaneously to overwhelm the server? And "overwhelm" here would have to mean something like overflowing the incoming connection queue. The receiver binary doesn't have to actually read the data from the connections immedately, and things should function just fine if it takes a minute or two to process data. A typical "overwhelm" scenario that we might be talking about would be a massive machine restart after a power failure. A typical amount of log messages generating during boot is rather small: less than 1MB on F21. The receiver should be able to process data at around disk speed, so it should be able to handle *hundreds* of boot machines without actually developing a delay of more than a few seconds. In addition, it would be great to add jitter to starting of the uploader, which would lessen the load on the server anyway.
I am pretty sure that a pull model should be the default for everything we do, and push only be done where realtimish behaviour is desired to do live debugging or suchlike.
My biggest gripe with the pull model is the configuration issue mentioned by mattdm elsewhere in the thread. If I have a few machines on my home network, some VMs, a notebook or two, it is much easier to keep the configuration of the receiver stable and configure all hosts identically to push to it, then the other way around. Especially that both network addresses and host names change, so it's really hard to even to tell the receiver where to pull from. A list of hosts to pull from residing on the server is bound to become out of date.
I am pretty sure the push model concept is one of the major weaknesses of the BSD syslog protocol.
It's a problem, but mostly because there's very little buffering and things are mostly synchronous. But anyway, let's get both models working... I wouldn't be surprised if both find their niches.
Zbyszek
On Tuesday 22 April 2014 06:34:48 Lennart Poettering wrote:
On Wed, 16.04.14 12:46, Bill Nottingham (notting@splat.cc) wrote:
Zbigniew Jędrzejewski-Szmek (zbyszek@in.waw.pl) said:
On Mon, Apr 14, 2014 at 04:20:16PM -0400, Bill Nottingham wrote:
Jaroslav Reznik (jreznik@redhat.com) said:
= Proposed Self Contained Change: Remote Journal Logging = https://fedoraproject.org/wiki/Changes/Remote_Journal_Logging
Change owner(s): Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl
Systemd journal can be configured to forward events to a remote server. Entries are forwarded including full metadata, and are stored in normal journal files, identically to locally generated logs.
What's the future of gatewayd if this becomes more widely used?
gatewayd works in pull mode. Here I'm proposing a push model, where the "client" (i.e. machine generating the logs) pushes logs to the server at the time of its own chosing. gatewayd is probably better for some use cases, this for others.
I understand the pull vs push distinction ... I'm just not clear why pull would ever be a model you'd want to use. (vs something like a local cockpit agent.)
Pull is the only model that scales, since the centralized log infrastructure can schedule when it pulls from where and thus do this according to available resources. THe push model is prone to logging bursts overwhelming log servers if you scale your network up.
"Push" mechanism seems to scale pretty well when we talk about web sites. Specially taking into account that you mentioned that systemd is going to use https protocol, you can take advantage of many existing solutions to scale/load balance your systems. (although that doesn't mean I agree on the election of the protocol)
I am pretty sure that a pull model should be the default for everything we do, and push only be done where realtimish behaviour is desired to do live debugging or suchlike.
For me it should be precisely the other way around(in syslogging context)
I am pretty sure the push model concept is one of the major weaknesses of the BSD syslog protocol.
Again I disagree: for me is a strong point :-)
On 04/16/2014 06:46 PM, Bill Nottingham wrote:
I understand the pull vs push distinction ... I'm just not clear why pull would ever be a model you'd want to use. (vs something like a local cockpit agent.)
Isn't remote Windows event logging pull-only (unless you somehow gate it to syslog)? So it would be consistent.
Back in the old days, there was a lot of loathing going around for Windows event logging. People loved syslog, but really hated the Windows counterpart. The pull model was criticized as well, but I'm not sure if it was really pulling stuff or the bad interaction with other design considerations (like the need for DLLs with parsers for the log blobs).
Does anybody know what the current Windows experience is like, especially related to the pull model?
On Mon, 2014-04-14 at 15:07 +0200, Jaroslav Reznik wrote:
= Proposed Self Contained Change: Remote Journal Logging =
The communication between the two daemons is done over standard HTTPS, following rather simple rules, so it is possible to create alternate implementations without much work. For example, curl can be easily used to upload journal entries from a text file containing entries in the export format. Basically, the data are sent in an HTTP POST to /upload with Content- Type: application/vnd.fdo.journal. When doing "live" forwarding, the size of the transfer cannot be known in advance, so Transfer-Encoding: chunked is used. All communication is encrypted, and the identity of both sides is verified by checking for appropriate signatures on the certificates.
HTTP seem like a bad idea in terms of security, certificates are notoriously very hard to manage, even with the help of things like certmonger, and hard to properly validate in most libraries today.
Let alone dealing with setting up a CA just for enabling remote logging (or otherwise painfully exchange fingerprints and white list certificates for each client-server pair.
And please do not tell me this is deferred to the admin to figure out, because then it would mean this feature cannot seriously be used in normal setups.
Is there any reason why a better custom protocol that can be secured using things like SASL or GSSAPI is not used ? Has it been considered ? What are the pros of using HTTP if all you are doing are POSTS to a hardcoded URL ?
Simo.
On Tue, Apr 15, 2014 at 11:00:45AM -0400, Simo Sorce wrote:
On Mon, 2014-04-14 at 15:07 +0200, Jaroslav Reznik wrote:
= Proposed Self Contained Change: Remote Journal Logging =
The communication between the two daemons is done over standard HTTPS, following rather simple rules, so it is possible to create alternate implementations without much work. For example, curl can be easily used to upload journal entries from a text file containing entries in the export format. Basically, the data are sent in an HTTP POST to /upload with Content- Type: application/vnd.fdo.journal. When doing "live" forwarding, the size of the transfer cannot be known in advance, so Transfer-Encoding: chunked is used. All communication is encrypted, and the identity of both sides is verified by checking for appropriate signatures on the certificates.
What are the pros of using HTTP if all you are doing are POSTS to a hardcoded URL ?
Using HTTP makes it possible to use e.g. use curl to upload some logs from the commandline. It should also be fairly easy for people to write e.g. Python code to upload logs. I also expect people to want to send json formatted logs at some point, and the HTTP headers make things fairly extensible. I think that using standard HTTP is easier than designing a biderectional protocol.
HTTP seem like a bad idea in terms of security, certificates are notoriously very hard to manage, even with the help of things like certmonger, and hard to properly validate in most libraries today.
Let alone dealing with setting up a CA just for enabling remote logging (or otherwise painfully exchange fingerprints and white list certificates for each client-server pair.
And please do not tell me this is deferred to the admin to figure out, because then it would mean this feature cannot seriously be used in normal setups.
I think you exaggarate a bit. Managing certificates is annoying, true, but there's lots of advice on the web and howto and various helper software. I'd imagine that in a setup with a few servers one would create the certificates on the receiver machine, copy&pasting some instructions from Fedora docs, and scp them to the other hosts.
Is there any reason why a better custom protocol that can be secured using things like SASL or GSSAPI is not used ?
It doesn't really fit well with the overall approach of using HTTP. But I'm open to suggestions how to do this better. The Change page is so obnoxiously detailed so that I can get feedback :)
Zbyszek
On Tue, 2014-04-15 at 20:28 +0200, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Apr 15, 2014 at 11:00:45AM -0400, Simo Sorce wrote:
On Mon, 2014-04-14 at 15:07 +0200, Jaroslav Reznik wrote:
= Proposed Self Contained Change: Remote Journal Logging =
The communication between the two daemons is done over standard HTTPS, following rather simple rules, so it is possible to create alternate implementations without much work. For example, curl can be easily used to upload journal entries from a text file containing entries in the export format. Basically, the data are sent in an HTTP POST to /upload with Content- Type: application/vnd.fdo.journal. When doing "live" forwarding, the size of the transfer cannot be known in advance, so Transfer-Encoding: chunked is used. All communication is encrypted, and the identity of both sides is verified by checking for appropriate signatures on the certificates.
What are the pros of using HTTP if all you are doing are POSTS to a hardcoded URL ?
Using HTTP makes it possible to use e.g. use curl to upload some logs from the commandline. It should also be fairly easy for people to write e.g. Python code to upload logs. I also expect people to want to send json formatted logs at some point, and the HTTP headers make things fairly extensible. I think that using standard HTTP is easier than designing a biderectional protocol.
I understand why HTTP looks convenient for some things, but really, aside for development, would you ever expect someone lo log to your interface except from a journald client ?
HTTP seem like a bad idea in terms of security, certificates are notoriously very hard to manage, even with the help of things like certmonger, and hard to properly validate in most libraries today.
Let alone dealing with setting up a CA just for enabling remote logging (or otherwise painfully exchange fingerprints and white list certificates for each client-server pair.
And please do not tell me this is deferred to the admin to figure out, because then it would mean this feature cannot seriously be used in normal setups.
I think you exaggarate a bit.
I think you haven't tried for real :-)
Managing certificates is annoying, true, but there's lots of advice on the web and howto and various helper software.
There is a lot of "wrong" advice, mostly you find people telling other people to disable verification, and to use self signed certificates.
I'd imagine that in a setup with a few servers one would create the certificates on the receiver machine, copy&pasting some instructions from Fedora docs, and scp them to the other hosts.
What I am asking is: How do you validate certificates on the client ? Are you going to use white lists ? Are you going to depend on a CA white listed ? Are you going to create an extensions to be put in the certificates to restrict their use to logging ? Or are you going to allow to use any certificate as long as the CN matches the machine name ? Are you going to have white lists on the server ? Are you going to require client certificate authentication, or will you allow any anonymous client to flood your server with garbage ?
How do you generate certificates ? NSS comes with certutil, but it is not very flexible, I am not sure about wat GnuTLS comes with, but that library is not something I would like to depend on in the first place.
Also how are you going to harmonize client versus server management, the 2 stacks (NSS and GnuTLS) are quite different, having to learn not 1 but 2 stacks seem quite steep.
Is there any reason why a better custom protocol that can be secured using things like SASL or GSSAPI is not used ?
It doesn't really fit well with the overall approach of using HTTP. But I'm open to suggestions how to do this better. The Change page is so obnoxiously detailed so that I can get feedback :)
If you used something like protobuffers or dbus for the message formatting and use a socket for the transport where you can choose whether you want to do TLS vs GSSAPI easily (as you are not tied by the inability of HTTP to use anything but TLS) then you would have a much more flexible system. Also you wouldn't be tied to use 2 different crypto stacks in the same project which is really a shame. You have to duplicate everything and that will be prone to errors or incompatibilities and you'll have to use the lower common denominator if there are feature mismatches.
You could also eventually tunnel over HTTPS, after it is just buffers going back and forth, but you wouldn't be tied to it.
HTH, Simo.
On Tue, Apr 15, 2014 at 03:30:57PM -0400, Simo Sorce wrote:
On Tue, 2014-04-15 at 20:28 +0200, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Apr 15, 2014 at 11:00:45AM -0400, Simo Sorce wrote:
On Mon, 2014-04-14 at 15:07 +0200, Jaroslav Reznik wrote:
= Proposed Self Contained Change: Remote Journal Logging =
The communication between the two daemons is done over standard HTTPS, following rather simple rules, so it is possible to create alternate implementations without much work. For example, curl can be easily used to upload journal entries from a text file containing entries in the export format. Basically, the data are sent in an HTTP POST to /upload with Content- Type: application/vnd.fdo.journal. When doing "live" forwarding, the size of the transfer cannot be known in advance, so Transfer-Encoding: chunked is used. All communication is encrypted, and the identity of both sides is verified by checking for appropriate signatures on the certificates.
What are the pros of using HTTP if all you are doing are POSTS to a hardcoded URL ?
Using HTTP makes it possible to use e.g. use curl to upload some logs from the commandline. It should also be fairly easy for people to write e.g. Python code to upload logs. I also expect people to want to send json formatted logs at some point, and the HTTP headers make things fairly extensible. I think that using standard HTTP is easier than designing a biderectional protocol.
I understand why HTTP looks convenient for some things, but really, aside for development, would you ever expect someone lo log to your interface except from a journald client ?
HTTP seem like a bad idea in terms of security, certificates are notoriously very hard to manage, even with the help of things like certmonger, and hard to properly validate in most libraries today.
Let alone dealing with setting up a CA just for enabling remote logging (or otherwise painfully exchange fingerprints and white list certificates for each client-server pair.
And please do not tell me this is deferred to the admin to figure out, because then it would mean this feature cannot seriously be used in normal setups.
I think you exaggarate a bit.
I think you haven't tried for real :-)
Managing certificates is annoying, true, but there's lots of advice on the web and howto and various helper software.
There is a lot of "wrong" advice, mostly you find people telling other people to disable verification, and to use self signed certificates.
I'd imagine that in a setup with a few servers one would create the certificates on the receiver machine, copy&pasting some instructions from Fedora docs, and scp them to the other hosts.
Hi, in current setup, I require (on both ends) for the certificate on the other to be signed by a specified authority. So some of the questions you ask don't really apply, since the certificates are supposed to be generated specifically for this purpose.
What I am asking is: How do you validate certificates on the client ?
I check if they are signed by a specified authoriy.
Are you going to use white lists ?
No.
Are you going to depend on a CA white listed ?
Yes.
Are you going to create an extensions to be put in the certificates to restrict their use to logging ? Or are you going to allow to use any certificate as long as the CN matches the machine name ? Are you going to have white lists on the server ? Are you going to require client certificate authentication, or will you allow any anonymous client to flood your server with garbage ?
The client is authenticated by having a certificate signed by specified authority.
How do you generate certificates ? NSS comes with certutil, but it is not very flexible, I am not sure about wat GnuTLS comes with, but that library is not something I would like to depend on in the first place.
I leave the generation of the certificates to the admin. I personally used openssl so far, but this shouldn't really matter.
Also how are you going to harmonize client versus server management, the 2 stacks (NSS and GnuTLS) are quite different, having to learn not 1 but 2 stacks seem quite steep.
I don't think that's something that the user would care about. On the programming side it's annoying though.
Is there any reason why a better custom protocol that can be secured using things like SASL or GSSAPI is not used ?
It doesn't really fit well with the overall approach of using HTTP. But I'm open to suggestions how to do this better. The Change page is so obnoxiously detailed so that I can get feedback :)
If you used something like protobuffers or dbus for the message formatting and use a socket for the transport where you can choose whether you want to do TLS vs GSSAPI easily (as you are not tied by the inability of HTTP to use anything but TLS) then you would have a much more flexible system. Also you wouldn't be tied to use 2 different crypto stacks in the same project which is really a shame. You have to duplicate everything and that will be prone to errors or incompatibilities and you'll have to use the lower common denominator if there are feature mismatches.
You could also eventually tunnel over HTTPS, after it is just buffers going back and forth, but you wouldn't be tied to it.
I'll reconsider using SASL instead. I have the HTTPS-transport version almost ready, so for now I'll go with that, to have a working solution. There's still some other questions, mostly related to how the data should be stored on the receiver, so I want to get it out for people to test. The underlying transport is an implementation detail, mostly visible in the way that authentication is done, so a new transport can be added, and the old one deprecated or even removed. Thank you for those suggestions.
Zbyszek
On Wed, 2014-04-16 at 15:04 +0200, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Apr 15, 2014 at 03:30:57PM -0400, Simo Sorce wrote:
I'd imagine that in a setup with a few servers one would create the certificates on the receiver machine, copy&pasting some instructions from Fedora docs, and scp them to the other hosts.
Hi, in current setup, I require (on both ends) for the certificate on the other to be signed by a specified authority. So some of the questions you ask don't really apply, since the certificates are supposed to be generated specifically for this purpose.
What I am asking is: How do you validate certificates on the client ?
I check if they are signed by a specified authoriy.
What authority ? Is this a CA ? Or do you allow listing fingerprints of self-signed certificates ?
Are you going to use white lists ?
No.
So if I have a large domain the only option is to allow all/or nothing, or to build many subCAs ?
Are you going to depend on a CA white listed ?
Yes.
Are you going to create an extensions to be put in the certificates to restrict their use to logging ? Or are you going to allow to use any certificate as long as the CN matches the machine name ? Are you going to have white lists on the server ? Are you going to require client certificate authentication, or will you allow any anonymous client to flood your server with garbage ?
The client is authenticated by having a certificate signed by specified authority.
So you rely on X509 auth, if the client has no cert you refuse the connection, correct ?
How do you generate certificates ? NSS comes with certutil, but it is not very flexible, I am not sure about wat GnuTLS comes with, but that library is not something I would like to depend on in the first place.
I leave the generation of the certificates to the admin. I personally used openssl so far, but this shouldn't really matter.
Ok so we are a 3 separate implementations .. wow :)
Also how are you going to harmonize client versus server management, the 2 stacks (NSS and GnuTLS) are quite different, having to learn not 1 but 2 stacks seem quite steep.
I don't think that's something that the user would care about. On the programming side it's annoying though.
Is there any reason why a better custom protocol that can be secured using things like SASL or GSSAPI is not used ?
It doesn't really fit well with the overall approach of using HTTP. But I'm open to suggestions how to do this better. The Change page is so obnoxiously detailed so that I can get feedback :)
If you used something like protobuffers or dbus for the message formatting and use a socket for the transport where you can choose whether you want to do TLS vs GSSAPI easily (as you are not tied by the inability of HTTP to use anything but TLS) then you would have a much more flexible system. Also you wouldn't be tied to use 2 different crypto stacks in the same project which is really a shame. You have to duplicate everything and that will be prone to errors or incompatibilities and you'll have to use the lower common denominator if there are feature mismatches.
You could also eventually tunnel over HTTPS, after it is just buffers going back and forth, but you wouldn't be tied to it.
I'll reconsider using SASL instead. I have the HTTPS-transport version almost ready, so for now I'll go with that, to have a working solution. There's still some other questions, mostly related to how the data should be stored on the receiver, so I want to get it out for people to test. The underlying transport is an implementation detail, mostly visible in the way that authentication is done, so a new transport can be added, and the old one deprecated or even removed. Thank you for those suggestions.
You are welcome, note that I like the idea of being able to send logs remotely, I just want not to be trapped in that horrible thing that is HTTPS, it is very bad security wise, we have no choice for the "web" atm because browsers are very slow to adopt anything, but we shouldn't tie ourselves into this bad world for stuff that is not man to be exposed to web browsers.
Hopefully you'll also make sure the only TLS 1.2 or higher is accepted ? I didn't go on a rant on how horrible TLS is in itself, and the amount of attacks you can mount to it given it's use of crypto ...
Another advantage of SASL is that you can switch to use symmetric crypto from publickey crypto should public key encryption receive another setback (ie switch form TLS to GSSAPI). Shared secrets are much harder to "break" than public key crypto in some conditions, or simply require different types of attacks, but with HTTP/TLS you have no option but to use public key crypto.
Simo.
On Wed, Apr 16, 2014 at 12:48:21PM -0400, Simo Sorce wrote:
On Wed, 2014-04-16 at 15:04 +0200, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Apr 15, 2014 at 03:30:57PM -0400, Simo Sorce wrote:
I'd imagine that in a setup with a few servers one would create the certificates on the receiver machine, copy&pasting some instructions from Fedora docs, and scp them to the other hosts.
Hi, in current setup, I require (on both ends) for the certificate on the other to be signed by a specified authority. So some of the questions you ask don't really apply, since the certificates are supposed to be generated specifically for this purpose.
What I am asking is: How do you validate certificates on the client ?
I check if they are signed by a specified authoriy.
What authority ? Is this a CA ? Or do you allow listing fingerprints of self-signed certificates ?
Yes, signed by a specific CA.
Are you going to use white lists ?
No.
So if I have a large domain the only option is to allow all/or nothing, or to build many subCAs ?
Yes.
Are you going to depend on a CA white listed ?
Yes.
Are you going to create an extensions to be put in the certificates to restrict their use to logging ? Or are you going to allow to use any certificate as long as the CN matches the machine name ? Are you going to have white lists on the server ? Are you going to require client certificate authentication, or will you allow any anonymous client to flood your server with garbage ?
The client is authenticated by having a certificate signed by specified authority.
So you rely on X509 auth, if the client has no cert you refuse the connection, correct ?
Yes.
How do you generate certificates ? NSS comes with certutil, but it is not very flexible, I am not sure about wat GnuTLS comes with, but that library is not something I would like to depend on in the first place.
I leave the generation of the certificates to the admin. I personally used openssl so far, but this shouldn't really matter.
Ok so we are a 3 separate implementations .. wow :)
Also how are you going to harmonize client versus server management, the 2 stacks (NSS and GnuTLS) are quite different, having to learn not 1 but 2 stacks seem quite steep.
I don't think that's something that the user would care about. On the programming side it's annoying though.
Is there any reason why a better custom protocol that can be secured using things like SASL or GSSAPI is not used ?
It doesn't really fit well with the overall approach of using HTTP. But I'm open to suggestions how to do this better. The Change page is so obnoxiously detailed so that I can get feedback :)
If you used something like protobuffers or dbus for the message formatting and use a socket for the transport where you can choose whether you want to do TLS vs GSSAPI easily (as you are not tied by the inability of HTTP to use anything but TLS) then you would have a much more flexible system. Also you wouldn't be tied to use 2 different crypto stacks in the same project which is really a shame. You have to duplicate everything and that will be prone to errors or incompatibilities and you'll have to use the lower common denominator if there are feature mismatches.
You could also eventually tunnel over HTTPS, after it is just buffers going back and forth, but you wouldn't be tied to it.
I'll reconsider using SASL instead. I have the HTTPS-transport version almost ready, so for now I'll go with that, to have a working solution. There's still some other questions, mostly related to how the data should be stored on the receiver, so I want to get it out for people to test. The underlying transport is an implementation detail, mostly visible in the way that authentication is done, so a new transport can be added, and the old one deprecated or even removed. Thank you for those suggestions.
You are welcome, note that I like the idea of being able to send logs remotely, I just want not to be trapped in that horrible thing that is HTTPS, it is very bad security wise, we have no choice for the "web" atm because browsers are very slow to adopt anything, but we shouldn't tie ourselves into this bad world for stuff that is not man to be exposed to web browsers.
Hopefully you'll also make sure the only TLS 1.2 or higher is accepted ? I didn't go on a rant on how horrible TLS is in itself, and the amount of attacks you can mount to it given it's use of crypto ...
Good point. I haven't modified the defaults, to older versions are certainly accepted. I'll add it on the list of things to fix.
Zbyszek
Hello, 2014-04-16 15:04 GMT+02:00 Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl:
I'll reconsider using SASL instead. I have the HTTPS-transport version almost ready, so for now I'll go with that, to have a working solution. There's still some other questions, mostly related to how the data should be stored on the receiver, so I want to get it out for people to test. The underlying transport is an implementation detail, mostly visible in the way that authentication is done, so a new transport can be added, and the old one deprecated or even removed.
After this first ships in a product, that's not how networks work; protocols, once defined, are pretty much forever—or, well, until users' setups are broken and users are unhappy. Mirek
On Thu, Apr 17, 2014 at 10:12:24PM +0200, Miloslav Trmač wrote:
Hello, 2014-04-16 15:04 GMT+02:00 Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl:
I'll reconsider using SASL instead. I have the HTTPS-transport version almost ready, so for now I'll go with that, to have a working solution. There's still some other questions, mostly related to how the data should be stored on the receiver, so I want to get it out for people to test. The underlying transport is an implementation detail, mostly visible in the way that authentication is done, so a new transport can be added, and the old one deprecated or even removed.
After this first ships in a product, that's not how networks work; protocols, once defined, are pretty much forever—or, well, until users' setups are broken and users are unhappy. Mirek
It was an error on my part: I should have added a few big fat warnings that this code (and protocol) is not yet ready for general consumption. I'll add that now... I'd be pretty hard for anyone to use it so far, since the client part is missing.
Zbyszek
Zbigniew =?utf-8?Q?J=C4=99drzejewski-Szmek?= zbyszek@in.waw.pl writes:
[...] Using HTTP makes it possible to use e.g. use curl to upload some logs from the commandline. It should also be fairly easy for people to write e.g. Python code to upload logs. [...]
Are you envisioning these journal files being created by anything other than systemd? If not, are these python/curl/etc. clients just substitutes for journald neglecting to upload its files earlier?
- FChE
On Wed, Apr 16, 2014 at 11:39:07AM -0400, Frank Ch. Eigler wrote:
Zbigniew =?utf-8?Q?J=C4=99drzejewski-Szmek?= zbyszek@in.waw.pl writes:
[...] Using HTTP makes it possible to use e.g. use curl to upload some logs from the commandline. It should also be fairly easy for people to write e.g. Python code to upload logs. [...]
Are you envisioning these journal files being created by anything other than systemd? If not, are these python/curl/etc. clients just substitutes for journald neglecting to upload its files earlier?
I'm not sure what you mean by "neglecting". It is *not* supposed to be a workaround for a malfunction where systemd-journal-upload e.g. crashed and did not upload some logs; that case should be handled by the (automatic) restart and systemd-journal-upload continuing at the last known uploaded place.
What I had in mind is e.g. some custom filter / log-mangling software, which would upload a subset/superset of logs.
Zbyszek
On Mon, Apr 14, 2014 at 9:07 AM, Jaroslav Reznik jreznik@redhat.com wrote:
The communication between the two daemons is done over standard HTTPS,
Interesting. One quirk of current syslog-style remote logging over UDP is that it is fairly tolerant to dataloss.
With quite a bit of experience in the field... I have to say that this is both a bug, and a feature.
The bug is obvious, so let me explain the "feature" side
- if the remote server is unreachable or unresponsive, clients continue running without any adverse effects (other than loss of logging data)
- if the network link carrying the logging traffic is overwhelmed by the traffic, client nodes continue running without any adverse effects (other than...) at least from the logging machinery -- other traffic on that saturated link may lead other sw to misbehave)
I hear you holler "OMG you have to build full redundancy in your logging backend"; and... I have not seen a single operation where the logging backed was fully redundant.
And in fact it may be too much to ask -- in most setups log entries are not _that_ precious. I know I can reconfigure a syslog server and restart it without the 1K VMs that talk to it glitching. A recent "loop" in our syslog configuration was a relatively minor problem because it just dropped the traffic it couldn't handle for a brief time while we fixed things.
This is the reality of system configs I know. Fully redundant, "perfect" log servers are very hard to run, might not be worth it, and anyway such a change won't happen overnight.
To avoid gridlocking operations, IMO this logging will need a local queue, (for later retry), with a "drop anything older than X" escape hatch...
cheers,
m
Once upon a time, Martin Langhoff martin.langhoff@gmail.com said:
On Mon, Apr 14, 2014 at 9:07 AM, Jaroslav Reznik jreznik@redhat.com wrote:
The communication between the two daemons is done over standard HTTPS,
Interesting. One quirk of current syslog-style remote logging over UDP is that it is fairly tolerant to dataloss.
Yes, and there are multiple solutions to that issue. TCP syslog, rsyslog's replication protocol, etc. It would be good if systemd could use or extend an existing logging protocol, rather than invent yet another method.
2014-04-16 19:08 GMT+02:00 Chris Adams linux@cmadams.net:
It would be good if systemd could use or extend an existing logging protocol, rather than invent yet another method.
Yes. Going by the feature page and from what I can see from journal-remote.c, because Transfer-Encoding: chunked does not require application-level acknowledgment from the recipient, and there is no other mechanism to synchronize state, the proposed use of HTTP will be *losing data*!
See http://blog.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.htmlan... http://blog.gerhards.net/2008/05/why-you-cant-build-reliable-tcp.html , and http://www.rsyslog.com/doc/relp.html . Am I missing something? Mirek
On Thu, Apr 17, 2014 at 10:17:28PM +0200, Miloslav Trmač wrote:
2014-04-16 19:08 GMT+02:00 Chris Adams linux@cmadams.net:
It would be good if systemd could use or extend an existing logging protocol, rather than invent yet another method.
Yes. Going by the feature page and from what I can see from journal-remote.c, because Transfer-Encoding: chunked does not require application-level acknowledgment from the recipient, and there is no other mechanism to synchronize state, the proposed use of HTTP will be *losing data*!
There's another mechanism to synchronize state: the server replies with 202 accepted after it has successfully parsed and saved the transmission to disk. The client can send each journal entry as a single POST upload (using the same connection, so it's not terrible inefficient).
Zbyszek
2014-04-17 23:34 GMT+02:00 Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl:
On Thu, Apr 17, 2014 at 10:17:28PM +0200, Miloslav Trmač wrote:
2014-04-16 19:08 GMT+02:00 Chris Adams linux@cmadams.net:
It would be good if systemd could use or extend an existing logging protocol, rather than invent yet another method.
Yes. Going by the feature page and from what I can see from journal-remote.c, because Transfer-Encoding: chunked does not require application-level acknowledgment from the recipient, and there is no
other
mechanism to synchronize state, the proposed use of HTTP will be *losing data*!
There's another mechanism to synchronize state: the server replies with 202 accepted after it has successfully parsed and saved the transmission to disk. The client can send each journal entry as a single POST upload (using the same connection, so it's not terrible inefficient).
Sure, single POST per entry, or even per batch, would work (AFAICS equivalently with a 200 OK)—note that the data really needs to be on disk before the acknowledgment is sent. Mirek
On Wed, Apr 16, 2014 at 12:50:53PM -0400, Martin Langhoff wrote:
On Mon, Apr 14, 2014 at 9:07 AM, Jaroslav Reznik jreznik@redhat.com wrote:
The communication between the two daemons is done over standard HTTPS,
I hear you holler "OMG you have to build full redundancy in your logging backend"; and... I have not seen a single operation where the logging backed was fully redundant.
Hi, the upload client is like any other journal client — it is fully asynchronous wrt. to journald writing log entries. (It's something like 'journalctl -o export|curl -X POST https://some.where/upload'.) So it is assumed that local storage of journal logs is enabled (either in /var/log/journal or in /run/log/journal), and the uploader simply reads data from there at its leisure. In fact, the uploader is only started late during boot, after the network is up, while journald writes logs from very early in the boot.
There's no guarantee that the logs will ever be uploaded, but in practice they will only be lost if enough new logs come in to force the old entries out. Even with the journal capped at a few megabytes, this shouldn't happen but rarely.
And in fact it may be too much to ask -- in most setups log entries are not _that_ precious. I know I can reconfigure a syslog server and restart it without the 1K VMs that talk to it glitching. A recent "loop" in our syslog configuration was a relatively minor problem because it just dropped the traffic it couldn't handle for a brief time while we fixed things.
This is the reality of system configs I know. Fully redundant, "perfect" log servers are very hard to run, might not be worth it, and anyway such a change won't happen overnight.
To avoid gridlocking operations, IMO this logging will need a local queue, (for later retry), with a "drop anything older than X" escape hatch...
I think that this describes my current implementation to a large extent. The client remembers the position ("cursor") of the last uploaded entry and restarts from there. "drop anything older than X" feature sounds useful, and should be easy to add.
Zbyszek
On Wed, Apr 16, 2014 at 4:40 PM, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
the upload client is like any other journal client -- it is fully asynchronous wrt. to journald writing log entries. (It's something like 'journalctl -o export|curl -X POST https://some.where/upload'.)
Fantastic, so there is a "local spool". Somehow I had assumed there would not be a local storage.
So I'll ask you about this other aspect -- what about stateless clients with very limited or no local storage?
cheers,
m
On Wed, Apr 16, 2014 at 04:57:25PM -0400, Martin Langhoff wrote:
On Wed, Apr 16, 2014 at 4:40 PM, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
the upload client is like any other journal client -- it is fully asynchronous wrt. to journald writing log entries. (It's something like 'journalctl -o export|curl -X POST https://some.where/upload'.)
Fantastic, so there is a "local spool". Somehow I had assumed there would not be a local storage.
So I'll ask you about this other aspect -- what about stateless clients with very limited or no local storage?
Not supported by this, unfortunately. There needs to be at least temporary storage in tmpfs for this scheme to work.
Zbyszek
On Wed, Apr 16, 2014 at 5:08 PM, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
So I'll ask you about this other aspect -- what about stateless clients with very limited or no local storage?
Not supported by this, unfortunately. There needs to be at least temporary storage in tmpfs for this scheme to work.
Can the storage space used be limited with a parameter to avoid ENOSPC? Can journald be told to start discarding the data (oldest or latest) when it reaches the limit?
cheers,
m
On Thu, Apr 17, 2014 at 07:33:16AM -0400, Martin Langhoff wrote:
On Wed, Apr 16, 2014 at 5:08 PM, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
So I'll ask you about this other aspect -- what about stateless clients with very limited or no local storage?
Not supported by this, unfortunately. There needs to be at least temporary storage in tmpfs for this scheme to work.
Can the storage space used be limited with a parameter to avoid ENOSPC? Can journald be told to start discarding the data (oldest or latest) when it reaches the limit?
Yeah, it's standard journald behaviour. By default journald will limit both the total disk usage, and will also not fill the fs above 90% (iirc). There were various issues with this logic in the past, but it seems to behave fairly robustly now. The idea is that journald does the "right thing" without any configuration or maintenance. The logic is suitable for normal fs layout, but if something different is needed, especially if you have a dedicated fs for log storage that can be filled to the brim, the limits and thresholds are all configurable (see http://www.freedesktop.org/software/systemd/man/journald.conf.html#SystemMax... for available options).
Zbyszek
So, this change went to fesco last week, but there were some questions/issues around it. Could change owners respond to:
1) sgallagh wasn't sure this was a self contained change: see: https://fedorahosted.org/fesco/ticket/1250#comment:19
2) FESCo in general wondered if we advertised this as a change if people would see it as the recommended/default way to handle remote system logs. Is it planned to be that, or is it just a 'here's a preview of how we hope to do this down the road'?
3) There were general concerns around the protocol/setup... but I think those were raised before in this thread. Is there any revisiting of the protocol/etc planned? Or things are pretty set at this point?
kevin
On Wed, Apr 30, 2014 at 12:07:25PM -0600, Kevin Fenzi wrote: Hi, sorry for the late reply, I'm away on a workshop...
So, this change went to fesco last week, but there were some questions/issues around it. Could change owners respond to:
- sgallagh wasn't sure this was a self contained change:
sgallagh wrote "I don't consider this to be a self-contained change. If it goes into effect, we (FESCo) are basically stating that this a recommended way for users to gather journald logs in a central location"
I don't know where this conclusion comes from. This change is about adding some new software... When e.g. Ganesha is proposed as a Change, it doesn't that it is the recommended network filesystem implementation.
I am open to adding some text making it explicit that this is a new technology and probably only suitable for some use cases.
He also writes "particularly since it appears to have been developed without the input of the journald creators". The code in question has been reviewed on the systemd mailing list, and discussed internally. Also, although I didn't have anything to do with initial journald creation, I have been one the people handling bugs and adding features to it over the last two years.
- FESCo in general wondered if we advertised this as a change if
people would see it as the recommended/default way to handle remote system logs. Is it planned to be that, or is it just a 'here's a preview of how we hope to do this down the road'?
The latter. Not that I think it will not work, but making it the default/recommended thing when it's not even written yet seems premature.
- There were general concerns around the protocol/setup... but I think
those were raised before in this thread. Is there any revisiting of the protocol/etc planned? Or things are pretty set at this point?
This was proposed as a Change exactly for the purpose of gathering feedback, even of the fargoing kind like that. There's lot of merit in the proposed protocol modifications, but it's a fairly big modification, and I haven't had time to properly think about it.
Zbyszek
On Thu, 1 May 2014 16:40:48 +0200 Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
...snip...
This was proposed as a Change exactly for the purpose of gathering feedback, even of the fargoing kind like that. There's lot of merit in the proposed protocol modifications, but it's a fairly big modification, and I haven't had time to properly think about it.
Thanks! I will pass this back to fesco and see if that answers any questions folks had.
kevin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 05/01/2014 10:40 AM, Zbigniew Jędrzejewski-Szmek wrote:
On Wed, Apr 30, 2014 at 12:07:25PM -0600, Kevin Fenzi wrote: Hi, sorry for the late reply, I'm away on a workshop...
So, this change went to fesco last week, but there were some questions/issues around it. Could change owners respond to:
- sgallagh wasn't sure this was a self contained change: see:
sgallagh wrote "I don't consider this to be a self-contained change. If it goes into effect, we (FESCo) are basically stating that this a recommended way for users to gather journald logs in a central location"
I don't know where this conclusion comes from. This change is about adding some new software... When e.g. Ganesha is proposed as a Change, it doesn't that it is the recommended network filesystem implementation.
It's kind of implicit in the Change proposal. When you submit a Change, you are indicating that you want this to be something that Fedora promotes (both from an engineering standpoint and a marketing one).
With something like Ganesha, in part because there are other network filesystem implementations that are well-known, it's fairly easily understood that this is an additive change.
However, with remote journal logging, there currently exists no mechanism for this (except for dumping syslog, which loses all of the enhanced capabilities of journald)
I am open to adding some text making it explicit that this is a new technology and probably only suitable for some use cases.
That would make this much more palatable, thank you.
He also writes "particularly since it appears to have been developed without the input of the journald creators". The code in question has been reviewed on the systemd mailing list, and discussed internally. Also, although I didn't have anything to do with initial journald creation, I have been one the people handling bugs and adding features to it over the last two years.
This was an impression I got from the discussion thread up to that point. I didn't get a sense that the core journald developers were in agreement with this as the approach (and as noted above, it appeared that this was being pushed as the One True Way to do this).
- FESCo in general wondered if we advertised this as a change
if people would see it as the recommended/default way to handle remote system logs. Is it planned to be that, or is it just a 'here's a preview of how we hope to do this down the road'?
The latter. Not that I think it will not work, but making it the default/recommended thing when it's not even written yet seems premature.
Right, as noted above we should be clear about this when we talk about it.
- There were general concerns around the protocol/setup... but I
think those were raised before in this thread. Is there any revisiting of the protocol/etc planned? Or things are pretty set at this point?
This was proposed as a Change exactly for the purpose of gathering feedback, even of the fargoing kind like that. There's lot of merit in the proposed protocol modifications, but it's a fairly big modification, and I haven't had time to properly think about it.
So the obvious question here is whether we should accept this as a Change now. Given that you are contemplating a significant protocol modification, would it make more sense to defer this Change until F22? (that doesn't mean you can't have the code in Fedora, just that we won't market it and push people towards testing it in F21).
On Fri, 02.05.14 10:53, Stephen Gallagher (sgallagh@redhat.com) wrote:
He also writes "particularly since it appears to have been developed without the input of the journald creators". The code in question has been reviewed on the systemd mailing list, and discussed internally. Also, although I didn't have anything to do with initial journald creation, I have been one the people handling bugs and adding features to it over the last two years.
This was an impression I got from the discussion thread up to that point. I didn't get a sense that the core journald developers were in agreement with this as the approach (and as noted above, it appeared that this was being pushed as the One True Way to do this).
Wut?
I think Zibigniew and I are actually very much on the same page. Zbigniew is a systemd commiter, I trust him, and I think it's good stuff he is doing. We have discussed the protocol choices many times within the systemd community and we are all behind this approach.
I am pretty sure HTTP(s) is the right choice, and that we cover both push and pull models, and Zbigniew's brings us a good step closer to that goal, even if it might not bring us all the way there yet.
Lennart
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 05/02/2014 11:40 AM, Lennart Poettering wrote:
On Fri, 02.05.14 10:53, Stephen Gallagher (sgallagh@redhat.com) wrote:
He also writes "particularly since it appears to have been developed without the input of the journald creators". The code in question has been reviewed on the systemd mailing list, and discussed internally. Also, although I didn't have anything to do with initial journald creation, I have been one the people handling bugs and adding features to it over the last two years.
This was an impression I got from the discussion thread up to that point. I didn't get a sense that the core journald developers were in agreement with this as the approach (and as noted above, it appeared that this was being pushed as the One True Way to do this).
Wut?
I think Zibigniew and I are actually very much on the same page. Zbigniew is a systemd commiter, I trust him, and I think it's good stuff he is doing. We have discussed the protocol choices many times within the systemd community and we are all behind this approach.
I am pretty sure HTTP(s) is the right choice, and that we cover both push and pull models, and Zbigniew's brings us a good step closer to that goal, even if it might not bring us all the way there yet.
Sorry, I wasn't sufficiently clear when I said "up to that point". I no longer feel that way.
I am pretty sure HTTP(s) is the right choice
Hi Lennart,
The choice of HTTPS does complicate the network infrastructure moving log records into a "network management" QoS class (ie, making sure that remote logging works during a DoS attack caused by malware).
If you feel that HTTPS is the correct protocol then please consider using another port number than 443.
Cheers, glen
On Mon, 05.05.14 18:54, Glen Turner (gdt@gdt.id.au) wrote:
I am pretty sure HTTP(s) is the right choice
Hi Lennart,
The choice of HTTPS does complicate the network infrastructure moving log records into a "network management" QoS class (ie, making sure that remote logging works during a DoS attack caused by malware).
If you feel that HTTPS is the correct protocol then please consider using another port number than 443.
It's port 19531 by default.
Lennart
On 05/05/2014 12:36 PM, Lennart Poettering wrote:
If you feel that HTTPS is the correct protocol then please consider using another port number than 443.
It's port 19531 by default.
Have you requested official assignment? It's not yet listed in http://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.txt.
On Mon, 05.05.14 12:49, Florian Weimer (fweimer@redhat.com) wrote:
On 05/05/2014 12:36 PM, Lennart Poettering wrote:
If you feel that HTTPS is the correct protocol then please consider using another port number than 443.
It's port 19531 by default.
Have you requested official assignment? It's not yet listed in http://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.txt.
No I have not. What's the benefit of doing so?
Lennart
On 05/05/2014 12:52 PM, Lennart Poettering wrote:
Have you requested official assignment? It's not yet listed in http://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.txt.
No I have not. What's the benefit of doing so?
It avoids collisions, and tools like tcpdump (including third-party software we do not control) are more likely to label the traffic as related to systemd logging.
On Fri, May 02, 2014 at 10:53:59AM -0400, Stephen Gallagher wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 05/01/2014 10:40 AM, Zbigniew Jędrzejewski-Szmek wrote:
On Wed, Apr 30, 2014 at 12:07:25PM -0600, Kevin Fenzi wrote: Hi, sorry for the late reply, I'm away on a workshop...
So, this change went to fesco last week, but there were some questions/issues around it. Could change owners respond to:
- sgallagh wasn't sure this was a self contained change: see:
sgallagh wrote "I don't consider this to be a self-contained change. If it goes into effect, we (FESCo) are basically stating that this a recommended way for users to gather journald logs in a central location"
I don't know where this conclusion comes from. This change is about adding some new software... When e.g. Ganesha is proposed as a Change, it doesn't that it is the recommended network filesystem implementation.
It's kind of implicit in the Change proposal. When you submit a Change, you are indicating that you want this to be something that Fedora promotes (both from an engineering standpoint and a marketing one).
I modifed the Change text to indicate that it is an alternative to exisiting solutions, and can be used in parallel.
He also writes "particularly since it appears to have been developed without the input of the journald creators". The code in question has been reviewed on the systemd mailing list, and discussed internally. Also, although I didn't have anything to do with initial journald creation, I have been one the people handling bugs and adding features to it over the last two years.
This was an impression I got from the discussion thread up to that point. I didn't get a sense that the core journald developers were in agreement with this as the approach (and as noted above, it appeared that this was being pushed as the One True Way to do this).
OK.
- FESCo in general wondered if we advertised this as a change
if people would see it as the recommended/default way to handle remote system logs. Is it planned to be that, or is it just a 'here's a preview of how we hope to do this down the road'?
The latter. Not that I think it will not work, but making it the default/recommended thing when it's not even written yet seems premature.
Right, as noted above we should be clear about this when we talk about it.
- There were general concerns around the protocol/setup... but I
think those were raised before in this thread. Is there any revisiting of the protocol/etc planned? Or things are pretty set at this point?
This was proposed as a Change exactly for the purpose of gathering feedback, even of the fargoing kind like that. There's lot of merit in the proposed protocol modifications, but it's a fairly big modification, and I haven't had time to properly think about it.
So the obvious question here is whether we should accept this as a Change now. Given that you are contemplating a significant protocol modification, would it make more sense to defer this Change until F22? (that doesn't mean you can't have the code in Fedora, just that we won't market it and push people towards testing it in F21).
It's still a couple of months which should be enough to implement the protocol change before F21. I would like to keep the Change for F21.
Zbyszek
On Fri, 2014-05-02 at 17:51 +0200, Zbigniew Jędrzejewski-Szmek wrote:
It's kind of implicit in the Change proposal. When you submit a Change, you are indicating that you want this to be something that Fedora promotes (both from an engineering standpoint and a marketing one).
I modifed the Change text to indicate that it is an alternative to exisiting solutions, and can be used in parallel.
Just as a note for future messaging purposes: the Server product's tech spec explicitly nominates rsyslog as the supported remote logging mechanism for Server. We did explicitly choose between rsyslog and the new native systemd capability, and chose rsyslog at least until the native systemd capability is a bit more mature.
No specific immediate impact or 'required action' here, just wanted to note it for anyone following this discussion for purposes of future public communication.
devel@lists.stg.fedoraproject.org
-
Adam Williamson -
Alexander Bokovoy -
Bill Nottingham -
Chris Adams -
Florian Weimer -
Frank Ch. Eigler -
Glen Turner -
Jaroslav Reznik -
Kevin Fenzi -
Lennart Poettering -
Marc Deop Argemí -
Martin Langhoff -
Matthew Miller -
Miloslav Trmač -
Simo Sorce -
Stephen Gallagher -
Zbigniew Jędrzejewski-Szmek