Recently a number of bugs [1-5] have come up regarding the new default Kerberos Ccache location that we changed according to [6].
We originally thought that reusing the same directory used by XDG_USER_DIR was a good idea as systemd/logind would pre-create it for us and we'd all be happy.
Unfortunately it turns out there a re a number of cases where the directory is not pre-created for us (sshd, sudo, su) and a number of cases where even if we created it out of systemd/logind supervision we'd risk it being yanked from under the process that is using it as by default systemd/logind uses a refcount system to know when to remove it.
I have an idea that can solve the problem relatively easily. Create a small dbus program (reuse oddjob ?) that will be called by libkrb5 to request creation of the credential cache. This would be a small Fedora19 patch to libkrb5, I do not think upstream would like it in this form (more on it later)[*].
The dbus program would simply get the unix cred structure of the calling application via dbus services and unconditionally create /var/kerberos/user/%uidnumber/krbcc[**] directory based exclusively on the uid number of the peer and a system configured template, it would also create a symlink in /run/user/%uidnumber/krbcc for backwards compatibility in Fedora 19 only, and we transition completely to the new dir in F20.
Why a new dir ? Because we do not want systemd/logind to yank the directory under us. There are a number of cases where it would be beneficial to keep it around (example a cron job that starts every 10 minutes and uses cached credentials valid for hours to do some task).
Simo.
[1] https://bugzilla.redhat.com/961235 - Credential cache directory /run/user/0/krb5cc does not exist [2] https://bugzilla.redhat.com/977972 - kinit: Credential cache directory /run/user/0/krb5cc does not exist while getting default ccache [3] https://bugzilla.redhat.com/904720 - ipa-adtrust-install fails unexpectedly [4] https://bugzilla.redhat.com/796429 - sssd and kerberos should change the default location for create the Credential Caches to /run/usr/USERNAME/krb5cc [5] https://bugzilla.redhat.com/987792 - KRB5CCNAME is not set in PAM session with GSSAPI SSH auth [6] https://fedoraproject.org/wiki/Features/KRB5CacheMove
[*] I asked the MIT Kerberos team a while ago if we can make the cache type pluggable or revive the project to build a KCM (Kerberos Cache Manager) so we could decide to implement the cache manager functionality later, and there is interest. So that would be my long term strategy.
[**] I am flexible about where the dir should reside, if we want to keep tmpfs behavior we can put it under /run/kerberos/user
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/26/2013 10:48 AM, Simo Sorce wrote:
Recently a number of bugs [1-5] have come up regarding the new default Kerberos Ccache location that we changed according to [6].
We originally thought that reusing the same directory used by XDG_USER_DIR was a good idea as systemd/logind would pre-create it for us and we'd all be happy.
Unfortunately it turns out there a re a number of cases where the directory is not pre-created for us (sshd, sudo, su) and a number of cases where even if we created it out of systemd/logind supervision we'd risk it being yanked from under the process that is using it as by default systemd/logind uses a refcount system to know when to remove it.
I have an idea that can solve the problem relatively easily. Create a small dbus program (reuse oddjob ?) that will be called by libkrb5 to request creation of the credential cache. This would be a small Fedora19 patch to libkrb5, I do not think upstream would like it in this form (more on it later)[*].
The dbus program would simply get the unix cred structure of the calling application via dbus services and unconditionally create /var/kerberos/user/%uidnumber/krbcc[**] directory based exclusively on the uid number of the peer and a system configured template, it would also create a symlink in /run/user/%uidnumber/krbcc for backwards compatibility in Fedora 19 only, and we transition completely to the new dir in F20.
I'm CCing Lennart and Kay on the discussion about this. Creating this symlink in the oddjob would be somewhat error-prone. I'd rather that we ask logind to carry a patch just for F19 where the creation of XDG_RUNTIME_DIR would include this symlink.
I think this would be better behavior, since if the /run/user/UID directory doesn't exist before we create our new /var/kerberos/user/%uidnumber/krbcc version then the symlink won't be created and cannot be recognized later if the /run/user/UID directory appears.
Lennart/Kay, would that be an acceptable hack for you (just within F19)?
Why a new dir ? Because we do not want systemd/logind to yank the directory under us. There are a number of cases where it would be beneficial to keep it around (example a cron job that starts every 10 minutes and uses cached credentials valid for hours to do some task).
Simo.
[1] https://bugzilla.redhat.com/961235 - Credential cache directory /run/user/0/krb5cc does not exist [2] https://bugzilla.redhat.com/977972 - kinit: Credential cache directory /run/user/0/krb5cc does not exist while getting default ccache [3] https://bugzilla.redhat.com/904720 - ipa-adtrust-install fails unexpectedly [4] https://bugzilla.redhat.com/796429 - sssd and kerberos should change the default location for create the Credential Caches to /run/usr/USERNAME/krb5cc [5] https://bugzilla.redhat.com/987792 - KRB5CCNAME is not set in PAM session with GSSAPI SSH auth [6] https://fedoraproject.org/wiki/Features/KRB5CacheMove
[*] I asked the MIT Kerberos team a while ago if we can make the cache type pluggable or revive the project to build a KCM (Kerberos Cache Manager) so we could decide to implement the cache manager functionality later, and there is interest. So that would be my long term strategy.
[**] I am flexible about where the dir should reside, if we want to keep tmpfs behavior we can put it under /run/kerberos/user
I'd prefer to keep the tmpfs-by-default behavior because it prevents the theft of credentials from a stolen hard drive. Users can always modify the location they want to use by using the KRB5CCNAME environment variable and various configuration options (such as krb5_ccname_template in SSSD) to select a persistent location if they choose to.
For the record, I like this plan. It should also serve to address a number of unfortunate edge-cases, particularly those around the semi-sessions created by 'su' and 'sudo'.
On Fri, 2013-07-26 at 11:01 -0400, Stephen Gallagher wrote:
On 07/26/2013 10:48 AM, Simo Sorce wrote:
Recently a number of bugs [1-5] have come up regarding the new default Kerberos Ccache location that we changed according to [6].
We originally thought that reusing the same directory used by XDG_USER_DIR was a good idea as systemd/logind would pre-create it for us and we'd all be happy.
Unfortunately it turns out there a re a number of cases where the directory is not pre-created for us (sshd, sudo, su) and a number of cases where even if we created it out of systemd/logind supervision we'd risk it being yanked from under the process that is using it as by default systemd/logind uses a refcount system to know when to remove it.
I have an idea that can solve the problem relatively easily. Create a small dbus program (reuse oddjob ?) that will be called by libkrb5 to request creation of the credential cache. This would be a small Fedora19 patch to libkrb5, I do not think upstream would like it in this form (more on it later)[*].
The dbus program would simply get the unix cred structure of the calling application via dbus services and unconditionally create /var/kerberos/user/%uidnumber/krbcc[**] directory based exclusively on the uid number of the peer and a system configured template, it would also create a symlink in /run/user/%uidnumber/krbcc for backwards compatibility in Fedora 19 only, and we transition completely to the new dir in F20.
I'm CCing Lennart and Kay on the discussion about this. Creating this symlink in the oddjob would be somewhat error-prone. I'd rather that we ask logind to carry a patch just for F19 where the creation of XDG_RUNTIME_DIR would include this symlink.
I think this would be better behavior, since if the /run/user/UID directory doesn't exist before we create our new /var/kerberos/user/%uidnumber/krbcc version then the symlink won't be created and cannot be recognized later if the /run/user/UID directory appears.
Lennart/Kay, would that be an acceptable hack for you (just within F19)?
Why a new dir ? Because we do not want systemd/logind to yank the directory under us. There are a number of cases where it would be beneficial to keep it around (example a cron job that starts every 10 minutes and uses cached credentials valid for hours to do some task).
Simo.
[1] https://bugzilla.redhat.com/961235 - Credential cache directory /run/user/0/krb5cc does not exist [2] https://bugzilla.redhat.com/977972 - kinit: Credential cache directory /run/user/0/krb5cc does not exist while getting default ccache [3] https://bugzilla.redhat.com/904720 - ipa-adtrust-install fails unexpectedly [4] https://bugzilla.redhat.com/796429 - sssd and kerberos should change the default location for create the Credential Caches to /run/usr/USERNAME/krb5cc [5] https://bugzilla.redhat.com/987792 - KRB5CCNAME is not set in PAM session with GSSAPI SSH auth [6] https://fedoraproject.org/wiki/Features/KRB5CacheMove
[*] I asked the MIT Kerberos team a while ago if we can make the cache type pluggable or revive the project to build a KCM (Kerberos Cache Manager) so we could decide to implement the cache manager functionality later, and there is interest. So that would be my long term strategy.
[**] I am flexible about where the dir should reside, if we want to keep tmpfs behavior we can put it under /run/kerberos/user
I'd prefer to keep the tmpfs-by-default behavior because it prevents the theft of credentials from a stolen hard drive. Users can always modify the location they want to use by using the KRB5CCNAME environment variable and various configuration options (such as krb5_ccname_template in SSSD) to select a persistent location if they choose to.
For the record, I like this plan. It should also serve to address a number of unfortunate edge-cases, particularly those around the semi-sessions created by 'su' and 'sudo'.
I'd rather like to see a plan that would fix also other similar uses off /run/user/<uid> and not just the Kerberos. Unfortunately Lennart insists otherwise so I am afraid that this is doomed to fail and hacks like this one above will prevail.
On Fri, 2013-07-26 at 17:07 +0200, Tomas Mraz wrote:
For the record, I like this plan. It should also serve to address a number of unfortunate edge-cases, particularly those around the semi-sessions created by 'su' and 'sudo'.
I'd rather like to see a plan that would fix also other similar uses off /run/user/<uid> and not just the Kerberos. Unfortunately Lennart insists otherwise so I am afraid that this is doomed to fail and hacks like this one above will prevail.
While I'd like to have a solution for the world, I really want to fix the kerberos issue now. We can transition later to a better way to do this, but I do not think we can live with these bugs for long while we wait for the perfect solution.
In other words consider this a testdrive, if it works you have a blueprint for how to fix the problem, if not we screwed only one subsystem and the damage is smaller.
Simo.
-- Tomas Mraz No matter how far down the wrong road you've gone, turn back. Turkish proverb
On Fri, Jul 26, 2013 at 5:17 PM, Simo Sorce simo@redhat.com wrote:
On Fri, 2013-07-26 at 17:07 +0200, Tomas Mraz wrote:
For the record, I like this plan. It should also serve to address a number of unfortunate edge-cases, particularly those around the semi-sessions created by 'su' and 'sudo'.
I'd rather like to see a plan that would fix also other similar uses off /run/user/<uid> and not just the Kerberos. Unfortunately Lennart insists otherwise so I am afraid that this is doomed to fail and hacks like this one above will prevail.
While I'd like to have a solution for the world, I really want to fix the kerberos issue now. We can transition later to a better way to do this, but I do not think we can live with these bugs for long while we wait for the perfect solution.
If the expectation is that the cache may be moving around again once or twice, should we be thinking about hiding/deemphasizing the location from users so that they aren't troubled by the moves? Or perhaps the exact opposite - make KRB5CCNAME prominent to give everybody a working solution until we get something that works well in the default case (... while possibly limiting our flexibility for the future)? Mirek
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/26/2013 11:31 AM, Miloslav Trmač wrote:
On Fri, Jul 26, 2013 at 5:17 PM, Simo Sorce simo@redhat.com wrote:
On Fri, 2013-07-26 at 17:07 +0200, Tomas Mraz wrote:
For the record, I like this plan. It should also serve to address a number of unfortunate edge-cases, particularly those around the semi-sessions created by 'su' and 'sudo'.
I'd rather like to see a plan that would fix also other similar uses off /run/user/<uid> and not just the Kerberos. Unfortunately Lennart insists otherwise so I am afraid that this is doomed to fail and hacks like this one above will prevail.
While I'd like to have a solution for the world, I really want to fix the kerberos issue now. We can transition later to a better way to do this, but I do not think we can live with these bugs for long while we wait for the perfect solution.
If the expectation is that the cache may be moving around again once or twice, should we be thinking about hiding/deemphasizing the location from users so that they aren't troubled by the moves? Or perhaps the exact opposite - make KRB5CCNAME prominent to give everybody a working solution until we get something that works well in the default case (... while possibly limiting our flexibility for the future)?
Well, *users* should not really notice or care where these things are unless they go looking for them. All a user cares about is whether they can authenticate and have access to their SSO credentials after doing so.
Moving this location around really impacts only packagers and certain developers, which is a different audience (and one that's usually more accepting of change, provided that the benefits are clearly communicated). In this case, we've gone from "Ugly, works most of the time but very fragile" (/tmp) to "When it works, it's nearly perfect, but it is broken for some very important use-cases" (/run/user/UID) and now we're trying for "Works for everyone without the fragility of the original approach." (/run/kerberos/UID)
The middle step was necessary - I think - in order to get us to the place we really wanted to be. I agree with Simo that it would be nice to have a solution that could solve all problems for every subsystem, but at the moment we really need to fix this one because it's causing problems for a lot of people.
Also, there's no guarantee that a perfect solution for other subsystems will ever actually show up. It may be that each service/application will have its own needs that have to be met separately. We can investigate this, but I don't think it should hold up the short-term fix.
Also, Tomas, you stated: "I'd rather like to see a plan that would fix also other similar uses off /run/user/<uid> and not just the Kerberos.". Can you point us at some places that have similar issues? I have no doubt they exist, but they're not on my radar right now and I'd like to keep track of them.
On Fri, 2013-07-26 at 12:22 -0400, Stephen Gallagher wrote:
Well, *users* should not really notice or care where these things are unless they go looking for them. All a user cares about is whether they can authenticate and have access to their SSO credentials after doing so.
Moving this location around really impacts only packagers and certain developers, which is a different audience (and one that's usually more accepting of change, provided that the benefits are clearly communicated). In this case, we've gone from "Ugly, works most of the time but very fragile" (/tmp) to "When it works, it's nearly perfect, but it is broken for some very important use-cases" (/run/user/UID) and now we're trying for "Works for everyone without the fragility of the original approach." (/run/kerberos/UID)
The middle step was necessary - I think - in order to get us to the place we really wanted to be. I agree with Simo that it would be nice to have a solution that could solve all problems for every subsystem, but at the moment we really need to fix this one because it's causing problems for a lot of people.
Also, there's no guarantee that a perfect solution for other subsystems will ever actually show up. It may be that each service/application will have its own needs that have to be met separately. We can investigate this, but I don't think it should hold up the short-term fix.
Also, Tomas, you stated: "I'd rather like to see a plan that would fix also other similar uses off /run/user/<uid> and not just the Kerberos.". Can you point us at some places that have similar issues? I have no doubt they exist, but they're not on my radar right now and I'd like to keep track of them.
See the lengthy related discussion here https://bugzilla.redhat.com/show_bug.cgi?id=753882
On Fri, 2013-07-26 at 11:01 -0400, Stephen Gallagher wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/26/2013 10:48 AM, Simo Sorce wrote:
Recently a number of bugs [1-5] have come up regarding the new default Kerberos Ccache location that we changed according to [6].
We originally thought that reusing the same directory used by XDG_USER_DIR was a good idea as systemd/logind would pre-create it for us and we'd all be happy.
Unfortunately it turns out there a re a number of cases where the directory is not pre-created for us (sshd, sudo, su) and a number of cases where even if we created it out of systemd/logind supervision we'd risk it being yanked from under the process that is using it as by default systemd/logind uses a refcount system to know when to remove it.
I have an idea that can solve the problem relatively easily. Create a small dbus program (reuse oddjob ?) that will be called by libkrb5 to request creation of the credential cache. This would be a small Fedora19 patch to libkrb5, I do not think upstream would like it in this form (more on it later)[*].
The dbus program would simply get the unix cred structure of the calling application via dbus services and unconditionally create /var/kerberos/user/%uidnumber/krbcc[**] directory based exclusively on the uid number of the peer and a system configured template, it would also create a symlink in /run/user/%uidnumber/krbcc for backwards compatibility in Fedora 19 only, and we transition completely to the new dir in F20.
I'm CCing Lennart and Kay on the discussion about this. Creating this symlink in the oddjob would be somewhat error-prone. I'd rather that we ask logind to carry a patch just for F19 where the creation of XDG_RUNTIME_DIR would include this symlink.
I think this would be better behavior, since if the /run/user/UID directory doesn't exist before we create our new /var/kerberos/user/%uidnumber/krbcc version then the symlink won't be created and cannot be recognized later if the /run/user/UID directory appears.
Ah yeah this would be even better.
Lennart/Kay, would that be an acceptable hack for you (just within F19)?
Why a new dir ? Because we do not want systemd/logind to yank the directory under us. There are a number of cases where it would be beneficial to keep it around (example a cron job that starts every 10 minutes and uses cached credentials valid for hours to do some task).
Simo.
[1] https://bugzilla.redhat.com/961235 - Credential cache directory /run/user/0/krb5cc does not exist [2] https://bugzilla.redhat.com/977972 - kinit: Credential cache directory /run/user/0/krb5cc does not exist while getting default ccache [3] https://bugzilla.redhat.com/904720 - ipa-adtrust-install fails unexpectedly [4] https://bugzilla.redhat.com/796429 - sssd and kerberos should change the default location for create the Credential Caches to /run/usr/USERNAME/krb5cc [5] https://bugzilla.redhat.com/987792 - KRB5CCNAME is not set in PAM session with GSSAPI SSH auth [6] https://fedoraproject.org/wiki/Features/KRB5CacheMove
[*] I asked the MIT Kerberos team a while ago if we can make the cache type pluggable or revive the project to build a KCM (Kerberos Cache Manager) so we could decide to implement the cache manager functionality later, and there is interest. So that would be my long term strategy.
[**] I am flexible about where the dir should reside, if we want to keep tmpfs behavior we can put it under /run/kerberos/user
I'd prefer to keep the tmpfs-by-default behavior because it prevents the theft of credentials from a stolen hard drive. Users can always modify the location they want to use by using the KRB5CCNAME environment variable and various configuration options (such as krb5_ccname_template in SSSD) to select a persistent location if they choose to.
Admins can also add a tempfiles.d script to create a symlink to a stable location only for those user's they want to keep a permanent location.
So I am fine with tmpfs.
For the record, I like this plan. It should also serve to address a number of unfortunate edge-cases, particularly those around the semi-sessions created by 'su' and 'sudo'.
Yep.
Simo.
On Fri, 26.07.13 11:01, Stephen Gallagher (sgallagh@redhat.com) wrote:
I'm CCing Lennart and Kay on the discussion about this. Creating this symlink in the oddjob would be somewhat error-prone. I'd rather that we ask logind to carry a patch just for F19 where the creation of XDG_RUNTIME_DIR would include this symlink.
Stephen, so Kay convinced me that it might be OK to handle su/sudo and su -l/sudo -i differently from the rest and accept that we have sessions that are started out of other sessions. This would mean you would get your new cred cache dir on "sudo -i", but not on "sudo". This would come with a number of uglinesses though, for example the session IDs (as returned by $XDG_SESSION_ID or listed by loginctl) would be in two different namespaces for "real logins" and for these su-l/sudo-i logins (the real ones would be named "1", "2", "3" and so on, the su-l/sudo-i ones "c1", "c2", "c3").
If this is acceptable, then this should probably look like this:
pam_systemd would need a new argument force-new=1 or so, which is passed for "su -l" and "sudo -i" but not otherwise. THis would be passed along CreateSession() to logind. When specified this would result in a session to be created independent of the existing one. Special care needs to be taken that this will not inherit anything like the seat or VT assignment or other session credentials from the original session.
Then, then PAM krb module would pre-create XDG_RUNTIME_DIR via mkdir() when it initializes and logind would take this over later on. When the user finally logs out the dir would be removed by logind, as usual.
I am not keen on this thing, because it introduces two different kinds of sessions ("real ones" and "child sessions", and detaches sessions from other definitions of sessions such as the audit one), but, well, if that's what it takes...
Anyway, I'd to take a patch if that solution is OK for you.
This doesn't require changes to the XDG_RUNTIME_DIR spec, since we just shift around what a session is, the existance of XDG_RUNTIME_DIR is then just effect of that.
Lennart
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/26/2013 01:40 PM, Lennart Poettering wrote:
On Fri, 26.07.13 11:01, Stephen Gallagher (sgallagh@redhat.com) wrote:
I'm CCing Lennart and Kay on the discussion about this. Creating this symlink in the oddjob would be somewhat error-prone. I'd rather that we ask logind to carry a patch just for F19 where the creation of XDG_RUNTIME_DIR would include this symlink.
Stephen, so Kay convinced me that it might be OK to handle su/sudo and su -l/sudo -i differently from the rest and accept that we have sessions that are started out of other sessions. This would mean you would get your new cred cache dir on "sudo -i", but not on "sudo". This would come with a number of uglinesses though, for example the session IDs (as returned by $XDG_SESSION_ID or listed by loginctl) would be in two different namespaces for "real logins" and for these su-l/sudo-i logins (the real ones would be named "1", "2", "3" and so on, the su-l/sudo-i ones "c1", "c2", "c3").
If this is acceptable, then this should probably look like this:
pam_systemd would need a new argument force-new=1 or so, which is passed for "su -l" and "sudo -i" but not otherwise. THis would be passed along CreateSession() to logind. When specified this would result in a session to be created independent of the existing one. Special care needs to be taken that this will not inherit anything like the seat or VT assignment or other session credentials from the original session.
Then, then PAM krb module would pre-create XDG_RUNTIME_DIR via mkdir() when it initializes and logind would take this over later on. When the user finally logs out the dir would be removed by logind, as usual.
I am not keen on this thing, because it introduces two different kinds of sessions ("real ones" and "child sessions", and detaches sessions from other definitions of sessions such as the audit one), but, well, if that's what it takes...
Anyway, I'd to take a patch if that solution is OK for you.
This doesn't require changes to the XDG_RUNTIME_DIR spec, since we just shift around what a session is, the existance of XDG_RUNTIME_DIR is then just effect of that.
First of all, Lennart: I really appreciate your willingness to meet half-way and try to get this to work. It's a difficult problem with a lot of edge-cases and no easy solutions. So thank you.
This would help us out on the 'su -l' and 'sudo -l' cases, but it still potentially leaves us with two problems that I'm not sure how to solve (without breaking the XDG_RUNTIME_DIR lifecycle definition).
1) https://bugzilla.redhat.com/show_bug.cgi?id=987792 has a problem where a user is logging in through SSH with delegated Kerberos credentials and is using AFS. After login is complete, his new session is available with the credential cache dir in place, but unlike in Fedora 18 and earlier, the KRB5CCNAME variable is not available to other PAM modules prior to pam_systemd, meaning that his pam_afs_session.so does not get set up correctly. It's presumed that the reason for this is that SSH sees that it can't create the cache on disk yet, so it waits and does so later.
2) We still need to consider use-cases where a cron job or other long-running service needs to use credentials given to it by the user, though they are no longer signed in. With the current approach, we still need to be concerned that the /run/user/UID directory may just cease to exist if there are no more active sessions on the machine.
On Fri, 26.07.13 13:57, Stephen Gallagher (sgallagh@redhat.com) wrote:
This would help us out on the 'su -l' and 'sudo -i' cases, but it still potentially leaves us with two problems that I'm not sure how to solve (without breaking the XDG_RUNTIME_DIR lifecycle definition).
- https://bugzilla.redhat.com/show_bug.cgi?id=987792 has a problem
where a user is logging in through SSH with delegated Kerberos credentials and is using AFS. After login is complete, his new session is available with the credential cache dir in place, but unlike in Fedora 18 and earlier, the KRB5CCNAME variable is not available to other PAM modules prior to pam_systemd, meaning that his pam_afs_session.so does not get set up correctly. It's presumed that the reason for this is that SSH sees that it can't create the cache on disk yet, so it waits and does so later.
I am not sure I grok this.
So ssh is supposed to store the tickets in XDG_RUNTIME_DIR but you don't want to patch it to create the dir the same way as the krb PAM module does?
Doesn't SSH and the PAM module use the same code to store the tickets in the cache? This really sounds like something that could be shuffled out between krb, pam, sshd, no? Or am I totally not grokking this?
- We still need to consider use-cases where a cron job or other
long-running service needs to use credentials given to it by the user, though they are no longer signed in. With the current approach, we still need to be concerned that the /run/user/UID directory may just cease to exist if there are no more active sessions on the machine.
I am pretty sure we should be careful with leaving around runtime data of a user in /run if he's not logged in. He should not be able to consume resources unrestricted unless this is OK'ed explicitly by the admin.
logind knows the concept of "lingering" user. If a user is lingering the life-time of its runtime resources is from boot-time to shut-down, rather than only as long as he is logged in. The cron case sounds like something which could be handled with that. "loginctl enable-linger foobar" will enable this linger state for a specific user. DEfaults to off, only root can alter this.
Note that this might cause other resources of the user to stay around too, it's not just something that affects the life-cycle of XDG_RUNTIME_DIR and nothing else. For example, in the longer run this will also mean that the user may have user services running longer than just during the login.
Lennart
On Fri, 2013-07-26 at 20:15 +0200, Lennart Poettering wrote:
On Fri, 26.07.13 13:57, Stephen Gallagher (sgallagh@redhat.com) wrote:
This would help us out on the 'su -l' and 'sudo -i' cases, but it still potentially leaves us with two problems that I'm not sure how to solve (without breaking the XDG_RUNTIME_DIR lifecycle definition).
- https://bugzilla.redhat.com/show_bug.cgi?id=987792 has a problem
where a user is logging in through SSH with delegated Kerberos credentials and is using AFS. After login is complete, his new session is available with the credential cache dir in place, but unlike in Fedora 18 and earlier, the KRB5CCNAME variable is not available to other PAM modules prior to pam_systemd, meaning that his pam_afs_session.so does not get set up correctly. It's presumed that the reason for this is that SSH sees that it can't create the cache on disk yet, so it waits and does so later.
I am not sure I grok this.
So ssh is supposed to store the tickets in XDG_RUNTIME_DIR but you don't want to patch it to create the dir the same way as the krb PAM module does?
Doesn't SSH and the PAM module use the same code to store the tickets in the cache? This really sounds like something that could be shuffled out between krb, pam, sshd, no? Or am I totally not grokking this?
What I am proposing is that libkrb5 take care of this using a helper and get our of XDG_RUNTIME_DIR completely given lifetime of the ccache is not necessarily aligned with that of this directory.
so yeah if we go with my proposal we will have the right component do the right thing.
- We still need to consider use-cases where a cron job or other
long-running service needs to use credentials given to it by the user, though they are no longer signed in. With the current approach, we still need to be concerned that the /run/user/UID directory may just cease to exist if there are no more active sessions on the machine.
I am pretty sure we should be careful with leaving around runtime data of a user in /run if he's not logged in. He should not be able to consume resources unrestricted unless this is OK'ed explicitly by the admin.
Same as for /tmp right now, so I do not see much of a difference here.
logind knows the concept of "lingering" user. If a user is lingering the life-time of its runtime resources is from boot-time to shut-down, rather than only as long as he is logged in. The cron case sounds like something which could be handled with that. "loginctl enable-linger foobar" will enable this linger state for a specific user. DEfaults to off, only root can alter this.
We want this thing to work by default, having normal users to find out this lingering concept exist because operations that currently works start failing is already a big failure.
Note that this might cause other resources of the user to stay around too, it's not just something that affects the life-cycle of XDG_RUNTIME_DIR and nothing else. For example, in the longer run this will also mean that the user may have user services running longer than just during the login.
I am not sure I understand what's the point here, but I do not think it is relevant.
Simo.
On Fri, 26.07.13 14:20, Simo Sorce (simo@redhat.com) wrote:
We want this thing to work by default, having normal users to find out this lingering concept exist because operations that currently works start failing is already a big failure.
OK, this is the deal-breaker. The thing about XDG_RUNTIME_DIR is that it has a strict life-time. If you don't want that, then XDG_RUNTIME_DIR is simply not the tool for the job.
(That said, I don't agree with your requirement. Earlier you said you want to make /tmp poly-instantiated -- which is a goal I agree with -- which would mean that $HOME is the *only* persistent storage you have)
Lennart
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/26/2013 02:40 PM, Lennart Poettering wrote:
On Fri, 26.07.13 14:20, Simo Sorce (simo@redhat.com) wrote:
We want this thing to work by default, having normal users to find out this lingering concept exist because operations that currently works start failing is already a big failure.
OK, this is the deal-breaker. The thing about XDG_RUNTIME_DIR is that it has a strict life-time. If you don't want that, then XDG_RUNTIME_DIR is simply not the tool for the job.
(That said, I don't agree with your requirement. Earlier you said you want to make /tmp poly-instantiated -- which is a goal I agree with -- which would mean that $HOME is the *only* persistent storage you have)
Home isn't persistent in this situation, because in enterprise use-cases, $HOME is often contained on an NFS mount relying on Kerberos authentication. So we need to be able to keep the keying material somewhere else. This is why we were opting for /run or /var (with my preference being /run for the safety of purging the credentials on power-off).
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/26/2013 02:15 PM, Lennart Poettering wrote:
On Fri, 26.07.13 13:57, Stephen Gallagher (sgallagh@redhat.com) wrote:
This would help us out on the 'su -l' and 'sudo -i' cases, but it still potentially leaves us with two problems that I'm not sure how to solve (without breaking the XDG_RUNTIME_DIR lifecycle definition).
problem where a user is logging in through SSH with delegated Kerberos credentials and is using AFS. After login is complete, his new session is available with the credential cache dir in place, but unlike in Fedora 18 and earlier, the KRB5CCNAME variable is not available to other PAM modules prior to pam_systemd, meaning that his pam_afs_session.so does not get set up correctly. It's presumed that the reason for this is that SSH sees that it can't create the cache on disk yet, so it waits and does so later.
I am not sure I grok this.
So ssh is supposed to store the tickets in XDG_RUNTIME_DIR but you don't want to patch it to create the dir the same way as the krb PAM module does?
Doesn't SSH and the PAM module use the same code to store the tickets in the cache? This really sounds like something that could be shuffled out between krb, pam, sshd, no? Or am I totally not grokking this?
My point here was more to highlight that we're currently playing whack-a-mole to figure out which applications need access to the directory ahead of the session phase. This was representative in that it showed a situation where we needed to have access to it right from the beginning and available to the PAM environment itself.
Essentially, I'm saying we need to have a way to deal with creating it when it needs to be created, which Simo's proposal does. Having this helper binary attached to libkrb5 is one way to accomplish it. It's pretty close to the suggestion I made in a private thread with you a while ago where I asked for logind to offer us a public API to just request early creation. Since you expressed concerns with that use of XDG_RUNTIME_DIR, we've opted in this proposal to just create a separate directory that is carefully-purposed for only this use to avoid stopping on the toes of the session-based features.
- We still need to consider use-cases where a cron job or other
long-running service needs to use credentials given to it by the user, though they are no longer signed in. With the current approach, we still need to be concerned that the /run/user/UID directory may just cease to exist if there are no more active sessions on the machine.
I am pretty sure we should be careful with leaving around runtime data of a user in /run if he's not logged in. He should not be able to consume resources unrestricted unless this is OK'ed explicitly by the admin.
logind knows the concept of "lingering" user. If a user is lingering the life-time of its runtime resources is from boot-time to shut-down, rather than only as long as he is logged in. The cron case sounds like something which could be handled with that. "loginctl enable-linger foobar" will enable this linger state for a specific user. DEfaults to off, only root can alter this.
Note that this might cause other resources of the user to stay around too, it's not just something that affects the life-cycle of XDG_RUNTIME_DIR and nothing else. For example, in the longer run this will also mean that the user may have user services running longer than just during the login.
As Simo noted in the other thread, the availability of credentials outside the normal user session is an expectation of existing tools. The exposure here is significantly mitigated by the fact that Kerberos credentials are time-limited by the KDC.
On Fri, 26.07.13 14:32, Stephen Gallagher (sgallagh@redhat.com) wrote:
As Simo noted in the other thread, the availability of credentials outside the normal user session is an expectation of existing tools. The exposure here is significantly mitigated by the fact that Kerberos credentials are time-limited by the KDC.
So, let me get this right: you want a host-specific tmpfs location which is never automatically cleaned up, but is a private namespace of the user (though the system sometimes writes to it), correct?
That really sounds like a step backwards. Defining new runtime dirs without immediately thinking about life-cycles is something we really shouldn't do anymore.
XDG_RUNTIME_DIRS was introduced just because we want a clear life-cycle.
Lennart
PS: as a side note. what do you actually create in XDG_RUNTIME_DIR? A subdirectory? You are aware of the inherent risks of sharing a directory between system code and user code? It's extremely hard to properly get a subdir created in such a dir without opening a security hole.
PPS: if you give up on the unrestricted life-cycle and hence do still want to use XDG_RUNTIME_DIR, and you don't want to pre-create the dir on your own: you could just stick the cred cache into some PAM context var instead of writing it to XDG_RUNTIME_DIR right away, and then write it to the fs only at the very last step, long after pam_systemd set it up for you. sshd could place its creds there, and the PAM auth modules could add more into it, and then as last step you just flush all that was collected to the dir. This would be quite nice given that that way an aborted PAM sessions setup could never leave the half setup pre-created dir around.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/26/2013 02:52 PM, Lennart Poettering wrote:
On Fri, 26.07.13 14:32, Stephen Gallagher (sgallagh@redhat.com) wrote:
As Simo noted in the other thread, the availability of credentials outside the normal user session is an expectation of existing tools. The exposure here is significantly mitigated by the fact that Kerberos credentials are time-limited by the KDC.
So, let me get this right: you want a host-specific tmpfs location which is never automatically cleaned up, but is a private namespace of the user (though the system sometimes writes to it), correct?
That really sounds like a step backwards. Defining new runtime dirs without immediately thinking about life-cycles is something we really shouldn't do anymore.
XDG_RUNTIME_DIRS was introduced just because we want a clear life-cycle.
Lennart
PS: as a side note. what do you actually create in XDG_RUNTIME_DIR? A subdirectory? You are aware of the inherent risks of sharing a directory between system code and user code? It's extremely hard to properly get a subdir created in such a dir without opening a security hole.
Yes, we're creating a directory that will contain within it one or more file-based credential caches along with a special file that tells libkrb5 which one is the active default at a particular time. The actual cache directory is *never* created by system code. It's always generated by the libkrb5 in the user context. What *may* be created by the system is the enclosing directory. In the current situation, SSSD (as root) creates /run/user/UID and then libkrb5 creates /run/user/UID/krb5cc.
What we're looking for is a more generic way to have a root process create the containing directory (either /run/user/UID or /run/kerberos/UID as appropriate).
PPS: if you give up on the unrestricted life-cycle and hence do still want to use XDG_RUNTIME_DIR, and you don't want to pre-create the dir on your own: you could just stick the cred cache into some PAM context var instead of writing it to XDG_RUNTIME_DIR right away, and then write it to the fs only at the very last step, long after pam_systemd set it up for you. sshd could place its creds there, and the PAM auth modules could add more into it, and then as last step you just flush all that was collected to the dir. This would be quite nice given that that way an aborted PAM sessions setup could never leave the half setup pre-created dir around.
I talked about a similar approach on IRC today with Nalin and Ray Strode, but there's another edge-case problem there with pam_afs_session.so which will make a copy of the KRB5CCNAME env var during its execution (and consumes credentials from that location), so we can't just hold onto the creds and write them later, unfortunately.
<halfline_laptop> maybe the credentials should be stuffed away in a MEMORY credential cache type tucked away in an AUTHTOK for PAM modules to use until pam_open_session time <halfline_laptop> when it can get serialized to XDG_RUNTIME_DIR <halfline_laptop> i guess "rewrite all the pam modules" isn't a very effective answer <sgallagh> Well, they wouldn't necessarily need to be rewritten <sgallagh> As long as the KRB5CCNAME env var is set properly, they should just be using that. <sgallagh> I'm not sure how clean a transition we could make in pam_open_session, though <halfline_laptop> yea i guess the problem is afs or whatever would still be looking in the old cache after the transition <sgallagh> exactly <halfline_laptop> so we'd need to make sure the afs session module told afs about the new location <halfline_laptop> -> back to rewriting all the pam modules
On Fri, 26.07.13 15:03, Stephen Gallagher (sgallagh@redhat.com) wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/26/2013 02:52 PM, Lennart Poettering wrote:
On Fri, 26.07.13 14:32, Stephen Gallagher (sgallagh@redhat.com) wrote:
As Simo noted in the other thread, the availability of credentials outside the normal user session is an expectation of existing tools. The exposure here is significantly mitigated by the fact that Kerberos credentials are time-limited by the KDC.
So, let me get this right: you want a host-specific tmpfs location which is never automatically cleaned up, but is a private namespace of the user (though the system sometimes writes to it), correct?
That really sounds like a step backwards. Defining new runtime dirs without immediately thinking about life-cycles is something we really shouldn't do anymore.
XDG_RUNTIME_DIRS was introduced just because we want a clear life-cycle.
Lennart
PS: as a side note. what do you actually create in XDG_RUNTIME_DIR? A subdirectory? You are aware of the inherent risks of sharing a directory between system code and user code? It's extremely hard to properly get a subdir created in such a dir without opening a security hole.
Yes, we're creating a directory that will contain within it one or more file-based credential caches along with a special file that tells libkrb5 which one is the active default at a particular time. The actual cache directory is *never* created by system code. It's always generated by the libkrb5 in the user context. What *may* be created by the system is the enclosing directory. In the current situation, SSSD (as root) creates /run/user/UID and then libkrb5 creates /run/user/UID/krb5cc.
So, /run/user/$UID/krb5cc/ is a directroy you say is not created by system code, but by user code. Yet you say that you need this very early in the PAM chain, which however is in code that runs with full privileges as system code. This seems to contradict? Can you elaborate?
If you create /run/user/$UID/krb5cc/ from privileged code then it is very easy for unprivileged code to exploit that unless you are extra careful.
I talked about a similar approach on IRC today with Nalin and Ray Strode, but there's another edge-case problem there with pam_afs_session.so which will make a copy of the KRB5CCNAME env var during its execution (and consumes credentials from that location), so we can't just hold onto the creds and write them later, unfortunately.
Well, you could fix pam_afs_session.so, too, no? I mean this is all open source, and under your control. Instead of adding work-arounds to systemd/logind and friends it sounds pretty OK to fix pam_afs_session.so, ssh and the krb pam module instead.
<halfline_laptop> maybe the credentials should be stuffed away in a MEMORY credential cache type tucked away in an AUTHTOK for PAM modules to use until pam_open_session time <halfline_laptop> when it can get serialized to XDG_RUNTIME_DIR <halfline_laptop> i guess "rewrite all the pam modules" isn't a very effective answer <sgallagh> Well, they wouldn't necessarily need to be rewritten <sgallagh> As long as the KRB5CCNAME env var is set properly, they should just be using that. <sgallagh> I'm not sure how clean a transition we could make in pam_open_session, though <halfline_laptop> yea i guess the problem is afs or whatever would still be looking in the old cache after the transition <sgallagh> exactly <halfline_laptop> so we'd need to make sure the afs session module told afs about the new location <halfline_laptop> -> back to rewriting all the pam modules
Well, it's not all PAM modules, it's just the few which use kerberos tickets...
Lennart
On Sat, 2013-07-27 at 18:23 +0200, Lennart Poettering wrote:
On Fri, 26.07.13 15:03, Stephen Gallagher (sgallagh@redhat.com) wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/26/2013 02:52 PM, Lennart Poettering wrote:
On Fri, 26.07.13 14:32, Stephen Gallagher (sgallagh@redhat.com) wrote:
As Simo noted in the other thread, the availability of credentials outside the normal user session is an expectation of existing tools. The exposure here is significantly mitigated by the fact that Kerberos credentials are time-limited by the KDC.
So, let me get this right: you want a host-specific tmpfs location which is never automatically cleaned up, but is a private namespace of the user (though the system sometimes writes to it), correct?
That really sounds like a step backwards. Defining new runtime dirs without immediately thinking about life-cycles is something we really shouldn't do anymore.
XDG_RUNTIME_DIRS was introduced just because we want a clear life-cycle.
Lennart
PS: as a side note. what do you actually create in XDG_RUNTIME_DIR? A subdirectory? You are aware of the inherent risks of sharing a directory between system code and user code? It's extremely hard to properly get a subdir created in such a dir without opening a security hole.
Yes, we're creating a directory that will contain within it one or more file-based credential caches along with a special file that tells libkrb5 which one is the active default at a particular time. The actual cache directory is *never* created by system code. It's always generated by the libkrb5 in the user context. What *may* be created by the system is the enclosing directory. In the current situation, SSSD (as root) creates /run/user/UID and then libkrb5 creates /run/user/UID/krb5cc.
So, /run/user/$UID/krb5cc/ is a directroy you say is not created by system code, but by user code. Yet you say that you need this very early in the PAM chain, which however is in code that runs with full privileges as system code. This seems to contradict? Can you elaborate?
We create /run/user/$UID if missing, then spawn child process helper which setuids to $UID and runs kerberos code which will create /run/user/$UID/krb5cc as needed as $UID.
If you create /run/user/$UID/krb5cc/ from privileged code then it is very easy for unprivileged code to exploit that unless you are extra careful.
We try to be careful, feel free to review SSSD code and communicate any issue you may see in that area.
Simo.
On Fri, 2013-07-26 at 13:57 -0400, Stephen Gallagher wrote:
- We still need to consider use-cases where a cron job or other
long-running service needs to use credentials given to it by the user, though they are no longer signed in. With the current approach, we still need to be concerned that the /run/user/UID directory may just cease to exist if there are no more active sessions on the machine.
One kind of nontrivial approach, but at least worth thinking about, is changing cron to keep a zygote process around (with a session) for users with active cron jobs.
Futhermore if this zygote is terminated, then no cron jobs for the user will be run.
A nice advantage of this is is say you have an abusive user and do "killall -u baduser" - they can't "escape" this by scheduling cron jobs.
And cron jobs could theoretically be stored in ~username/.config/<machine-id> instead of /var/spool/cron.
On 26/07/13 14:58 -0400, Colin Walters wrote:
On Fri, 2013-07-26 at 13:57 -0400, Stephen Gallagher wrote: One kind of nontrivial approach, but at least worth thinking about, is changing cron to keep a zygote process around (with a session) for users with active cron jobs.
Futhermore if this zygote is terminated, then no cron jobs for the user will be run.
A nice advantage of this is is say you have an abusive user and do "killall -u baduser" - they can't "escape" this by scheduling cron jobs.
Good (side)note; just recently I've noticed there is no single step solution to prevent selected user from cron without removing this user or disabling cron altogether or perhaps applying some SELinux magic (plus there are possibly more related issues like bypassing cron.{allow,deny} through another user, but haven't checked it).
On Fri, Jul 26, 2013 at 07:40:39PM +0200, Lennart Poettering wrote:
On Fri, 26.07.13 11:01, Stephen Gallagher (sgallagh@redhat.com) wrote:
I'm CCing Lennart and Kay on the discussion about this. Creating this symlink in the oddjob would be somewhat error-prone. I'd rather that we ask logind to carry a patch just for F19 where the creation of XDG_RUNTIME_DIR would include this symlink.
Stephen, so Kay convinced me that it might be OK to handle su/sudo and su -l/sudo -i differently from the rest and accept that we have sessions that are started out of other sessions.
Thanks, good work Kay! ;-)
pam_systemd would need a new argument force-new=1 or so, which is passed for "su -l" and "sudo -i" but not otherwise. THis would be passed along CreateSession() to logind. When specified this would result in a
It would be nice to backport this feature to f19 too.
Karel
On Fri, 26.07.13 10:48, Simo Sorce (simo@redhat.com) wrote:
(Coming back to the original suggestion, now that the XDG_RUNTIME_DIR thing is ruled out.)
Recently a number of bugs [1-5] have come up regarding the new default Kerberos Ccache location that we changed according to [6].
We originally thought that reusing the same directory used by XDG_USER_DIR was a good idea as systemd/logind would pre-create it for us and we'd all be happy.
Unfortunately it turns out there a re a number of cases where the directory is not pre-created for us (sshd, sudo, su) and a number of cases where even if we created it out of systemd/logind supervision we'd risk it being yanked from under the process that is using it as by default systemd/logind uses a refcount system to know when to remove it.
I have an idea that can solve the problem relatively easily. Create a small dbus program (reuse oddjob ?) that will be called by libkrb5 to request creation of the credential cache. This would be a small Fedora19 patch to libkrb5, I do not think upstream would like it in this form (more on it later)[*].
The dbus program would simply get the unix cred structure of the calling application via dbus services and unconditionally create /var/kerberos/user/%uidnumber/krbcc[**] directory based exclusively on the uid number of the peer and a system configured template, it would also create a symlink in /run/user/%uidnumber/krbcc for backwards compatibility in Fedora 19 only, and we transition completely to the new dir in F20.
Why a new dir ? Because we do not want systemd/logind to yank the directory under us. There are a number of cases where it would be beneficial to keep it around (example a cron job that starts every 10 minutes and uses cached credentials valid for hours to do some task).
So, let me get this straight. You want to avoid any kind of automatic clean-up, and thus you want to introduce a new runtime directory backed by tmpfs?
The cleanup mechanisms of XDG_RUNTIME_DIR and /tmp have been introduced to solve real problems. We should not introduce a new scheme without any concept of automatic clean-up. /tmp already has awful clean-up semantics, XDG_RUNTIME_DIR is supposed to make things better, but by introducing a new user-writable dir which is entirely unrestricted you go back to start. I think this is a *really bad idea*.
So, one question, why again not just use the kernel keyring? If this is about the size of the objects then maybe you can convince the kernel keyring guys to make it backed by tmpfs, the same way as GEM/DRM or kdbus is backed by tmpfs these days. That makes the memory swappable and somewhat sanely attributed to the users creating them. The difference between using raw tmpfs and the keyring is that the keyring actually has time-based clean-up built in via expiry timeouts.
Anyway, please don't blindly introduce new places where users can acquire unbounded resources with no scheme of cleaning them up again. We already have too many places like that, like /dev/shm or SysV IPC which are entirely unpoliced. Instead of introducing more and more places with no-cleanup logics we should much rather work on fixing the existing ones.
Lennart
On Mon, 2013-07-29 at 22:50 +0200, Lennart Poettering wrote:
On Fri, 26.07.13 10:48, Simo Sorce (simo@redhat.com) wrote:
(Coming back to the original suggestion, now that the XDG_RUNTIME_DIR thing is ruled out.)
Recently a number of bugs [1-5] have come up regarding the new default Kerberos Ccache location that we changed according to [6].
We originally thought that reusing the same directory used by XDG_USER_DIR was a good idea as systemd/logind would pre-create it for us and we'd all be happy.
Unfortunately it turns out there a re a number of cases where the directory is not pre-created for us (sshd, sudo, su) and a number of cases where even if we created it out of systemd/logind supervision we'd risk it being yanked from under the process that is using it as by default systemd/logind uses a refcount system to know when to remove it.
I have an idea that can solve the problem relatively easily. Create a small dbus program (reuse oddjob ?) that will be called by libkrb5 to request creation of the credential cache. This would be a small Fedora19 patch to libkrb5, I do not think upstream would like it in this form (more on it later)[*].
The dbus program would simply get the unix cred structure of the calling application via dbus services and unconditionally create /var/kerberos/user/%uidnumber/krbcc[**] directory based exclusively on the uid number of the peer and a system configured template, it would also create a symlink in /run/user/%uidnumber/krbcc for backwards compatibility in Fedora 19 only, and we transition completely to the new dir in F20.
Why a new dir ? Because we do not want systemd/logind to yank the directory under us. There are a number of cases where it would be beneficial to keep it around (example a cron job that starts every 10 minutes and uses cached credentials valid for hours to do some task).
So, let me get this straight. You want to avoid any kind of automatic clean-up, and thus you want to introduce a new runtime directory backed by tmpfs?
The cleanup mechanisms of XDG_RUNTIME_DIR and /tmp have been introduced to solve real problems. We should not introduce a new scheme without any concept of automatic clean-up. /tmp already has awful clean-up semantics, XDG_RUNTIME_DIR is supposed to make things better, but by introducing a new user-writable dir which is entirely unrestricted you go back to start. I think this is a *really bad idea*.
So, one question, why again not just use the kernel keyring?
Size.
If this is about the size of the objects then maybe you can convince the kernel keyring guys to make it backed by tmpfs, the same way as GEM/DRM or kdbus is backed by tmpfs these days.
Sure, feel free to go that way, it is a *long* road, and it is simpler to have a daemon that checks for valid credentials and delete expired ones than deal with this in the kernel. The value of the keyring is in using non-swappable memory, if you allow it to swap then it makes no sense to use a custom, complex, kernel interface anymore, files are just simpler and easier to manage for admins and applications and can be protected easily by ACLs and Selinux.
That makes the memory swappable and somewhat sanely attributed to the users creating them. The difference between using raw tmpfs and the keyring is that the keyring actually has time-based clean-up built in via expiry timeouts.
Anyway, please don't blindly introduce new places where users can acquire unbounded resources with no scheme of cleaning them up again. We already have too many places like that, like /dev/shm or SysV IPC which are entirely unpoliced. Instead of introducing more and more places with no-cleanup logics we should much rather work on fixing the existing ones.
Long term we want to introduce and use a KCM that doesn't require users to directly use and manage cache files, it will use and manage it's own database. Short term we need to fix the pain introduced with more urgency than we are allowed to deal with less critical issues. Yes there is a possibility for users to abuse this space just like there has always been with /tmp, we are not making the situation really any worse than that.
If need be we can add a scanning cron job (or daemon or whatever) that will walk through the caches and eliminate those that have expired credentials in them, it wouldn't be too hard to add, just a matter of spending some time on the binary that check this data in a 'safe' way.
Simo.
On Mon, 29.07.13 17:25, Simo Sorce (simo@redhat.com) wrote:
So, one question, why again not just use the kernel keyring?
Size.
If this is about the size of the objects then maybe you can convince the kernel keyring guys to make it backed by tmpfs, the same way as GEM/DRM or kdbus is backed by tmpfs these days.
Sure, feel free to go that way, it is a *long* road, and it is simpler to have a daemon that checks for valid credentials and delete expired ones than deal with this in the kernel. The value of the keyring is in using non-swappable memory, if you allow it to swap then it makes no sense to use a custom, complex, kernel interface anymore, files are just simpler and easier to manage for admins and applications and can be protected easily by ACLs and Selinux.
Well, the point is that the kernel keyring is by purpose and by semantics pretty much exactly what you guys need. It has expiry logic, access control, namespacing, and it's main purpose is actually handling of authentication tokens. The only issue appears to be object size, but that's totally fixable.
I can understand that it is easier to intrdouce a new userspace daemon that works around a kernel limitations, but the right approach is still to just fix the kernel interface.
The kernel keyring folks work for Red Hat, have you pinged them?
That makes the memory swappable and somewhat sanely attributed to the users creating them. The difference between using raw tmpfs and the keyring is that the keyring actually has time-based clean-up built in via expiry timeouts.
Anyway, please don't blindly introduce new places where users can acquire unbounded resources with no scheme of cleaning them up again. We already have too many places like that, like /dev/shm or SysV IPC which are entirely unpoliced. Instead of introducing more and more places with no-cleanup logics we should much rather work on fixing the existing ones.
Long term we want to introduce and use a KCM that doesn't require users to directly use and manage cache files, it will use and manage it's own database.
What is "KCM"?
Short term we need to fix the pain introduced with more urgency than we are allowed to deal with less critical issues. Yes there is a possibility for users to abuse this space just like there has always been with /tmp, we are not making the situation really any worse than that.
So, why don't you revert to using /tmp then?
I mean, it appears that you don't like /tmp because of the broken namespace and because it might not be in tmpfs. And then you move it to some place which has equally bad problems, just different ones. This doesn't make things any better, does it? (Especially given that /tmp is tmpfs by default, so your major driver seems to be the exception?)
It appears to me the best would be to stay with /tmp for kerberos for as long as nothing appears that is clearly better and solves the issues. Then, work on the kernel keyring, and move to it in the longer run as soon as it can deal with large keys without issues?
If need be we can add a scanning cron job (or daemon or whatever) that will walk through the caches and eliminate those that have expired credentials in them, it wouldn't be too hard to add, just a matter of spending some time on the binary that check this data in a 'safe' way.
Well, tmpfiles does that on /tmp already.
Lennart
On Tue, 2013-07-30 at 00:50 +0200, Lennart Poettering wrote:
So, why don't you revert to using /tmp then?
The problem with /tmp is that if you want predictable filenames for the storage, you open yourself to a denial-of-service attack where another user can create a file with the same name.
It's really not that hard to create /run/kerberos and manage its contents by deleting them when they expire. And it doesn't tie us into anything permanent either; by the time the ccache moves for the *second* time, everyone is already removing the hard-coded assumptions from their code and just asking libkrb5 "where is it today?".
On Mon, 29.07.13 23:56, David Woodhouse (dwmw2@infradead.org) wrote:
On Tue, 2013-07-30 at 00:50 +0200, Lennart Poettering wrote:
So, why don't you revert to using /tmp then?
The problem with /tmp is that if you want predictable filenames for the storage, you open yourself to a denial-of-service attack where another user can create a file with the same name.
Well, but that's not unsurmountable, just pick a randomly named directory in /tmp and make sure to have a symlink:
ln -s /tmp/krb.XXXXXX "$HOME/.krb-`cat /etc/machine-id`"
to give it a stable, machine-local name. That's more or less what PulseAudio did in absence of XDG_RUNTIME_DIR in order not to be vulnerable to namespace attacks but still having a stable, machine-local place to put runtime objects.
Lennart
On Tue, 2013-07-30 at 02:08 +0200, Lennart Poettering wrote:
On Mon, 29.07.13 23:56, David Woodhouse (dwmw2@infradead.org) wrote:
On Tue, 2013-07-30 at 00:50 +0200, Lennart Poettering wrote:
So, why don't you revert to using /tmp then?
The problem with /tmp is that if you want predictable filenames for the storage, you open yourself to a denial-of-service attack where another user can create a file with the same name.
Well, but that's not unsurmountable, just pick a randomly named directory in /tmp and make sure to have a symlink:
ln -s /tmp/krb.XXXXXX "$HOME/.krb-`cat /etc/machine-id`"
What would create this directory ?
to give it a stable, machine-local name.
in what case /tmp contains non-'machine-local' files ?
Also I need one directory per-user and not per-machine.
And how is this different than /run/kerberos in the end ?
Simo.
On Mon, 29.07.13 21:11, Simo Sorce (simo@redhat.com) wrote:
On Tue, 2013-07-30 at 02:08 +0200, Lennart Poettering wrote:
On Mon, 29.07.13 23:56, David Woodhouse (dwmw2@infradead.org) wrote:
On Tue, 2013-07-30 at 00:50 +0200, Lennart Poettering wrote:
So, why don't you revert to using /tmp then?
The problem with /tmp is that if you want predictable filenames for the storage, you open yourself to a denial-of-service attack where another user can create a file with the same name.
Well, but that's not unsurmountable, just pick a randomly named directory in /tmp and make sure to have a symlink:
ln -s /tmp/krb.XXXXXX "$HOME/.krb-`cat /etc/machine-id`"
What would create this directory ?
The same component that creates the temporary directory?
In pseudo code:
char temp[] = "/tmp/krb.XXXXXX", link[PATH_MAX]; char *machine_id, *home;
mkdtemp(temp); machine_id = get_file_contents("/etc/machine_id"); *strchrnul(machined_id, '\n') = 0; home = getenv("HOME); snprintf(link, "%s/.krb-%s", home, machine_id); symlink(temp, link);
Of course, you should skip this if the symlink already exists and points to a valid directory...
to give it a stable, machine-local name.
in what case /tmp contains non-'machine-local' files ?
/tmp doesn't. But /home does. Hence you include the machine ID in the symlink name.
Also I need one directory per-user and not per-machine.
Well, you want it per-user *and* per-machine.
And how is this different than /run/kerberos in the end ?
That there's a sane cleanup scheme done via /tmp and not yet another place where we have unrestricted runtime objects of the user.
Lennart
On Tue, 2013-07-30 at 03:27 +0200, Lennart Poettering wrote:
On Mon, 29.07.13 21:11, Simo Sorce (simo@redhat.com) wrote:
On Tue, 2013-07-30 at 02:08 +0200, Lennart Poettering wrote:
On Mon, 29.07.13 23:56, David Woodhouse (dwmw2@infradead.org) wrote:
On Tue, 2013-07-30 at 00:50 +0200, Lennart Poettering wrote:
So, why don't you revert to using /tmp then?
The problem with /tmp is that if you want predictable filenames for the storage, you open yourself to a denial-of-service attack where another user can create a file with the same name.
Well, but that's not unsurmountable, just pick a randomly named directory in /tmp and make sure to have a symlink:
ln -s /tmp/krb.XXXXXX "$HOME/.krb-`cat /etc/machine-id`"
What would create this directory ?
The same component that creates the temporary directory?
In pseudo code:
char temp[] = "/tmp/krb.XXXXXX", link[PATH_MAX]; char *machine_id, *home;
mkdtemp(temp); machine_id = get_file_contents("/etc/machine_id"); *strchrnul(machined_id, '\n') = 0; home = getenv("HOME); snprintf(link, "%s/.krb-%s", home, machine_id); symlink(temp, link);
Of course, you should skip this if the symlink already exists and points to a valid directory...
This doesn't make any sense whatsoever, if we are back to unpredictable file names and trolling I may as well just use /tmp/krb5cc_XXXXXX only.
Ccaches do not belong in the home directory, period.
to give it a stable, machine-local name.
in what case /tmp contains non-'machine-local' files ?
/tmp doesn't. But /home does. Hence you include the machine ID in the symlink name.
Also I need one directory per-user and not per-machine.
Well, you want it per-user *and* per-machine.
And how is this different than /run/kerberos in the end ?
That there's a sane cleanup scheme done via /tmp and not yet another place where we have unrestricted runtime objects of the user.
Except the stuff that cleans /tmp has no idea how to look *into* the ccache to see if it is still valid or not, so it is a mechanism I do not care for.
Simo.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/30/2013 08:07 AM, Simo Sorce wrote:
On Tue, 2013-07-30 at 03:27 +0200, Lennart Poettering wrote:
Of course, you should skip this if the symlink already exists and points to a valid directory...
This doesn't make any sense whatsoever, if we are back to unpredictable file names and trolling I may as well just use /tmp/krb5cc_XXXXXX only.
Ccaches do not belong in the home directory, period.
To clarify this statement slightly: ccaches *cannot* be in the home directory because this causes a chicken-and-egg problem with Secure NFS home directories. We need the credentials to be accessible in order to mount the home directory at all.
On Tue, 30.07.13 08:07, Simo Sorce (simo@redhat.com) wrote:
The same component that creates the temporary directory?
In pseudo code:
char temp[] = "/tmp/krb.XXXXXX", link[PATH_MAX]; char *machine_id, *home;
mkdtemp(temp); machine_id = get_file_contents("/etc/machine_id"); *strchrnul(machined_id, '\n') = 0; home = getenv("HOME); snprintf(link, "%s/.krb-%s", home, machine_id); symlink(temp, link);
Of course, you should skip this if the symlink already exists and points to a valid directory...
This doesn't make any sense whatsoever, if we are back to unpredictable file names and trolling I may as well just use /tmp/krb5cc_XXXXXX only.
Ccaches do not belong in the home directory, period.
Read again what I wrote above. The ccche is not in the home directory following my proposal. Only a symlink to give it a stable name is. And the ccache is stored in /tmp, i.e. on tmpfs.
That there's a sane cleanup scheme done via /tmp and not yet another place where we have unrestricted runtime objects of the user.
Except the stuff that cleans /tmp has no idea how to look *into* the ccache to see if it is still valid or not, so it is a mechanism I do not care for.
Well, I guess with your attitude you actually know already what you want and do not care for the big picture, so I don't think we have to continue the discussion.
Lennart
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/30/2013 05:53 PM, Lennart Poettering wrote:
On Tue, 30.07.13 08:07, Simo Sorce (simo@redhat.com) wrote:
The same component that creates the temporary directory?
In pseudo code:
char temp[] = "/tmp/krb.XXXXXX", link[PATH_MAX]; char *machine_id, *home;
mkdtemp(temp); machine_id = get_file_contents("/etc/machine_id"); *strchrnul(machined_id, '\n') = 0; home = getenv("HOME); snprintf(link, "%s/.krb-%s", home, machine_id); symlink(temp, link);
Of course, you should skip this if the symlink already exists and points to a valid directory...
This doesn't make any sense whatsoever, if we are back to unpredictable file names and trolling I may as well just use /tmp/krb5cc_XXXXXX only.
Ccaches do not belong in the home directory, period.
Read again what I wrote above. The ccche is not in the home directory following my proposal. Only a symlink to give it a stable name is. And the ccache is stored in /tmp, i.e. on tmpfs.
That still doesn't help us, because we cannot create the symlink at the same time we create the directory, since it may not yet be mounted. I'm not sure how this improves the situation.
On Tue, 2013-07-30 at 08:07 -0400, Simo Sorce wrote:
On Tue, 2013-07-30 at 03:27 +0200, Lennart Poettering wrote:
On Mon, 29.07.13 21:11, Simo Sorce (simo@redhat.com) wrote:
On Tue, 2013-07-30 at 02:08 +0200, Lennart Poettering wrote:
On Mon, 29.07.13 23:56, David Woodhouse (dwmw2@infradead.org) wrote:
On Tue, 2013-07-30 at 00:50 +0200, Lennart Poettering wrote:
So, why don't you revert to using /tmp then?
The problem with /tmp is that if you want predictable filenames for the storage, you open yourself to a denial-of-service attack where another user can create a file with the same name.
Well, but that's not unsurmountable, just pick a randomly named directory in /tmp and make sure to have a symlink:
ln -s /tmp/krb.XXXXXX "$HOME/.krb-`cat /etc/machine-id`"
What would create this directory ?
The same component that creates the temporary directory?
In pseudo code:
char temp[] = "/tmp/krb.XXXXXX", link[PATH_MAX]; char *machine_id, *home;
mkdtemp(temp); machine_id = get_file_contents("/etc/machine_id"); *strchrnul(machined_id, '\n') = 0; home = getenv("HOME); snprintf(link, "%s/.krb-%s", home, machine_id); symlink(temp, link);
Of course, you should skip this if the symlink already exists and points to a valid directory...
This doesn't make any sense whatsoever, if we are back to unpredictable file names and trolling I may as well just use /tmp/krb5cc_XXXXXX only.
Ccaches do not belong in the home directory, period.
Couldn't agree more. There's no guarantee the home directory will be available when the credential is needed.
to give it a stable, machine-local name.
in what case /tmp contains non-'machine-local' files ?
/tmp doesn't. But /home does. Hence you include the machine ID in the symlink name.
Also I need one directory per-user and not per-machine.
Well, you want it per-user *and* per-machine.
And how is this different than /run/kerberos in the end ?
That there's a sane cleanup scheme done via /tmp and not yet another place where we have unrestricted runtime objects of the user.
Except the stuff that cleans /tmp has no idea how to look *into* the ccache to see if it is still valid or not, so it is a mechanism I do not care for.
Simo.
-- Simo Sorce * Red Hat, Inc * New York
On 07/30/2013 03:27 AM, Lennart Poettering wrote:
The same component that creates the temporary directory?
In pseudo code:
char temp[] = "/tmp/krb.XXXXXX", link[PATH_MAX]; char *machine_id, *home; mkdtemp(temp); machine_id = get_file_contents("/etc/machine_id"); *strchrnul(machined_id, '\n') = 0; home = getenv("HOME); snprintf(link, "%s/.krb-%s", home, machine_id); symlink(temp, link);Of course, you should skip this if the symlink already exists and points to a valid directory...
I'm not sure if this is a supported use case, but the above breaks if /home is mounted on multiple machines.
On Fri, 02.08.13 11:52, Florian Weimer (fweimer@redhat.com) wrote:
On 07/30/2013 03:27 AM, Lennart Poettering wrote:
The same component that creates the temporary directory?
In pseudo code:
char temp[] = "/tmp/krb.XXXXXX", link[PATH_MAX]; char *machine_id, *home;
mkdtemp(temp); machine_id = get_file_contents("/etc/machine_id"); *strchrnul(machined_id, '\n') = 0; home = getenv("HOME); snprintf(link, "%s/.krb-%s", home, machine_id); symlink(temp, link);
Of course, you should skip this if the symlink already exists and points to a valid directory...
I'm not sure if this is a supported use case, but the above breaks if /home is mounted on multiple machines.
No, it doesn't, since we explicitly include the machine ID in the name, which is supposed to be unique to the local system.
Lennart
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/29/2013 08:08 PM, Lennart Poettering wrote:
On Mon, 29.07.13 23:56, David Woodhouse (dwmw2@infradead.org) wrote:
On Tue, 2013-07-30 at 00:50 +0200, Lennart Poettering wrote:
So, why don't you revert to using /tmp then?
The problem with /tmp is that if you want predictable filenames for the storage, you open yourself to a denial-of-service attack where another user can create a file with the same name.
Well, but that's not unsurmountable, just pick a randomly named directory in /tmp and make sure to have a symlink:
ln -s /tmp/krb.XXXXXX "$HOME/.krb-`cat /etc/machine-id`"
Picking a random name brings us back to one of the original problems we needed to solve: random names make life *miserable* for daemons like GSSD that need to find the appropriate credentials for a user. Right now, it is forced to check every file (whose name corresponds to a certain pattern) in /tmp until it finds one that is still valid and it uses that to set up credentials. This is highly error-prone (for example, you might have two caches in /tmp, one that has eight hours of useful life left and one that has 30s, but if the 30s one is found first, that's the one that will be used). That's not even including the various exploits that might be present (though that's a well-investigated case).
Moving somewhere that we could safely use a well-known name was a major step forward.
to give it a stable, machine-local name. That's more or less what PulseAudio did in absence of XDG_RUNTIME_DIR in order not to be vulnerable to namespace attacks but still having a stable, machine-local place to put runtime objects.
Lennart
On Tue, 30.07.13 08:34, Stephen Gallagher (sgallagh@redhat.com) wrote:
The problem with /tmp is that if you want predictable filenames for the storage, you open yourself to a denial-of-service attack where another user can create a file with the same name.
Well, but that's not unsurmountable, just pick a randomly named directory in /tmp and make sure to have a symlink:
ln -s /tmp/krb.XXXXXX "$HOME/.krb-`cat /etc/machine-id`"
Picking a random name brings us back to one of the original problems we needed to solve: random names make life *miserable* for daemons like GSSD that need to find the appropriate credentials for a user.
Hence give the random name stability via a stable symlink. Please read what I actually wrote.
Lennart
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/29/2013 06:50 PM, Lennart Poettering wrote:
On Mon, 29.07.13 17:25, Simo Sorce (simo@redhat.com) wrote:
If need be we can add a scanning cron job (or daemon or whatever) that will walk through the caches and eliminate those that have expired credentials in them, it wouldn't be too hard to add, just a matter of spending some time on the binary that check this data in a 'safe' way.
Well, tmpfiles does that on /tmp already.
While I appreciate the intent of tmpfs on /tmp, it's pretty much a guarantee that anyone running a server who knows about it will shut it off because it causes more problems than it solves when using applications like MySQL, which stores a lot of data in /tmp, thus resulting in massive swapping and slowing the entire system down.
We need a solution that is consistently tmpfs here.
On Tue, 30.07.13 08:36, Stephen Gallagher (sgallagh@redhat.com) wrote:
On 07/29/2013 06:50 PM, Lennart Poettering wrote:
On Mon, 29.07.13 17:25, Simo Sorce (simo@redhat.com) wrote:
If need be we can add a scanning cron job (or daemon or whatever) that will walk through the caches and eliminate those that have expired credentials in them, it wouldn't be too hard to add, just a matter of spending some time on the binary that check this data in a 'safe' way.
Well, tmpfiles does that on /tmp already.
While I appreciate the intent of tmpfs on /tmp, it's pretty much a guarantee that anyone running a server who knows about it will shut it off because it causes more problems than it solves when using applications like MySQL, which stores a lot of data in /tmp, thus resulting in massive swapping and slowing the entire system down.
We need a solution that is consistently tmpfs here.
Well, if MySQL still has that limitation, then it should be fixed. Fix problems where they are, don't work around them.
Also, writing things via tmpfs into swap to disk, or directly to disk is not that much of a difference.
Lennart
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/30/2013 05:57 PM, Lennart Poettering wrote:
Fix problems where they are, don't work around them.
I think this is the crux of our disconnect here, Lennart. I appreciate that your goal is to have a perfect solution in all projects from the kernel to the cloud. I respect that and laud it.
However, perfect solutions are impractical here. We cannot reasonably approach a problem that our users are having *right now* by saying "we'll fix a half-dozen packages (some of them with hostile upstreams) over the course of the next 18 months". That's unacceptable to our users.
What we need to do is come up with a solution that works *for now*, ideally with technology that we already have, so that we can get our users back up and running in an acceptable manner.
I agree that we should be looking at better, more supportable long-term solutions, but right now this discussion is turning into worrying about how big a garden to plant in the backyard while the house is on fire. It is answering some very valid questions while at the same time completely missing the main points.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/29/2013 06:50 PM, Lennart Poettering wrote:
On Mon, 29.07.13 17:25, Simo Sorce (simo@redhat.com) wrote:
So, one question, why again not just use the kernel keyring?
Size.
If this is about the size of the objects then maybe you can convince the kernel keyring guys to make it backed by tmpfs, the same way as GEM/DRM or kdbus is backed by tmpfs these days.
Sure, feel free to go that way, it is a *long* road, and it is simpler to have a daemon that checks for valid credentials and delete expired ones than deal with this in the kernel. The value of the keyring is in using non-swappable memory, if you allow it to swap then it makes no sense to use a custom, complex, kernel interface anymore, files are just simpler and easier to manage for admins and applications and can be protected easily by ACLs and Selinux.
Well, the point is that the kernel keyring is by purpose and by semantics pretty much exactly what you guys need. It has expiry logic, access control, namespacing, and it's main purpose is actually handling of authentication tokens. The only issue appears to be object size, but that's totally fixable.
I can understand that it is easier to intrdouce a new userspace daemon that works around a kernel limitations, but the right approach is still to just fix the kernel interface.
The kernel keyring folks work for Red Hat, have you pinged them?
Circling back around on this, we contacted the kernel keyring developers (specifically David Howells) and we are now working this direction. We initially expected a great deal more resistance than we actually got, which was why we hesitated to take this approach (that and past history with size issues).
So the current approach we are investigating looks something like the following (based on discussions between Simo, Nalin, David and myself)
1) We will add a new key type "big_key" that allows us to create keys up to 1MiB in size, backed by internal kernel tmpfs, allowing the contents to be swapped out to disk (unlike most other keyrings, which remain in unswappable kernel memory).
2) We will create a new public interface, keyctl_get_krbcache(uid_t, key_serial_t id). This API will allow the user (and certain privileged root processes such as rpc.gssd and gss-proxy) to access the keys for a particular UID. This keyring shall not be tied to a session (so it can outlive a user on the system if they need to perform actions while not logged in, such as access to secure NFS). The kernel keyring should be created automatically on the first request if it does not yet exist. It must also be possible to manipulate the lifetime timer with functions like keyctl_set_timeout() (to align the keyring life with the credential validity) and must also be possible to be destroyed immediately using the usual keyctl APIs (such as in the kdestroy case).
libkrb5 will now accept a new value for the credential cache string (for example: when used in KRB5CCNAME). It will take the form: KEYRING:krbcache:$UID to represent a credential cache collection and KEYRING:krbcache:$UID:tktXXXXX to represent a specific key within a cache collection.
This new interface will need to support the 'kswitch' kerberos feature for selecting the REALM against which to operate.
This plan does not come without risks. It is possible that the kernel and MIT upstream might not accept this new keyring. We believe at the moment (from conversations with them) that this should be considered a low risk.
We are currently planning to execute on this plan immediately and quickly. Once this is complete and readied, action *will* be required by some of our downstream dependencies, including (but not limited to): * sssd * openssh * rpc.gssd * gnome-online-accounts
On Thu, 01.08.13 14:47, Stephen Gallagher (sgallagh@redhat.com) wrote:
I can understand that it is easier to intrdouce a new userspace daemon that works around a kernel limitations, but the right approach is still to just fix the kernel interface.
The kernel keyring folks work for Red Hat, have you pinged them?
Circling back around on this, we contacted the kernel keyring developers (specifically David Howells) and we are now working this direction. We initially expected a great deal more resistance than we actually got, which was why we hesitated to take this approach (that and past history with size issues).
So the current approach we are investigating looks something like the following (based on discussions between Simo, Nalin, David and myself)
- We will add a new key type "big_key" that allows us to create keys
up to 1MiB in size, backed by internal kernel tmpfs, allowing the contents to be swapped out to disk (unlike most other keyrings, which remain in unswappable kernel memory).
Thank you! This sounds like the optimal solution for everybody with the best semantics!
Lennart
devel@lists.stg.fedoraproject.org
-
Colin Walters -
David Woodhouse -
Florian Weimer -
Ian Kent -
Jan Pokorný -
Karel Zak -
Lennart Poettering -
Miloslav Trmač -
Simo Sorce -
Stephen Gallagher -
Tomas Mraz