I've been working to rewrite and extend the script that we've been using to control playbook execution for folks who are not in sysadmin-main.
https://bitbucket.org/tflink/rbac-ansible
I've been testing the script but before we actually start using it on lockbox01, I'd appreciate a review of the code to make sure I didn't miss any security holes.
Injection attacks shouldn't be an issue due to usage of os.execv - all injection attempts are grouped as a single argument and will not be broken up.
Tim
Le mercredi 04 juin 2014 à 19:45 -0600, Tim Flink a écrit :
I've been working to rewrite and extend the script that we've been using to control playbook execution for folks who are not in sysadmin-main.
https://bitbucket.org/tflink/rbac-ansible
I've been testing the script but before we actually start using it on lockbox01, I'd appreciate a review of the code to make sure I didn't miss any security holes.
Injection attacks shouldn't be an issue due to usage of os.execv - all injection attempts are grouped as a single argument and will not be broken up.
So, I just have one question. how does the option -P of the script is supposed to behave ?
Can i assume that I would be able to say "use this playbook, but instead of using the port 22, use port 1234" without changing the playbook ?
In this case, I think this would mean that if I can create a ssh tunnel on the remote server ( listening to port 1234 to a server I control, with ssh -L 1234:servericontrol:22 ), then I can make the playbook played on a server I control, which in turn mean that I would potentially get access to files with password that I may not have access too.
Example, the playbook that deploy mediawiki would deploy mediawiki on my server, then i can go as root look at the mysql credentials deployed in the configuration file. Or I can look at the https certificates that were deployed, etc, etc.
I do not know if such attack schema would matter for Fedora infra, but if the user running ansible ( after sudo, is sudoed a word ? ) has access to passwords that the initial user don't, and if there is no firewall internally ( ie, the tunnel trick would work, no firewall between lockbock and the server ), and if the attack/initial user can ssh to a server without much access, then, this would work.
( if not, I may just have won the Oscar for the most convoluted attack of the week )
mmm, for your attack strategy to work, basically the "attacker" need to have enough permissions in the first place to be able to execute the playbook such that the playbooks have access to the mysql secret? And if he already has that kinda permission, then there is no need to do a setup first and then read it coz the attacker can read it upfront without doing the setup.
I think this is controlled by the..
def can_run(acl, groups, playbook_file):
?
On Sat, Jun 7, 2014 at 8:56 PM, Michael Scherer misc@zarb.org wrote:
Le mercredi 04 juin 2014 à 19:45 -0600, Tim Flink a écrit :
I've been working to rewrite and extend the script that we've been using to control playbook execution for folks who are not in sysadmin-main.
https://bitbucket.org/tflink/rbac-ansible
I've been testing the script but before we actually start using it on lockbox01, I'd appreciate a review of the code to make sure I didn't miss any security holes.
Injection attacks shouldn't be an issue due to usage of os.execv - all injection attempts are grouped as a single argument and will not be broken up.
So, I just have one question. how does the option -P of the script is supposed to behave ?
Can i assume that I would be able to say "use this playbook, but instead of using the port 22, use port 1234" without changing the playbook ?
In this case, I think this would mean that if I can create a ssh tunnel on the remote server ( listening to port 1234 to a server I control, with ssh -L 1234:servericontrol:22 ), then I can make the playbook played on a server I control, which in turn mean that I would potentially get access to files with password that I may not have access too.
Example, the playbook that deploy mediawiki would deploy mediawiki on my server, then i can go as root look at the mysql credentials deployed in the configuration file. Or I can look at the https certificates that were deployed, etc, etc.
I do not know if such attack schema would matter for Fedora infra, but if the user running ansible ( after sudo, is sudoed a word ? ) has access to passwords that the initial user don't, and if there is no firewall internally ( ie, the tunnel trick would work, no firewall between lockbock and the server ), and if the attack/initial user can ssh to a server without much access, then, this would work.
( if not, I may just have won the Oscar for the most convoluted attack of the week ) -- Michael Scherer
infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Le samedi 07 juin 2014 à 21:28 +0530, Anshu Prateek a écrit :
mmm, for your attack strategy to work, basically the "attacker" need to have enough permissions in the first place to be able to execute the playbook such that the playbooks have access to the mysql secret? And if he already has that kinda permission, then there is no need to do a setup first and then read it coz the attacker can read it upfront without doing the setup.
Then why do we use sudo and a filtering script if a attacker can inject any playbook ?
My understanding was that people did have to commit first before being able to run something ( in order to provide auditing ), and that the sudo user do have access to stuff that the user/attacker don't ( like ssh keys, for example ). My understanding was also that there is a private repo is not readable by everybody, with various password, but that the user running ansible ( ie, the one accessible by sudo ) can read.
And that sudo is used to make sure the initial user can only run ansible, nothing ore.
If these assumptions are false, yeah, the attacker is more complex than needed. But as the idea is to permit to people who are not in sysadmin-main to run playbooks, I think my assumptions are correct.
On Sat, Jun 07, 2014 at 04:26:45PM +0100, Michael Scherer wrote:
Can i assume that I would be able to say "use this playbook, but instead of using the port 22, use port 1234" without changing the playbook ?
In this case, I think this would mean that if I can create a ssh tunnel on the remote server ( listening to port 1234 to a server I control, with ssh -L 1234:servericontrol:22 ), then I can make the playbook played on a server I control, which in turn mean that I would potentially get access to files with password that I may not have access too.
As long as SSH host keys are properly verified, port forwarding should not matter, since the machine is identified by their SSH host key and not their IP address/port. The host key checking was enabled in Fedora Infrastructure a while ago. I hope it still is. If the attacker was administrative access a host, then it could also be changed to forward connections to port 22 to another host. So even without being able to specify the port, this might be exploited.
Regards Till
On Sat, 07 Jun 2014 17:48:16 +0100 Michael Scherer misc@zarb.org wrote:
Le samedi 07 juin 2014 à 21:28 +0530, Anshu Prateek a écrit :
mmm, for your attack strategy to work, basically the "attacker" need to have enough permissions in the first place to be able to execute the playbook such that the playbooks have access to the mysql secret? And if he already has that kinda permission, then there is no need to do a setup first and then read it coz the attacker can read it upfront without doing the setup.
Then why do we use sudo and a filtering script if a attacker can inject any playbook ?
They cannot. It requires them to be in the checked out location on lockbox that uses could only update by having root on lockbox or commiting to git.
My understanding was that people did have to commit first before being able to run something ( in order to provide auditing ), and that the sudo user do have access to stuff that the user/attacker don't ( like ssh keys, for example ). My understanding was also that there is a private repo is not readable by everybody, with various password, but that the user running ansible ( ie, the one accessible by sudo ) can read.
Yep. sudo is used to see if the user is allowed to run rbac-playbook at all, then it checks further before running ansible-playbook.
And that sudo is used to make sure the initial user can only run ansible, nothing ore.
well, rbac-playbook then ansible-playbook, but yeah.
If these assumptions are false, yeah, the attacker is more complex than needed. But as the idea is to permit to people who are not in sysadmin-main to run playbooks, I think my assumptions are correct.
Right.
kevin
On Sat, 7 Jun 2014 19:31:33 +0200 Till Maas opensource@till.name wrote:
As long as SSH host keys are properly verified, port forwarding should not matter, since the machine is identified by their SSH host key and not their IP address/port. The host key checking was enabled in Fedora Infrastructure a while ago. I hope it still is.
...snip...
It is.
kevin
Le samedi 07 juin 2014 à 19:31 +0200, Till Maas a écrit :
On Sat, Jun 07, 2014 at 04:26:45PM +0100, Michael Scherer wrote:
Can i assume that I would be able to say "use this playbook, but instead of using the port 22, use port 1234" without changing the playbook ?
In this case, I think this would mean that if I can create a ssh tunnel on the remote server ( listening to port 1234 to a server I control, with ssh -L 1234:servericontrol:22 ), then I can make the playbook played on a server I control, which in turn mean that I would potentially get access to files with password that I may not have access too.
As long as SSH host keys are properly verified, port forwarding should not matter, since the machine is identified by their SSH host key and not their IP address/port. The host key checking was enabled in Fedora Infrastructure a while ago.
I do not see that in /etc/ssh/ssh_config on lockbox ( could be in ~/.ssh/config however ), nor anything in /etc/ansible/ansible.cfg ( could again be a local config somewhere else ). I didn't find anything making see a different ~/.ssh/config, nor ~/.ansible/* , so I think the default is used, which is 'ask'.
And after a quick crude test, if you have ssh listening on 2 ports, ssh will treat each as a different entry in known_hosts, and so ask again. ( or at least on my laptop, I didn't dig more given the hour, will try to search a bit more ).
So while I am not affirmative at 100% ( again, could be different in the precise case of ansible in Fedora infra, could be one of the 360 lines of my own ssh config, could be me being tired ), I would not exclude a possible issue with what I do see.
I hope it still is. If the attacker was administrative access a host, then it could also be changed to forward connections to port 22 to another host. So even without being able to specify the port, this might be exploited.
On Sun, Jun 08, 2014 at 02:42:41AM +0100, Michael Scherer wrote:
And after a quick crude test, if you have ssh listening on 2 ports, ssh will treat each as a different entry in known_hosts, and so ask again. ( or at least on my laptop, I didn't dig more given the hour, will try to search a bit more ).
Yes, I thought about it some more as well. This might indeed be a problem. I see two kind of attacks. The one you describe is only interesting for someone who is able to run ansible for a host but does not have root access to the host. If it is possible to specify a non-privileged port (e.g. 1234), the attacker can run its own SSH server there to get access to all information sent by ansible but only for the host the attacker has already access to. The required privileges are:
- Being able to login to a host - Being able to run playbooks for a host - Being able to specify the port for ansible to connect to
The attacker does not gain any advantage if they have already root access to the host.
Getting information from a different host requires to modify the playbook/ansible repo to send the wrong credentials from the private repo to the system the attacker has access to. But then it could be stored in a way that it is accessible for the attacker without root access, so being able to change the port is not necessary.
Another possible attack is to run the playbook on a different host. For this an attacker could forward the ssh port from target01 to attacker01:1234 and run the playbook for attacker01 but port 1234. Then it will be executed on target01, if ansible uses the right SSH private key automatically and the same sudo password or no sudo password.
If the attacker has root access attacker01, they could also forward attacker01:22 to target01:22, so being able to specify a port to ansible is not required. But this requires the ssh hostkey not being checked.
Regards Till
On Sun, 08 Jun 2014 02:42:41 +0100 Michael Scherer misc@zarb.org wrote:
I do not see that in /etc/ssh/ssh_config on lockbox ( could be in ~/.ssh/config however ), nor anything in /etc/ansible/ansible.cfg ( could again be a local config somewhere else ). I didn't find anything making see a different ~/.ssh/config, nor ~/.ansible/* , so I think the default is used, which is 'ask'.
Yeah, although 'ask' means: If you don't know the host at all, ask. If the host key doesn't match, reject.
And after a quick crude test, if you have ssh listening on 2 ports, ssh will treat each as a different entry in known_hosts, and so ask again. ( or at least on my laptop, I didn't dig more given the hour, will try to search a bit more ).
So while I am not affirmative at 100% ( again, could be different in the precise case of ansible in Fedora infra, could be one of the 360 lines of my own ssh config, could be me being tired ), I would not exclude a possible issue with what I do see.
Sure. Keep in mind that while we like rbac-playbook to be nice and secure, it's use is for people that are already trusted to have access to lockbox, are in a group that has sudo for rbac-playbook, have entered their password and 2fa token. It's simply a way to restrict them to the machines that are in their group(s)...
kevin
On Sun, 8 Jun 2014 11:21:45 +0200 Till Maas opensource@till.name wrote:
Yes, I thought about it some more as well. This might indeed be a problem. I see two kind of attacks. The one you describe is only interesting for someone who is able to run ansible for a host but does not have root access to the host. If it is possible to specify a non-privileged port (e.g. 1234), the attacker can run its own SSH server there to get access to all information sent by ansible but only for the host the attacker has already access to. The required privileges are:
- Being able to login to a host
- Being able to run playbooks for a host
- Being able to specify the port for ansible to connect to
The attacker does not gain any advantage if they have already root access to the host.
And I think in all cases currently they do. At least they should... ie, docs can run the docs-backend playbook, and also have sudo access on that machine.
kevin
On Sat, 07 Jun 2014 16:26:45 +0100 Michael Scherer misc@zarb.org wrote:
Le mercredi 04 juin 2014 à 19:45 -0600, Tim Flink a écrit :
I've been working to rewrite and extend the script that we've been using to control playbook execution for folks who are not in sysadmin-main.
https://bitbucket.org/tflink/rbac-ansible
I've been testing the script but before we actually start using it on lockbox01, I'd appreciate a review of the code to make sure I didn't miss any security holes.
Injection attacks shouldn't be an issue due to usage of os.execv - all injection attempts are grouped as a single argument and will not be broken up.
So, I just have one question. how does the option -P of the script is supposed to behave ?
Can i assume that I would be able to say "use this playbook, but instead of using the port 22, use port 1234" without changing the playbook ?
I think that most of your concerns have been addressed or are being discussed in other parts of this thread but I wanted to speak towards the reason that -P is there at all.
You are correct in reading that it has ansible-playbook use an ssh port other than 22. That is set using -e 'ansible_ssh_port=<some port>' and giving direct access to the -e parameter would be problematic at best, so I added the -P parameter which is restricted to just that option even though it's rendered as -e
The QA devel folks use phabricator and phabricator supports git repo hosting (through http(s) and ssh). In order to support git over ssh while keeping user information in phabricator (username, ssh key for git, repo permissions etc.), it uses a short-circuited ssh daemon that uses phabricator for auth instead of system accounts (restricted to git commands, though). Git repos on alternate ports is a bit of a pain, so to support git+ssh on port 22 I change the real ssh daemon (that can do more than git) to an alternate port.
Tim
On Mon, 9 Jun 2014 08:44:38 -0600 Tim Flink tflink@redhat.com wrote:
I think that most of your concerns have been addressed or are being discussed in other parts of this thread but I wanted to speak towards the reason that -P is there at all.
You are correct in reading that it has ansible-playbook use an ssh port other than 22. That is set using -e 'ansible_ssh_port=<some port>' and giving direct access to the -e parameter would be port>problematic at best, so I added the -P parameter which is restricted to just that option even though it's rendered as -e
The QA devel folks use phabricator and phabricator supports git repo hosting (through http(s) and ssh). In order to support git over ssh while keeping user information in phabricator (username, ssh key for git, repo permissions etc.), it uses a short-circuited ssh daemon that uses phabricator for auth instead of system accounts (restricted to git commands, though). Git repos on alternate ports is a bit of a pain, so to support git+ssh on port 22 I change the real ssh daemon (that can do more than git) to an alternate port.
If those hosts always have ssh on the same different port, we could just add that to vars?
http://docs.ansible.com/faq.html#how-do-i-handle-different-machines-needing-...
kevin
Le lundi 09 juin 2014 à 08:44 -0600, Tim Flink a écrit :
The QA devel folks use phabricator and phabricator supports git repo hosting (through http(s) and ssh). In order to support git over ssh while keeping user information in phabricator (username, ssh key for git, repo permissions etc.), it uses a short-circuited ssh daemon that uses phabricator for auth instead of system accounts (restricted to git commands, though). Git repos on alternate ports is a bit of a pain, so to support git+ssh on port 22 I change the real ssh daemon (that can do more than git) to an alternate port.
What about having the real sshd listening on one ip ( if possible, a rfc1918 one in the VPN ) and git from phabricator on a second ?
On Mon, 9 Jun 2014 08:49:48 -0600 Kevin Fenzi kevin@scrye.com wrote:
On Mon, 9 Jun 2014 08:44:38 -0600 Tim Flink tflink@redhat.com wrote:
I think that most of your concerns have been addressed or are being discussed in other parts of this thread but I wanted to speak towards the reason that -P is there at all.
You are correct in reading that it has ansible-playbook use an ssh port other than 22. That is set using -e 'ansible_ssh_port=<some port>' and giving direct access to the -e parameter would be port>problematic at best, so I added the -P parameter which is restricted to just that option even though it's rendered as -e
The QA devel folks use phabricator and phabricator supports git repo hosting (through http(s) and ssh). In order to support git over ssh while keeping user information in phabricator (username, ssh key for git, repo permissions etc.), it uses a short-circuited ssh daemon that uses phabricator for auth instead of system accounts (restricted to git commands, though). Git repos on alternate ports is a bit of a pain, so to support git+ssh on port 22 I change the real ssh daemon (that can do more than git) to an alternate port.
If those hosts always have ssh on the same different port, we could just add that to vars?
http://docs.ansible.com/faq.html#how-do-i-handle-different-machines-needing-...
I've generally been using port 222 for real ssh on those hosts. We could set the port in the inventory file. While that would work for many cases, I've always used the -e directly for 2 reasons:
1) My understanding is that ansible convention discourages putting stuff like that in the inventory files
2) Hosts are listening for ssh on port 22 when initially deployed. Initial deployments would require changing the inventory information to use port 22 for initial deployment and then changing it back to the alternate port after running the playbook/role which sets up the alternate port for ssh.
If that's the way that we want to go, we'll have some extra commits to the ansible repo but it'll work.
Tim
On Mon, 9 Jun 2014 10:13:00 -0600 Tim Flink tflink@redhat.com wrote:
I've generally been using port 222 for real ssh on those hosts. We could set the port in the inventory file. While that would work for many cases, I've always used the -e directly for 2 reasons:
- My understanding is that ansible convention discourages putting stuff like that in the inventory files
Well, it's a bit ugly, but should work I would think.
- Hosts are listening for ssh on port 22 when initially deployed. Initial deployments would require changing the inventory
information to use port 22 for initial deployment and then changing it back to the alternate port after running the playbook/role which sets up the alternate port for ssh.
True... a bit messy.
If that's the way that we want to go, we'll have some extra commits to the ansible repo but it'll work.
Yeah, I think a bit of messyness in the repo is better than having to bother with -P in rbac-playbook... just to make it simplier.
kevin
On Wed, 11 Jun 2014 16:27:15 -0600 Kevin Fenzi kevin@scrye.com wrote:
<snip>
If that's the way that we want to go, we'll have some extra commits to the ansible repo but it'll work.
Yeah, I think a bit of messyness in the repo is better than having to bother with -P in rbac-playbook... just to make it simplier.
-P has been removed in the code and tests.
Tim
On Mon, 09 Jun 2014 17:37:06 +0200 Michael Scherer misc@zarb.org wrote:
Le lundi 09 juin 2014 à 08:44 -0600, Tim Flink a écrit :
The QA devel folks use phabricator and phabricator supports git repo hosting (through http(s) and ssh). In order to support git over ssh while keeping user information in phabricator (username, ssh key for git, repo permissions etc.), it uses a short-circuited ssh daemon that uses phabricator for auth instead of system accounts (restricted to git commands, though). Git repos on alternate ports is a bit of a pain, so to support git+ssh on port 22 I change the real ssh daemon (that can do more than git) to an alternate port.
What about having the real sshd listening on one ip ( if possible, a rfc1918 one in the VPN ) and git from phabricator on a second ?
I can't think of any reason why that wouldn't work but I don't see what's wrong with just using an alternate port instead of adding a second IP.
I don't have a strong opinion on the exact setup as long as the external port 22 is handled by phabricator and the machine remains manageable through ansible without too many odd workarounds.
Tim
On Wed, 4 Jun 2014 19:45:23 -0600 Tim Flink tflink@redhat.com wrote:
I've been working to rewrite and extend the script that we've been using to control playbook execution for folks who are not in sysadmin-main.
With the removal of the '-P' option, I think that all the concerns raised have been addressed.
Any objections to putting the script into place?
If not, where should it live? I put it up on bitbucket because that was easy for me but I realize that none of the infra stuff lives there.
Tim
On Wed, 25 Jun 2014 12:33:48 -0600 Tim Flink tflink@redhat.com wrote:
On Wed, 4 Jun 2014 19:45:23 -0600 Tim Flink tflink@redhat.com wrote:
I've been working to rewrite and extend the script that we've been using to control playbook execution for folks who are not in sysadmin-main.
With the removal of the '-P' option, I think that all the concerns raised have been addressed.
Any objections to putting the script into place?
Nope. We can do so. ;)
If not, where should it live? I put it up on bitbucket because that was easy for me but I realize that none of the infra stuff lives there.
So, two options I see:
a) we just put it in infra ansible. Although, twist there is that lockbox01 is still under puppet, so we would need to put at least some config under puppet.
b) keep it at bitbucket, package it as an epel/fedora package and we just install that and use it from the package.
I'm not sure it's worth the trouble to package but on the other hand, there may well be lots of ansible users who might like something like that?
kevin
infrastructure@lists.fedoraproject.org
-
Anshu Prateek -
Kevin Fenzi -
Michael Scherer -
Till Maas -
Tim Flink