Hi folks,
Over the last couple days I've been sorta-kinda helping Adam move OpenQA to AMQP and he has, through no fault of his own, had a rough time of it.
The problems boil down to permissions issues and a lack of visibility into the broker. I'd like to propose two changes so that folks can help themselves rather than blindly guessing and hoping I am around (there is a bus around every corner and lottery tickets in every shop).
Firstly, I propose we allow authenticated users to create objects in the /pubsub (private) vhost using the AMQP client. These users already can create objects, but they have to do it through Ansible roles rather than through the AMQP object. Since the AMQP client configuration is checked into Ansible, though, it's rather redundant to make them declare the queue separately. When they don't declare it properly, they can't tell and so it's actually quite a pain point. This is an easy change, we just change all the accounts to have configure permissions
Secondly, I propose we offer a read-only monitoring account to the web interface RabbitMQ provides. One great thing about switching from ZeroMQ to AMQP is that the broker gives us tons of tools to make monitoring and debugging easier. Allowing users to easily see the objects in the broker (queues, bindings, exchanges, connections) means they can solve their own deployment problems rather than pinging me.
I think if we make an account with no AMQP permissions and the "monitoring"[0] permission, folks should be able to see everything and not be able to do anything destructive. Then it's just a question of how to share the credentials (or just make the account name guest with the username guest?) and how to expose the management UI port (just require people to sshuttle?).
[0] https://www.rabbitmq.com/management.html#permissions
Thoughts?
- Jeremy
On 6/14/19 8:38 AM, Jeremy Cline wrote:
Hi folks,
Over the last couple days I've been sorta-kinda helping Adam move OpenQA to AMQP and he has, through no fault of his own, had a rough time of it.
The problems boil down to permissions issues and a lack of visibility into the broker. I'd like to propose two changes so that folks can help themselves rather than blindly guessing and hoping I am around (there is a bus around every corner and lottery tickets in every shop).
Firstly, I propose we allow authenticated users to create objects in the /pubsub (private) vhost using the AMQP client. These users already can create objects, but they have to do it through Ansible roles rather than through the AMQP object. Since the AMQP client configuration is checked into Ansible, though, it's rather redundant to make them declare the queue separately. When they don't declare it properly, they can't tell and so it's actually quite a pain point. This is an easy change, we just change all the accounts to have configure permissions
My fear here is that someone will manually create something and we have to redeploy for some reason. They will be broken untll they manually remember to do what they did again. :(
Or does the ansible module override whats already there?
Secondly, I propose we offer a read-only monitoring account to the web interface RabbitMQ provides. One great thing about switching from ZeroMQ to AMQP is that the broker gives us tons of tools to make monitoring and debugging easier. Allowing users to easily see the objects in the broker (queues, bindings, exchanges, connections) means they can solve their own deployment problems rather than pinging me.
I think if we make an account with no AMQP permissions and the "monitoring"[0] permission, folks should be able to see everything and not be able to do anything destructive. Then it's just a question of how to share the credentials (or just make the account name guest with the username guest?) and how to expose the management UI port (just require people to sshuttle?).
Could we add a suoders that would let people run these commands as the monitoring user on one of the machines? But then I guess we would have to give shell access, but I like that much better than guest/guest. Thats sure to be abused.
Or if we can get a page with this info available at least internally, people could just grab that? I don't think there's anything there we want to keep secret...
Would need to run any of these changes by our security officer too.
kevin
My fear here is that someone will manually create something and we have to redeploy for some reason. They will be broken untll they manually remember to do what they did again. :(
It's not manual at all actually, the queues that should be declared are in the app's configuration file, which is in ansible. The change that Jeremy proposes is to let the apps (the AMQP clients) create the queues instead of the ansible role. In case of redeploy, on startup the app will recreate the queues and it should work.
My concern, though, is that the broker will end up with queues resulting from previous mistakes or typos and that we'll have to remove once in a while to avoid polluting the UI. Also, there's one thing I'm not sure I'm getting right, Jeremy. Do you plan on giving users access to configure on any queues? ( ".*" ) Or just on a queue matching their username? In the former there's a security risk to allow a user to delete other user's queues, and in the latter I don't understand how it relieves pain from the end user, since they have to get their queue name right anyway. But I wasn't in your discussions with Adam so I'm not sure what we're trying to simplify here. Is it about removing the necessity to call the rabbit/queue Ansible role?
Could we add a suoders that would let people run these commands as the monitoring user on one of the machines? But then I guess we would have to give shell access, but I like that much better than guest/guest. Thats sure to be abused.
Would it be sufficient to only allow monitoring access from within our infrastructure network?
Would need to run any of these changes by our security officer too.
Absolutely.
A.
On 6/18/19 8:35 AM, Aurelien Bompard wrote:
My fear here is that someone will manually create something and we have to redeploy for some reason. They will be broken untll they manually remember to do what they did again. :(
It's not manual at all actually, the queues that should be declared are in the app's configuration file, which is in ansible. The change that Jeremy proposes is to let the apps (the AMQP clients) create the queues instead of the ansible role. In case of redeploy, on startup the app will recreate the queues and it should work.
Yes, this is correct. Also, I don't want to optimize for people who are not configuring their application in Ansible. Those people should not do that and when they have problems and outages, as harsh as it sounds, I don't care. They should do it right.
My concern, though, is that the broker will end up with queues resulting from previous mistakes or typos and that we'll have to remove once in a while to avoid polluting the UI. Also, there's one thing I'm not sure I'm getting right, Jeremy. Do you plan on giving users access to configure on any queues? ( ".*" ) Or just on a queue matching their username? In the former there's a security risk to allow a user to delete other user's queues, and in the latter I don't understand how it relieves pain from the end user, since they have to get their queue name right anyway. But I wasn't in your discussions with Adam so I'm not sure what we're trying to simplify here. Is it about removing the necessity to call the rabbit/queue Ansible role?
Yeah, so I think we can keep the namespacing just for sanity's sake, but the problems we hit were things like not knowing you had to add some extra Ansible steps to create your queue in the first place (and then asking the obvious question of why you have to repeat yourself in the config in Ansible and then separate roles), accidentally delegating the creation of the queue (but not the user) to the production broker instead of the staging broker, forgetting to set passive_declares = true, and so on.
Each of these things required me to stop what I was doing, look at someone's Ansible role, grok the situation, investigate the broker, etc. It's wasting a ton of my time doing system admin work I don't need to do, and it's not what I'm employed to do so I can only justify so much of it.
Could we add a suoders that would let people run these commands as the monitoring user on one of the machines? But then I guess we would have to give shell access, but I like that much better than guest/guest. Thats sure to be abused.
Would it be sufficient to only allow monitoring access from within our infrastructure network?
Would need to run any of these changes by our security officer too.
Absolutely.
Indeed. It doesn't have to be publicly accessible (and, in fact, I'd rather it wasn't). As it stands now I often proxy with ssh to get to it and that's easy enough.
I don't have strong opinions on the implementation, I just don't want people to have to struggle through what can be a very simple process with powerful debugging and monitoring tools.
- Jeremy
On Tue, 2019-06-18 at 14:06 +0000, Jeremy Cline wrote:
the problems we hit were things like not knowing you had to add some extra Ansible steps to create your queue in the first place (and then asking the obvious question of why you have to repeat yourself in the config in Ansible and then separate roles), accidentally delegating the creation of the queue (but not the user) to the production broker instead of the staging broker, forgetting to set passive_declares = true, and so on.
Every single one of these issues happened to me too, and I also had to interrupt jcline to get help. I don't think the permissions model we are using is actually preventing a real problem, but it certainly is causing one.
infrastructure@lists.fedoraproject.org
-
Aurelien Bompard -
Jeremy Cline -
Kevin Fenzi -
Randy Barlow