Hi guys,
I've got MOM up and running as a vdsm thread but I've hit two small problems. I'll bounce them around on the list to see if people have some ideas.
1.) Libvirt SASL authentication
I was able to easily modify MOM to connect to libvirt by hardcoding the vdsm credentials. Obviously this is not an acceptable long-term solution. What is the best way to share the vdsm libvirt password with MOM in a way that does not compromise security? Whatever method we choose should not involve vdsm-specific changes to MOM. For starters I think I will just place the username and password in the mom.conf file. We could make this file readable only by the vdsm user. Thoughts?
2.) Permissions
The first error I noticed was MOM failing to adjust KSM via sysfs:
2011-11-22 10:13:48,313 - mom.Controllers.KSM - WARNING - KSM: Failed to write /sys/kernel/mm/ksm/run: Permission denied
MOM is used to running as root so that it can adjust these settings. I would prefer not to complicate the MOM architecture by having a separate process that receives instructions from the main MOM thread and then applies the requested changes as root.
Another solution would be to allow MOM to run as a completely separate daemon (as it has been originally doing). In this scenario, vdsm would reconfigure MOM by replacing the default configuration file and policy. vdsm could then interact with the running momd via the existing xmlrpc interface.
Thoughts on these issues?
----- Original Message -----
Hi guys,
I've got MOM up and running as a vdsm thread but I've hit two small problems. I'll bounce them around on the list to see if people have some ideas.
1.) Libvirt SASL authentication
I was able to easily modify MOM to connect to libvirt by hardcoding the vdsm credentials. Obviously this is not an acceptable long-term solution. What is the best way to share the vdsm libvirt password with MOM in a way that does not compromise security? Whatever method we choose should not involve vdsm-specific changes to MOM. For starters I think I will just place the username and password in the mom.conf file. We could make this file readable only by the vdsm user. Thoughts?
2.) Permissions
The first error I noticed was MOM failing to adjust KSM via sysfs:
2011-11-22 10:13:48,313 - mom.Controllers.KSM - WARNING - KSM: Failed to write /sys/kernel/mm/ksm/run: Permission denied
MOM is used to running as root so that it can adjust these settings. I would prefer not to complicate the MOM architecture by having a separate process that receives instructions from the main MOM thread and then applies the requested changes as root.
Another solution would be to allow MOM to run as a completely separate daemon (as it has been originally doing). In this scenario, vdsm would reconfigure MOM by replacing the default configuration file and policy. vdsm could then interact with the running momd via the existing xmlrpc interface.
Thoughts on these issues?
In continuation to our discussion from yesterday, both issues would be non-issues if we were to fully integrate MOM into vdsm. However, I get the feeling we might be mixing things here. Correct me if I'm wrong but the general architecture is: different data collectors -> MOM policy engine -> actions to perform Seeing as vdsm already today monitors and collects info from the guests, it should just be defined as a data collector and solve that side of the issue. Then we have the issue of performing the actions. Currently mom executes these directly, right? if so, we could split that so that mom's output is a series of actions to perform which vdsm could then execute (if this sounds familiar it is because this is how pacemaker works and I think they got it right).
This means that in the vdsm-less scenario we would be missing an orchestrator, but that could easily be adapted.
Then whether the policy engine runs as a daemon or is just spawned internally is up to the application calling it.
-- Adam Litke agl@us.ibm.com IBM Linux Technology Center
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
----- Original Message -----
From: "Adam Litke" agl@us.ibm.com To: vdsm-devel@lists.fedorahosted.org Sent: Tuesday, November 22, 2011 4:29:41 PM Subject: MOM integration questions
Hi guys,
I've got MOM up and running as a vdsm thread but I've hit two small problems. I'll bounce them around on the list to see if people have some ideas.
1.) Libvirt SASL authentication
I was able to easily modify MOM to connect to libvirt by hardcoding the vdsm credentials. Obviously this is not an acceptable long-term solution. What is the best way to share the vdsm libvirt password with MOM in a way that does not compromise security? Whatever method we choose should not involve vdsm-specific changes to MOM. For starters I think I will just place the username and password in the mom.conf file. We could make this file readable only by the vdsm user. Thoughts?
Is it safe just to read it from /etc/pki/vdsm/keys/libvirt_password
What's the reason for not wanting VDSM specific changes in MOM, is this project is part of oVirt then we can always assume VDSM is present.
2.) Permissions
The first error I noticed was MOM failing to adjust KSM via sysfs:
2011-11-22 10:13:48,313 - mom.Controllers.KSM - WARNING - KSM: Failed to write /sys/kernel/mm/ksm/run: Permission denied
MOM is used to running as root so that it can adjust these settings. I would prefer not to complicate the MOM architecture by having a separate process that receives instructions from the main MOM thread and then applies the requested changes as root.
Another solution would be to allow MOM to run as a completely separate daemon (as it has been originally doing). In this scenario, vdsm would reconfigure MOM by replacing the default configuration file and policy. vdsm could then interact with the running momd via the existing xmlrpc interface.
Thoughts on these issues?
-- Adam Litke agl@us.ibm.com IBM Linux Technology Center
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On Tue 22 Nov 2011 04:53:33 PM EST, Andrew Cathrow wrote:
----- Original Message -----
From: "Adam Litke"agl@us.ibm.com To: vdsm-devel@lists.fedorahosted.org Sent: Tuesday, November 22, 2011 4:29:41 PM Subject: MOM integration questions
Hi guys,
I've got MOM up and running as a vdsm thread but I've hit two small problems. I'll bounce them around on the list to see if people have some ideas.
1.) Libvirt SASL authentication
I was able to easily modify MOM to connect to libvirt by hardcoding the vdsm credentials. Obviously this is not an acceptable long-term solution. What is the best way to share the vdsm libvirt password with MOM in a way that does not compromise security? Whatever method we choose should not involve vdsm-specific changes to MOM. For starters I think I will just place the username and password in the mom.conf file. We could make this file readable only by the vdsm user. Thoughts?
Is it safe just to read it from /etc/pki/vdsm/keys/libvirt_password
What's the reason for not wanting VDSM specific changes in MOM, is this project is part of oVirt then we can always assume VDSM is present.
2.) Permissions
The first error I noticed was MOM failing to adjust KSM via sysfs:
2011-11-22 10:13:48,313 - mom.Controllers.KSM - WARNING - KSM: Failed to write /sys/kernel/mm/ksm/run: Permission denied
MOM is used to running as root so that it can adjust these settings. I would prefer not to complicate the MOM architecture by having a separate process that receives instructions from the main MOM thread and then applies the requested changes as root.
Another solution would be to allow MOM to run as a completely separate daemon (as it has been originally doing). In this scenario, vdsm would reconfigure MOM by replacing the default configuration file and policy. vdsm could then interact with the running momd via the existing xmlrpc interface.
Thoughts on these issues?
-- Adam Litkeagl@us.ibm.com IBM Linux Technology Center
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
I would like to +1 what Ayal said. Seperating MOM to be a policy engine and using other ovirt projects for collecting and performing actions through APIs will allow us to integrate better while allowing solutions not wishing to use the full ovirt bundle to implement these interfaces.
As to the sticking MOM on VDSM solution. supervdsm is used to perform root operations. But the operation has to be as specific as possible to prevent compromised svdm from doing to much damage.
On Tue, Nov 22, 2011 at 04:53:33PM -0500, Andrew Cathrow wrote:
----- Original Message -----
From: "Adam Litke" agl@us.ibm.com To: vdsm-devel@lists.fedorahosted.org Sent: Tuesday, November 22, 2011 4:29:41 PM Subject: MOM integration questions
Hi guys,
I've got MOM up and running as a vdsm thread but I've hit two small problems. I'll bounce them around on the list to see if people have some ideas.
1.) Libvirt SASL authentication
I was able to easily modify MOM to connect to libvirt by hardcoding the vdsm credentials. Obviously this is not an acceptable long-term solution. What is the best way to share the vdsm libvirt password with MOM in a way that does not compromise security? Whatever method we choose should not involve vdsm-specific changes to MOM. For starters I think I will just place the username and password in the mom.conf file. We could make this file readable only by the vdsm user. Thoughts?
Is it safe just to read it from /etc/pki/vdsm/keys/libvirt_password
What's the reason for not wanting VDSM specific changes in MOM, is this project is part of oVirt then we can always assume VDSM is present.
My initial integration strategy for MOM is to alter it such that vdsm can load the MOM logic as a python module and execute it in a new vdsmd thread. I believe the cleanest interface is possible when the interaction between MOM and vdsm is one way (ie. vdsm controls MOM). In this scenario, MOM would provide all of the APIs that vdsm needs to have the required level of control but MOM would not call back into VDSM. By doing this, MOM can remain focused on its specific role in the stack without being aware of idiosyncrasies in vdsm. There are two exceptions to this rule: Collectors and Controllers. VDSM can be a data source. In this case, we can write a vdsm Collector that knows how to query VDSM for host and guest statistics. VDSM can also be a 'tuning mechanism' against which we can write a Controller to execute certain privileged operations. What I am concerned about is the need to update supervdsm with a new command each time we want to tweak a new sysfs file.
2.) Permissions
The first error I noticed was MOM failing to adjust KSM via sysfs:
2011-11-22 10:13:48,313 - mom.Controllers.KSM - WARNING - KSM: Failed to write /sys/kernel/mm/ksm/run: Permission denied
MOM is used to running as root so that it can adjust these settings. I would prefer not to complicate the MOM architecture by having a separate process that receives instructions from the main MOM thread and then applies the requested changes as root.
Another solution would be to allow MOM to run as a completely separate daemon (as it has been originally doing). In this scenario, vdsm would reconfigure MOM by replacing the default configuration file and policy. vdsm could then interact with the running momd via the existing xmlrpc interface.
Thoughts on these issues?
On Tue, Nov 22, 2011 at 04:45:56PM -0500, Ayal Baron wrote:
----- Original Message -----
Hi guys,
I've got MOM up and running as a vdsm thread but I've hit two small problems. I'll bounce them around on the list to see if people have some ideas.
1.) Libvirt SASL authentication
I was able to easily modify MOM to connect to libvirt by hardcoding the vdsm credentials. Obviously this is not an acceptable long-term solution. What is the best way to share the vdsm libvirt password with MOM in a way that does not compromise security? Whatever method we choose should not involve vdsm-specific changes to MOM. For starters I think I will just place the username and password in the mom.conf file. We could make this file readable only by the vdsm user. Thoughts?
2.) Permissions
The first error I noticed was MOM failing to adjust KSM via sysfs:
2011-11-22 10:13:48,313 - mom.Controllers.KSM - WARNING - KSM: Failed to write /sys/kernel/mm/ksm/run: Permission denied
MOM is used to running as root so that it can adjust these settings. I would prefer not to complicate the MOM architecture by having a separate process that receives instructions from the main MOM thread and then applies the requested changes as root.
Another solution would be to allow MOM to run as a completely separate daemon (as it has been originally doing). In this scenario, vdsm would reconfigure MOM by replacing the default configuration file and policy. vdsm could then interact with the running momd via the existing xmlrpc interface.
Thoughts on these issues?
In continuation to our discussion from yesterday, both issues would be non-issues if we were to fully integrate MOM into vdsm. However, I get the feeling we might be mixing things here. Correct me if I'm wrong but the general architecture is: different data collectors -> MOM policy engine -> actions to perform
Yes, you have the architecture correctly.
Seeing as vdsm already today monitors and collects info from the guests, it should just be defined as a data collector and solve that side of the issue.
Yes, this should be just fine.
Then we have the issue of performing the actions. Currently mom executes these directly, right? if so, we could split that so that mom's output is a series of actions to perform which vdsm could then execute (if this sounds familiar it is because this is how pacemaker works and I think they got it right).
Yes, MOM currently executes the actions directly. Given that the goal of supervdsm is to provide a limited set of very specific actions, I am wondering how extensible it will be for flexible management tasks. I guess I will try it out with KSM first and see how it would work.
This means that in the vdsm-less scenario we would be missing an orchestrator, but that could easily be adapted.
In the vdsm-less scenario, we'd just use the existing sets of Collectors and Controllers.
Then whether the policy engine runs as a daemon or is just spawned internally is up to the application calling it.
Yes, this is how I have changed MOM to work now. The RPM installs a system-wide daemon that can be started. Otherwise, an application can load the module, configure it, and run it at will.
----- Original Message -----
On Tue, Nov 22, 2011 at 04:45:56PM -0500, Ayal Baron wrote:
----- Original Message -----
Hi guys,
I've got MOM up and running as a vdsm thread but I've hit two small problems. I'll bounce them around on the list to see if people have some ideas.
1.) Libvirt SASL authentication
I was able to easily modify MOM to connect to libvirt by hardcoding the vdsm credentials. Obviously this is not an acceptable long-term solution. What is the best way to share the vdsm libvirt password with MOM in a way that does not compromise security? Whatever method we choose should not involve vdsm-specific changes to MOM. For starters I think I will just place the username and password in the mom.conf file. We could make this file readable only by the vdsm user. Thoughts?
2.) Permissions
The first error I noticed was MOM failing to adjust KSM via sysfs:
2011-11-22 10:13:48,313 - mom.Controllers.KSM - WARNING - KSM: Failed to write /sys/kernel/mm/ksm/run: Permission denied
MOM is used to running as root so that it can adjust these settings. I would prefer not to complicate the MOM architecture by having a separate process that receives instructions from the main MOM thread and then applies the requested changes as root.
Another solution would be to allow MOM to run as a completely separate daemon (as it has been originally doing). In this scenario, vdsm would reconfigure MOM by replacing the default configuration file and policy. vdsm could then interact with the running momd via the existing xmlrpc interface.
Thoughts on these issues?
In continuation to our discussion from yesterday, both issues would be non-issues if we were to fully integrate MOM into vdsm. However, I get the feeling we might be mixing things here. Correct me if I'm wrong but the general architecture is: different data collectors -> MOM policy engine -> actions to perform
Yes, you have the architecture correctly.
Seeing as vdsm already today monitors and collects info from the guests, it should just be defined as a data collector and solve that side of the issue.
Yes, this should be just fine.
Then we have the issue of performing the actions. Currently mom executes these directly, right? if so, we could split that so that mom's output is a series of actions to perform which vdsm could then execute (if this sounds familiar it is because this is how pacemaker works and I think they got it right).
Yes, MOM currently executes the actions directly. Given that the goal of supervdsm is to provide a limited set of very specific actions, I am wondering how extensible it will be for flexible management tasks. I guess I will try it out with KSM first and see how it would work.
I don't see any other viable way of doing this securely. The policy basically determines what actions have to be run. MOM is fed this policy by unprivileged users. If mom were to run arbitrary commands using root user based on a policy given to it by a non root user, that would be a security breach. Unless I'm missing something, the alternative (assuming again mom is root) is to mandate having a static policy persistent on disk, but it would still need to get there securely and again poses a problem. I think having to expose specific (high level) tuning operations in the privileged process makes you think about the things you are doing and whether there is a safer way for doing the same thing.
This means that in the vdsm-less scenario we would be missing an orchestrator, but that could easily be adapted.
In the vdsm-less scenario, we'd just use the existing sets of Collectors and Controllers.
Then whether the policy engine runs as a daemon or is just spawned internally is up to the application calling it.
Yes, this is how I have changed MOM to work now. The RPM installs a system-wide daemon that can be started. Otherwise, an application can load the module, configure it, and run it at will.
-- Adam Litke agl@us.ibm.com IBM Linux Technology Center
On 11/23/2011 07:09 AM, Adam Litke wrote:
On Tue, Nov 22, 2011 at 04:53:33PM -0500, Andrew Cathrow wrote:
----- Original Message -----
From: "Adam Litke"agl@us.ibm.com To: vdsm-devel@lists.fedorahosted.org Sent: Tuesday, November 22, 2011 4:29:41 PM Subject: MOM integration questions
Hi guys,
I've got MOM up and running as a vdsm thread but I've hit two small problems. I'll bounce them around on the list to see if people have some ideas.
1.) Libvirt SASL authentication
I was able to easily modify MOM to connect to libvirt by hardcoding the vdsm credentials. Obviously this is not an acceptable long-term solution. What is the best way to share the vdsm libvirt password with MOM in a way that does not compromise security? Whatever method we choose should not involve vdsm-specific changes to MOM. For starters I think I will just place the username and password in the mom.conf file. We could make this file readable only by the vdsm user. Thoughts?
Is it safe just to read it from /etc/pki/vdsm/keys/libvirt_password
What's the reason for not wanting VDSM specific changes in MOM, is this project is part of oVirt then we can always assume VDSM is present.
My initial integration strategy for MOM is to alter it such that vdsm can load the MOM logic as a python module and execute it in a new vdsmd thread. I believe the cleanest interface is possible when the interaction between MOM and vdsm is one way (ie. vdsm controls MOM). In this scenario, MOM would provide all of the APIs that vdsm needs to have the required level of control but MOM would not call back into VDSM. By doing this, MOM can remain focused on its specific role in the stack without being aware of idiosyncrasies in vdsm. There are two exceptions to this rule: Collectors and Controllers. VDSM can be a data source. In this case, we can write a vdsm Collector that knows how to query VDSM for host and guest statistics. VDSM can also be a 'tuning mechanism' against which we can write a Controller to execute certain privileged operations. What I am concerned about is the need to update supervdsm with a new command each time we want to tweak a new sysfs file.
Maybe a bit prior to discussing the exact integration between MOM and vdsm, we can discuss the set of classes and verbs it should control?
For instance we have 2 major classes (further away in the future): 1. Host controls - Set swap partitions (maybe per guest or guest groups) - Set zram (compressed ram swap partition) - THP on|off - KSM speed control - Global host stat collection
2. Guest controls - KSM per VM, on|off - THP per VM - Ballooning & auto ballooning - Future free page hinting - cgroup memory cap (per vm|group) - (I guess memory hot plug is outside of MOMs scope, but a bit related) - Guest&agent stat collection
Will all be done by Mom? How about defining just a RFC interface and then try to hook into vdsm's api
2.) Permissions
The first error I noticed was MOM failing to adjust KSM via sysfs:
2011-11-22 10:13:48,313 - mom.Controllers.KSM - WARNING - KSM: Failed to write /sys/kernel/mm/ksm/run: Permission denied
MOM is used to running as root so that it can adjust these settings. I would prefer not to complicate the MOM architecture by having a separate process that receives instructions from the main MOM thread and then applies the requested changes as root.
Another solution would be to allow MOM to run as a completely separate daemon (as it has been originally doing). In this scenario, vdsm would reconfigure MOM by replacing the default configuration file and policy. vdsm could then interact with the running momd via the existing xmlrpc interface.
Thoughts on these issues?
On Tue, Nov 22, 2011 at 03:29:41PM -0600, Adam Litke wrote:
Hi guys,
I've got MOM up and running as a vdsm thread but I've hit two small problems. I'll bounce them around on the list to see if people have some ideas.
1.) Libvirt SASL authentication
I was able to easily modify MOM to connect to libvirt by hardcoding the vdsm credentials. Obviously this is not an acceptable long-term solution. What is the best way to share the vdsm libvirt password with MOM in a way that does not compromise security? Whatever method we choose should not involve vdsm-specific changes to MOM. For starters I think I will just place the username and password in the mom.conf file. We could make this file readable only by the vdsm user. Thoughts?
If you are part of the vdsm process, you can import libvirtconnection and use its get() function to obtain a libvirt connection.
If running as a separate process, you can either still Vdsm's password, or add another user to libvirt's sasl db.
Note that this SASL auth that we are using cannot really be called a "security measure". If mom.conf is read only, I don't see a problem with it.
2.) Permissions
The first error I noticed was MOM failing to adjust KSM via sysfs:
2011-11-22 10:13:48,313 - mom.Controllers.KSM - WARNING - KSM: Failed to write /sys/kernel/mm/ksm/run: Permission denied
MOM is used to running as root so that it can adjust these settings. I would prefer not to complicate the MOM architecture by having a separate process that receives instructions from the main MOM thread and then applies the requested changes as root.
Another solution would be to allow MOM to run as a completely separate daemon (as it has been originally doing). In this scenario, vdsm would reconfigure MOM by replacing the default configuration file and policy. vdsm could then interact with the running momd via the existing xmlrpc interface.
This would be the simplest integration, I believe. Note that currently, Vdsm has a thin interaction with ksmtuned which ships with qemu-kvm. This should probably be turned off or redirected to momd interaction.
Dan.
----- Original Message -----
On 11/23/2011 07:09 AM, Adam Litke wrote:
On Tue, Nov 22, 2011 at 04:53:33PM -0500, Andrew Cathrow wrote:
----- Original Message -----
From: "Adam Litke"agl@us.ibm.com To: vdsm-devel@lists.fedorahosted.org Sent: Tuesday, November 22, 2011 4:29:41 PM Subject: MOM integration questions
Hi guys,
I've got MOM up and running as a vdsm thread but I've hit two small problems. I'll bounce them around on the list to see if people have some ideas.
1.) Libvirt SASL authentication
I was able to easily modify MOM to connect to libvirt by hardcoding the vdsm credentials. Obviously this is not an acceptable long-term solution. What is the best way to share the vdsm libvirt password with MOM in a way that does not compromise security? Whatever method we choose should not involve vdsm-specific changes to MOM. For starters I think I will just place the username and password in the mom.conf file. We could make this file readable only by the vdsm user. Thoughts?
Is it safe just to read it from /etc/pki/vdsm/keys/libvirt_password
What's the reason for not wanting VDSM specific changes in MOM, is this project is part of oVirt then we can always assume VDSM is present.
My initial integration strategy for MOM is to alter it such that vdsm can load the MOM logic as a python module and execute it in a new vdsmd thread. I believe the cleanest interface is possible when the interaction between MOM and vdsm is one way (ie. vdsm controls MOM). In this scenario, MOM would provide all of the APIs that vdsm needs to have the required level of control but MOM would not call back into VDSM. By doing this, MOM can remain focused on its specific role in the stack without being aware of idiosyncrasies in vdsm. There are two exceptions to this rule: Collectors and Controllers. VDSM can be a data source. In this case, we can write a vdsm Collector that knows how to query VDSM for host and guest statistics. VDSM can also be a 'tuning mechanism' against which we can write a Controller to execute certain privileged operations. What I am concerned about is the need to update supervdsm with a new command each time we want to tweak a new sysfs file.
Maybe a bit prior to discussing the exact integration between MOM and vdsm, we can discuss the set of classes and verbs it should control?
For instance we have 2 major classes (further away in the future):
- Host controls
- Set swap partitions (maybe per guest or guest groups)
What does this have to do with a policy rule engine? these are static definitions (or at most configured once VM goes up/down, no tweaking required while vm is running) why define a swap partition per guest/group ?
- Set zram (compressed ram swap partition)
Again, on host this would be static (according to #CPUs), what does the rule engine have to do with it?
- THP on|off
Isn't this on by default? iirc there is no reason we know of to disable it? In any event, for it to be controlled by a policy engine, there should be a policy?
- KSM speed control
Doesn't mom do this today?
- Global host stat collection
collectd?
- Guest controls
- KSM per VM, on|off
- THP per VM
Why not always on?
- Ballooning & auto ballooning - Future free page hinting - cgroup memory cap (per vm|group)
I don't understand why this is not in the host section.
- (I guess memory hot plug is outside of MOMs scope, but a bit related)
Assuming the guest is able to support this and the policy states that the VM is allowed to reach a certain amount of memory, why not control this according to guest memory usage and memory availability? i.e. why would you consider this out of MOMs scope?
- Guest&agent stat collection
You focused on memory only. What about cpu? network? storage?
Will all be done by Mom? How about defining just a RFC interface and then try to hook into vdsm's api
2.) Permissions
The first error I noticed was MOM failing to adjust KSM via sysfs:
2011-11-22 10:13:48,313 - mom.Controllers.KSM - WARNING - KSM: Failed to write /sys/kernel/mm/ksm/run: Permission denied
MOM is used to running as root so that it can adjust these settings. I would prefer not to complicate the MOM architecture by having a separate process that receives instructions from the main MOM thread and then applies the requested changes as root.
Another solution would be to allow MOM to run as a completely separate daemon (as it has been originally doing). In this scenario, vdsm would reconfigure MOM by replacing the default configuration file and policy. vdsm could then interact with the running momd via the existing xmlrpc interface.
Thoughts on these issues?
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On 11/24/2011 01:12 AM, Ayal Baron wrote:
----- Original Message -----
On 11/23/2011 07:09 AM, Adam Litke wrote:
On Tue, Nov 22, 2011 at 04:53:33PM -0500, Andrew Cathrow wrote:
----- Original Message -----
From: "Adam Litke"agl@us.ibm.com To: vdsm-devel@lists.fedorahosted.org Sent: Tuesday, November 22, 2011 4:29:41 PM Subject: MOM integration questions
Hi guys,
I've got MOM up and running as a vdsm thread but I've hit two small problems. I'll bounce them around on the list to see if people have some ideas.
1.) Libvirt SASL authentication
I was able to easily modify MOM to connect to libvirt by hardcoding the vdsm credentials. Obviously this is not an acceptable long-term solution. What is the best way to share the vdsm libvirt password with MOM in a way that does not compromise security? Whatever method we choose should not involve vdsm-specific changes to MOM. For starters I think I will just place the username and password in the mom.conf file. We could make this file readable only by the vdsm user. Thoughts?
Is it safe just to read it from /etc/pki/vdsm/keys/libvirt_password
What's the reason for not wanting VDSM specific changes in MOM, is this project is part of oVirt then we can always assume VDSM is present.
My initial integration strategy for MOM is to alter it such that vdsm can load the MOM logic as a python module and execute it in a new vdsmd thread. I believe the cleanest interface is possible when the interaction between MOM and vdsm is one way (ie. vdsm controls MOM). In this scenario, MOM would provide all of the APIs that vdsm needs to have the required level of control but MOM would not call back into VDSM. By doing this, MOM can remain focused on its specific role in the stack without being aware of idiosyncrasies in vdsm. There are two exceptions to this rule: Collectors and Controllers. VDSM can be a data source. In this case, we can write a vdsm Collector that knows how to query VDSM for host and guest statistics. VDSM can also be a 'tuning mechanism' against which we can write a Controller to execute certain privileged operations. What I am concerned about is the need to update supervdsm with a new command each time we want to tweak a new sysfs file.
Maybe a bit prior to discussing the exact integration between MOM and vdsm, we can discuss the set of classes and verbs it should control?
For instance we have 2 major classes (further away in the future):
- Host controls
- Set swap partitions (maybe per guest or guest groups)
What does this have to do with a policy rule engine? these are static definitions (or at most configured once VM goes up/down, no tweaking required while vm is running) why define a swap partition per guest/group ?
Maybe you wouldn't want some of the guests to swap while you would like other guest to do so (in addition, you might want to use different storage for the VM swap on a per guest basis).
- Set zram (compressed ram swap partition)Again, on host this would be static (according to #CPUs), what does the rule engine have to do with it?
I didn't refer to a rule engine at all, as I noted, I first wanted to define the problem scope and only then discuss the specific api.
Why should the definition be static, it should be dynamic and per VM (the best, in theory, not necessarily in day 0)
- THP on|offIsn't this on by default? iirc there is no reason we know of to disable it? In any event, for it to be controlled by a policy engine, there should be a policy?
Yap. Sometimes we'll need to control it per VM, in theory we can shut it down completely to have khuged do nothing, probably no worth iy.
- KSM speed controlDoesn't mom do this today?
Correct, read the beginning
- Global host stat collectioncollectd?
Would that be send to ovirt-engine?
- Guest controls
- KSM per VM, on|off
- THP per VM
Why not always on?
Plenty of reasons, ksm is good for over commit bad wastes cpu and slow VMs.
- Ballooning& auto ballooning - Future free page hinting - cgroup memory cap (per vm|group)I don't understand why this is not in the host section.
- (I guess memory hot plug is outside of MOMs scope, but a bit related)Assuming the guest is able to support this and the policy states that the VM is allowed to reach a certain amount of memory, why not control this according to guest memory usage and memory availability? i.e. why would you consider this out of MOMs scope?
- Guest&agent stat collectionYou focused on memory only. What about cpu? network? storage?
Out of MOMs scope, aint it
Will all be done by Mom? How about defining just a RFC interface and then try to hook into vdsm's api
2.) Permissions
The first error I noticed was MOM failing to adjust KSM via sysfs:
2011-11-22 10:13:48,313 - mom.Controllers.KSM - WARNING - KSM: Failed to write /sys/kernel/mm/ksm/run: Permission denied
MOM is used to running as root so that it can adjust these settings. I would prefer not to complicate the MOM architecture by having a separate process that receives instructions from the main MOM thread and then applies the requested changes as root.
Another solution would be to allow MOM to run as a completely separate daemon (as it has been originally doing). In this scenario, vdsm would reconfigure MOM by replacing the default configuration file and policy. vdsm could then interact with the running momd via the existing xmlrpc interface.
Thoughts on these issues?
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On Mon, Nov 28, 2011 at 03:01:59PM +0200, Dor Laor wrote:
On 11/24/2011 01:12 AM, Ayal Baron wrote:
----- Original Message -----
On 11/23/2011 07:09 AM, Adam Litke wrote:
On Tue, Nov 22, 2011 at 04:53:33PM -0500, Andrew Cathrow wrote:
----- Original Message -----
From: "Adam Litke"agl@us.ibm.com To: vdsm-devel@lists.fedorahosted.org Sent: Tuesday, November 22, 2011 4:29:41 PM Subject: MOM integration questions
Hi guys,
I've got MOM up and running as a vdsm thread but I've hit two small problems. I'll bounce them around on the list to see if people have some ideas.
1.) Libvirt SASL authentication
I was able to easily modify MOM to connect to libvirt by hardcoding the vdsm credentials. Obviously this is not an acceptable long-term solution. What is the best way to share the vdsm libvirt password with MOM in a way that does not compromise security? Whatever method we choose should not involve vdsm-specific changes to MOM. For starters I think I will just place the username and password in the mom.conf file. We could make this file readable only by the vdsm user. Thoughts?
Is it safe just to read it from /etc/pki/vdsm/keys/libvirt_password
What's the reason for not wanting VDSM specific changes in MOM, is this project is part of oVirt then we can always assume VDSM is present.
My initial integration strategy for MOM is to alter it such that vdsm can load the MOM logic as a python module and execute it in a new vdsmd thread. I believe the cleanest interface is possible when the interaction between MOM and vdsm is one way (ie. vdsm controls MOM). In this scenario, MOM would provide all of the APIs that vdsm needs to have the required level of control but MOM would not call back into VDSM. By doing this, MOM can remain focused on its specific role in the stack without being aware of idiosyncrasies in vdsm. There are two exceptions to this rule: Collectors and Controllers. VDSM can be a data source. In this case, we can write a vdsm Collector that knows how to query VDSM for host and guest statistics. VDSM can also be a 'tuning mechanism' against which we can write a Controller to execute certain privileged operations. What I am concerned about is the need to update supervdsm with a new command each time we want to tweak a new sysfs file.
Maybe a bit prior to discussing the exact integration between MOM and vdsm, we can discuss the set of classes and verbs it should control?
For instance we have 2 major classes (further away in the future):
- Host controls
- Set swap partitions (maybe per guest or guest groups)
What does this have to do with a policy rule engine? these are static definitions (or at most configured once VM goes up/down, no tweaking required while vm is running) why define a swap partition per guest/group ?
Maybe you wouldn't want some of the guests to swap while you would like other guest to do so (in addition, you might want to use different storage for the VM swap on a per guest basis).
- Set zram (compressed ram swap partition)Again, on host this would be static (according to #CPUs), what does the rule engine have to do with it?
I didn't refer to a rule engine at all, as I noted, I first wanted to define the problem scope and only then discuss the specific api.
Why should the definition be static, it should be dynamic and per VM (the best, in theory, not necessarily in day 0)
- THP on|offIsn't this on by default? iirc there is no reason we know of to disable it? In any event, for it to be controlled by a policy engine, there should be a policy?
Yap. Sometimes we'll need to control it per VM, in theory we can shut it down completely to have khuged do nothing, probably no worth iy.
- KSM speed controlDoesn't mom do this today?
Correct, read the beginning
- Global host stat collectioncollectd?
Would that be send to ovirt-engine?
- Guest controls
- KSM per VM, on|off
- THP per VM
Why not always on?
Plenty of reasons, ksm is good for over commit bad wastes cpu and slow VMs.
We need to consider cloud-like environments where the ovirt administrator may not be the guest administrator. In such an environment, we probably don't want to be making guest OS tuning decisions without first consulting the customer who owns the guest.
- Ballooning& auto ballooning - Future free page hinting - cgroup memory cap (per vm|group)I don't understand why this is not in the host section.
- (I guess memory hot plug is outside of MOMs scope, but a bit related)Assuming the guest is able to support this and the policy states that the VM is allowed to reach a certain amount of memory, why not control this according to guest memory usage and memory availability? i.e. why would you consider this out of MOMs scope?
- Guest&agent stat collectionYou focused on memory only. What about cpu? network? storage?
Out of MOMs scope, aint it
No. I admit that the name "Memory Overcommitment Manager" would give you that impression. It was designed to solve the memory tuning problems but is sufficiently generic that it can tune anything. I fully expect to perform cpu, network, and storage tuning in the future. Some of those might actually be easier than memory :)
Will all be done by Mom? How about defining just a RFC interface and then try to hook into vdsm's api
2.) Permissions
The first error I noticed was MOM failing to adjust KSM via sysfs:
2011-11-22 10:13:48,313 - mom.Controllers.KSM - WARNING - KSM: Failed to write /sys/kernel/mm/ksm/run: Permission denied
MOM is used to running as root so that it can adjust these settings. I would prefer not to complicate the MOM architecture by having a separate process that receives instructions from the main MOM thread and then applies the requested changes as root.
Another solution would be to allow MOM to run as a completely separate daemon (as it has been originally doing). In this scenario, vdsm would reconfigure MOM by replacing the default configuration file and policy. vdsm could then interact with the running momd via the existing xmlrpc interface.
Thoughts on these issues?
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel@lists.stg.fedorahosted.org
-
abaron＠redhat.com -
agl＠us.ibm.com -
Andrew Cathrow -
Dan Kenigsberg -
Dor Laor -
smizrahi＠redhat.com