Hi all,
http://www.ovirt.org/wiki/Project_Proposal_-_MOM
The link above will take you to the oVirt draft proposal for including Memory Overcommitment Manager (MOM) as an oVirt sub-project. I wanted to give folks on this list a chance to discuss MOM and an integration strategy before I ask the oVirt board to evaluate the proposal. Please take a look if you can and provide your feedback. Thanks.
On Mon, Oct 10, 2011 at 04:39:55PM -0500, Adam Litke wrote:
Hi all,
http://www.ovirt.org/wiki/Project_Proposal_-_MOM
The link above will take you to the oVirt draft proposal for including Memory Overcommitment Manager (MOM) as an oVirt sub-project. I wanted to give folks on this list a chance to discuss MOM and an integration strategy before I ask the oVirt board to evaluate the proposal. Please take a look if you can and provide your feedback. Thanks.
Thank you for this opportunity. It is nice to have the pop of MOM around (sorry, couldn't resist).
All Vdsm currently has in this front is the ability to notify ksmtuned that a new VM has joined the frill or left it. I would love to have something smarter than ksmtuned's policy (which MOM's ksm.rules is quite similar ;-)) and ballooning support which we completely lack.
However I have a concern or two. I must admit that MOM's dynamic policies, and policy declaration language seems excessive to me. I cannot think of a use case where a policy cannot be hard-coded in Python as a plugin for a MOM-like daemon. What is the motivation for writing the policy in a special-purpose language? Is Vdsm supposed to own MOM's policy and feed MOM with it? Disclaimer: I've only had few minutes' glimpse at MOM's code.
Note that there is a problem in putting MOM alongside Vdsm since Vdsm is a bit egotistical in its access to libvirt, and denies libvirt write access from non-vdsm users.
Regards, Dan.
----- Original Message -----
On Mon, Oct 10, 2011 at 04:39:55PM -0500, Adam Litke wrote:
Hi all,
http://www.ovirt.org/wiki/Project_Proposal_-_MOM
The link above will take you to the oVirt draft proposal for including Memory Overcommitment Manager (MOM) as an oVirt sub-project. I wanted to give folks on this list a chance to discuss MOM and an integration strategy before I ask the oVirt board to evaluate the proposal. Please take a look if you can and provide your feedback. Thanks.
Thank you for this opportunity. It is nice to have the pop of MOM around (sorry, couldn't resist).
All Vdsm currently has in this front is the ability to notify ksmtuned that a new VM has joined the frill or left it. I would love to have something smarter than ksmtuned's policy (which MOM's ksm.rules is quite similar ;-)) and ballooning support which we completely lack.
However I have a concern or two. I must admit that MOM's dynamic policies, and policy declaration language seems excessive to me. I cannot think of a use case where a policy cannot be hard-coded in Python as a plugin for a MOM-like daemon. What is the motivation for writing the policy in a special-purpose language? Is Vdsm supposed to own MOM's policy and feed MOM with it? Disclaimer: I've only had few minutes' glimpse at MOM's code.
Note that there is a problem in putting MOM alongside Vdsm since Vdsm is a bit egotistical in its access to libvirt, and denies libvirt write access from non-vdsm users.
Actually this was discussed on the last vdsm sync call and Adam suggested that mom would have a mode where it is run from within vdsm (under vdsm context) so that should not be a problem.
Regards, Dan. _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On Tue, Oct 11, 2011 at 10:25:38PM +0200, Dan Kenigsberg wrote:
On Mon, Oct 10, 2011 at 04:39:55PM -0500, Adam Litke wrote:
Hi all,
http://www.ovirt.org/wiki/Project_Proposal_-_MOM
The link above will take you to the oVirt draft proposal for including Memory Overcommitment Manager (MOM) as an oVirt sub-project. I wanted to give folks on this list a chance to discuss MOM and an integration strategy before I ask the oVirt board to evaluate the proposal. Please take a look if you can and provide your feedback. Thanks.
[snip]
Note that there is a problem in putting MOM alongside Vdsm since Vdsm is a bit egotistical in its access to libvirt, and denies libvirt write access from non-vdsm users.
That's no technical obstacle, it is only to prevent accidental use of apps that don't play nicely with VDSM. Once you've verified and MOM and VDSM will play nicely together without throwing their toys out of the pram, you can simply add another SASL user/password for MOM to use :-)
One of the features we're planning to add to libvirt in the next 6months is RBAC control. This will let you write a strict policy for exactly which individual APIs you want to allow MOM (or any other libvirt app) to be allowed to use while VDSM is present.
Daniel
On Tue, Oct 11, 2011 at 10:25:38PM +0200, Dan Kenigsberg wrote:
On Mon, Oct 10, 2011 at 04:39:55PM -0500, Adam Litke wrote:
Hi all,
http://www.ovirt.org/wiki/Project_Proposal_-_MOM
The link above will take you to the oVirt draft proposal for including Memory Overcommitment Manager (MOM) as an oVirt sub-project. I wanted to give folks on this list a chance to discuss MOM and an integration strategy before I ask the oVirt board to evaluate the proposal. Please take a look if you can and provide your feedback. Thanks.
Thank you for this opportunity. It is nice to have the pop of MOM around (sorry, couldn't resist).
All Vdsm currently has in this front is the ability to notify ksmtuned that a new VM has joined the frill or left it. I would love to have something smarter than ksmtuned's policy (which MOM's ksm.rules is quite similar ;-)) and ballooning support which we completely lack.
However I have a concern or two. I must admit that MOM's dynamic policies, and policy declaration language seems excessive to me. I cannot think of a use case where a policy cannot be hard-coded in Python as a plugin for a MOM-like daemon. What is the motivation for writing the policy in a special-purpose language? Is Vdsm supposed to own MOM's policy and feed MOM with it? Disclaimer: I've only had few minutes' glimpse at MOM's code.
We thought about using static python code to implement policies but felt that would simply be too limiting for what we wanted to achieve. While I think the default MOM ballooning policy is pretty good, it is certainly not optimal yet and might not even do what some users would want. The current policy tries to mitigate host memory pressure by ballooning guests. Guests with more free memory are ballooned more aggressively. The thinking is that we could permit a management tool (engine-core in this case) apply a ballooning "profile" according to the administrator's wishes. MOM itself is intended to provide the mechanism for this while oVirt can provide the policies. The policy handling is not very complex and has been working well.
Note that there is a problem in putting MOM alongside Vdsm since Vdsm is a bit egotistical in its access to libvirt, and denies libvirt write access from non-vdsm users.
This is a very addressable problem. We can either use Dan's suggestion (add SASL credentials for MOM), or we can enable MOM to run in module context so that it will simply create some threads for itself in the VDSM daemon.
On Wed, Oct 12, 2011 at 09:46:53AM -0500, Adam Litke wrote:
On Tue, Oct 11, 2011 at 10:25:38PM +0200, Dan Kenigsberg wrote:
On Mon, Oct 10, 2011 at 04:39:55PM -0500, Adam Litke wrote:
Hi all,
http://www.ovirt.org/wiki/Project_Proposal_-_MOM
The link above will take you to the oVirt draft proposal for including Memory Overcommitment Manager (MOM) as an oVirt sub-project. I wanted to give folks on this list a chance to discuss MOM and an integration strategy before I ask the oVirt board to evaluate the proposal. Please take a look if you can and provide your feedback. Thanks.
Thank you for this opportunity. It is nice to have the pop of MOM around (sorry, couldn't resist).
All Vdsm currently has in this front is the ability to notify ksmtuned that a new VM has joined the frill or left it. I would love to have something smarter than ksmtuned's policy (which MOM's ksm.rules is quite similar ;-)) and ballooning support which we completely lack.
However I have a concern or two. I must admit that MOM's dynamic policies, and policy declaration language seems excessive to me. I cannot think of a use case where a policy cannot be hard-coded in Python as a plugin for a MOM-like daemon. What is the motivation for writing the policy in a special-purpose language? Is Vdsm supposed to own MOM's policy and feed MOM with it? Disclaimer: I've only had few minutes' glimpse at MOM's code.
We thought about using static python code to implement policies but felt that would simply be too limiting for what we wanted to achieve. While I think the default MOM ballooning policy is pretty good, it is certainly not optimal yet and might not even do what some users would want. The current policy tries to mitigate host memory pressure by ballooning guests. Guests with more free memory are ballooned more aggressively. The thinking is that we could permit a management tool (engine-core in this case) apply a ballooning "profile" according to the administrator's wishes. MOM itself is intended to provide the mechanism for this while oVirt can provide the policies. The policy handling is not very complex and has been working well.
I find it quite awkward for ovirt-engine to send code to MOM. It is not like the programmers of ovirt-engine are smarter than you and could devise a better policy. It *is* important to have "ballooning profiles" attached to guests (e.g. "this VM is holy, do not balloon unless host is in deep manure") or hosts ("keep 100Mb free at all costs"). I think that defining the available profiles is the business of MOM - no one can do it better.
Dan.
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP>
I find it quite awkward for ovirt-engine to send code to MOM. It is not like the programmers of ovirt-engine are smarter than you and could devise a better policy. It *is* important to have "ballooning profiles" attached to guests (e.g. "this VM is holy, do not balloon unless host is in deep manure") or hosts ("keep 100Mb free at all costs"). I think that defining the available profiles is the business of MOM - no one can do it better.
Dan.
I disagree, for one reason - the engine has an overall view of the system, not just of a single node. However, I haven't found yet where it can be advantage. I'm sure there are such cases, though. The knowledge of an operating system, its properties (to which user it belongs, its priority, is it related to another VM, ...) probably has some value. Y.
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP> >I find it quite awkward for ovirt-engine to send code to MOM. It is not like the >programmers of ovirt-engine are smarter than you and could devise a better >policy. It *is* important to have "ballooning profiles" attached to guests (e.g. >"this VM is holy, do not balloon unless host is in deep manure") or hosts ("keep >100Mb free at all costs"). I think that defining the available profiles is the >business of MOM - no one can do it better. > >Dan.
I disagree, for one reason - the engine has an overall view of the system, not just of a single node. However, I haven't found yet where it can be advantage. I'm sure there are such cases, though. The knowledge of an operating system, its properties (to which user it belongs, its priority, is it related to another VM, ...) probably has some value.
Sure, but these should be a finite set policies and their parameters. Management layer should be provided with useful knobs and handles, not the ability to push generic code.
Can you ever envisage ovirt-engine compiling a completely new policy on its own volition? This would require AI capabilities out of its current scope. The Engine may have several policies dumped in its database, and it would have to apply one of them to a host every time it changes to Up. The only advantage I see in this is that introducing a new policy in a new Engine version is simpler, and does not require updating all of its nodes.
Dan.
On 10/15/2011 11:05 PM, Dan Kenigsberg wrote:
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP> > I find it quite awkward for ovirt-engine to send code to MOM. It is not like the > programmers of ovirt-engine are smarter than you and could devise a better > policy. It *is* important to have "ballooning profiles" attached to guests (e.g. > "this VM is holy, do not balloon unless host is in deep manure") or hosts ("keep > 100Mb free at all costs"). I think that defining the available profiles is the > business of MOM - no one can do it better. > > Dan. I disagree, for one reason - the engine has an overall view of the system, not just of a single node. However, I haven't found yet where it can be advantage. I'm sure there are such cases, though. The knowledge of an operating system, its properties (to which user it belongs, its priority, is it related to another VM, ...) probably has some value.
Sure, but these should be a finite set policies and their parameters. Management layer should be provided with useful knobs and handles, not the ability to push generic code.
Can you ever envisage ovirt-engine compiling a completely new policy on its own volition? This would require AI capabilities out of its current scope. The Engine may have several policies dumped in its database, and it would have to apply one of them to a host every time it changes to Up. The only advantage I see in this is that introducing a new policy in a new Engine version is simpler, and does not require updating all of its nodes.
Dan.
Unless the customers want to write their own policies. Y.
----- Original Message -----
On 10/15/2011 11:05 PM, Dan Kenigsberg wrote:
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP> > I find it quite awkward for ovirt-engine to send code to MOM. It > is not like the > programmers of ovirt-engine are smarter than you and could devise > a better > policy. It *is* important to have "ballooning profiles" attached > to guests (e.g. > "this VM is holy, do not balloon unless host is in deep manure") > or hosts ("keep > 100Mb free at all costs"). I think that defining the available > profiles is the > business of MOM - no one can do it better. > > Dan. I disagree, for one reason - the engine has an overall view of the system, not just of a single node. However, I haven't found yet where it can be advantage. I'm sure there are such cases, though. The knowledge of an operating system, its properties (to which user it belongs, its priority, is it related to another VM, ...) probably has some value.
Sure, but these should be a finite set policies and their parameters. Management layer should be provided with useful knobs and handles, not the ability to push generic code.
Can you ever envisage ovirt-engine compiling a completely new policy on its own volition? This would require AI capabilities out of its current scope. The Engine may have several policies dumped in its database, and it would have to apply one of them to a host every time it changes to Up. The only advantage I see in this is that introducing a new policy in a new Engine version is simpler, and does not require updating all of its nodes.
Dan.
Unless the customers want to write their own policies.
Customers would be allowed to write their own policies but that does not mean write code. A policy would be defined by the finite set of parameters we allow to tweak. The fact that mom works with a rule language does not mean that that is the level we need to expose to rhevm. vdsm can define the set of policies and the parameters we wish to expose in the higher level API. I think MOM generality will prove to be a good thing the more we add abilities (policies involving I/O and CPU and Network and combinations of all of the above).
Y.
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On Sat, Oct 15, 2011 at 11:16:04PM +0200, Yaniv Kaul wrote:
On 10/15/2011 11:05 PM, Dan Kenigsberg wrote:
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP> > I find it quite awkward for ovirt-engine to send code to MOM. It is not like the > programmers of ovirt-engine are smarter than you and could devise a better > policy. It *is* important to have "ballooning profiles" attached to guests (e.g. > "this VM is holy, do not balloon unless host is in deep manure") or hosts ("keep > 100Mb free at all costs"). I think that defining the available profiles is the > business of MOM - no one can do it better. > > Dan. I disagree, for one reason - the engine has an overall view of the system, not just of a single node. However, I haven't found yet where it can be advantage. I'm sure there are such cases, though. The knowledge of an operating system, its properties (to which user it belongs, its priority, is it related to another VM, ...) probably has some value.
Sure, but these should be a finite set policies and their parameters. Management layer should be provided with useful knobs and handles, not the ability to push generic code.
Can you ever envisage ovirt-engine compiling a completely new policy on its own volition? This would require AI capabilities out of its current scope. The Engine may have several policies dumped in its database, and it would have to apply one of them to a host every time it changes to Up. The only advantage I see in this is that introducing a new policy in a new Engine version is simpler, and does not require updating all of its nodes.
Dan.
Unless the customers want to write their own policies.
Yes, and beyond customers think about vendors. MOM as designed allows for additional Collectors and Controllers to be dropped in and used by new policies. Imagine adding a storage appliance that comes with a recommended policy addition.
On Mon, Oct 17, 2011 at 09:19:15AM -0500, Adam Litke wrote:
On Sat, Oct 15, 2011 at 11:16:04PM +0200, Yaniv Kaul wrote:
On 10/15/2011 11:05 PM, Dan Kenigsberg wrote:
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP> > I find it quite awkward for ovirt-engine to send code to MOM. It is not like the > programmers of ovirt-engine are smarter than you and could devise a better > policy. It *is* important to have "ballooning profiles" attached to guests (e.g. > "this VM is holy, do not balloon unless host is in deep manure") or hosts ("keep > 100Mb free at all costs"). I think that defining the available profiles is the > business of MOM - no one can do it better. > > Dan. I disagree, for one reason - the engine has an overall view of the system, not just of a single node. However, I haven't found yet where it can be advantage. I'm sure there are such cases, though. The knowledge of an operating system, its properties (to which user it belongs, its priority, is it related to another VM, ...) probably has some value.
Sure, but these should be a finite set policies and their parameters. Management layer should be provided with useful knobs and handles, not the ability to push generic code.
Can you ever envisage ovirt-engine compiling a completely new policy on its own volition? This would require AI capabilities out of its current scope. The Engine may have several policies dumped in its database, and it would have to apply one of them to a host every time it changes to Up. The only advantage I see in this is that introducing a new policy in a new Engine version is simpler, and does not require updating all of its nodes.
Dan.
Unless the customers want to write their own policies.
Yes, and beyond customers think about vendors. MOM as designed allows for additional Collectors and Controllers to be dropped in and used by new policies. Imagine adding a storage appliance that comes with a recommended policy addition.
I think that this creates an uncomfortable marriage of code deployment and control. Customets, as vendors, do not need the MOM api to put a policy definition on the node - they can use rpm and yum, scp, or a shrink-wrapped image.
Dan.
On Sat, Oct 15, 2011 at 11:05:09PM +0200, Dan Kenigsberg wrote:
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP> >I find it quite awkward for ovirt-engine to send code to MOM. It is not like the >programmers of ovirt-engine are smarter than you and could devise a better >policy. It *is* important to have "ballooning profiles" attached to guests (e.g. >"this VM is holy, do not balloon unless host is in deep manure") or hosts ("keep >100Mb free at all costs"). I think that defining the available profiles is the >business of MOM - no one can do it better. > >Dan.
I disagree, for one reason - the engine has an overall view of the system, not just of a single node. However, I haven't found yet where it can be advantage. I'm sure there are such cases, though. The knowledge of an operating system, its properties (to which user it belongs, its priority, is it related to another VM, ...) probably has some value.
Sure, but these should be a finite set policies and their parameters. Management layer should be provided with useful knobs and handles, not the ability to push generic code.
See my other response for why I don't think it is scalable to expose simple knobs for each and every slight behavior modification we wish to perform.
Can you ever envisage ovirt-engine compiling a completely new policy on its own volition? This would require AI capabilities out of its current scope. The Engine may have several policies dumped in its database, and it would have to apply one of them to a host every time it changes to Up. The only advantage I see in this is that introducing a new policy in a new Engine version is simpler, and does not require updating all of its nodes.
ovirt-engine would most certainly not self-program its own policy. My thinking is that as a community, we would design a library of policies that are useful for implementing a set of supported operating profiles (eg. aggressive overcommitment, no overcommitment (SLA/guaranteed performance), power saving, maintenance, etc). These policy files could be pushed (with modifications to certain tunable parameters as required) to the appropriate nodes according to user instructions or as deemed necessary by ovirt-engine. MOM's job is merely to enforce whatever policy is given to it. To start with, we'll probably just have a single policy (the current one) but I think we should keep ourselves open to expansion.
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP> > I find it quite awkward for ovirt-engine to send code to MOM. It is not like the > programmers of ovirt-engine are smarter than you and could devise a better > policy. It *is* important to have "ballooning profiles" attached to guests (e.g. > "this VM is holy, do not balloon unless host is in deep manure") or hosts ("keep > 100Mb free at all costs"). I think that defining the available profiles is the > business of MOM - no one can do it better. > > Dan.
I disagree, for one reason - the engine has an overall view of the system, not just of a single node. However, I haven't found yet where it can be advantage. I'm sure there are such cases, though. The knowledge of an operating system, its properties (to which user it belongs, its priority, is it related to another VM, ...) probably has some value. Y.
One example I might put forth is a policy that cooperates with node fencing. If a user plans to fence a node (in order to perform a hardware upgrade or for some other reason), we may want to push a different policy to that node while virtual machines are migrated away from it.
Some may argue that this could be implemented in a permanent policy through manipulation of simple variables. I don't tend to agree. The current policy is pretty basic but as we add additional control points and dependencies, the system will become more complex. It will become increasingly more difficult to write a comprehensive policy that can expose enough simple 'control knobs' to achieve the desired results for all of the potential use cases.
On Mon, Oct 17, 2011 at 09:03:41AM -0500, Adam Litke wrote:
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP> > I find it quite awkward for ovirt-engine to send code to MOM. It is not like the > programmers of ovirt-engine are smarter than you and could devise a better > policy. It *is* important to have "ballooning profiles" attached to guests (e.g. > "this VM is holy, do not balloon unless host is in deep manure") or hosts ("keep > 100Mb free at all costs"). I think that defining the available profiles is the > business of MOM - no one can do it better. > > Dan.
I disagree, for one reason - the engine has an overall view of the system, not just of a single node. However, I haven't found yet where it can be advantage. I'm sure there are such cases, though. The knowledge of an operating system, its properties (to which user it belongs, its priority, is it related to another VM, ...) probably has some value. Y.
One example I might put forth is a policy that cooperates with node fencing. If a user plans to fence a node (in order to perform a hardware upgrade or for some other reason), we may want to push a different policy to that node while virtual machines are migrated away from it.
Some may argue that this could be implemented in a permanent policy through manipulation of simple variables. I don't tend to agree. The current policy is pretty basic but as we add additional control points and dependencies, the system will become more complex. It will become increasingly more difficult to write a comprehensive policy that can expose enough simple 'control knobs' to achieve the desired results for all of the potential use cases.
It makes perfect sense to expose a knob to choose a policy. I'm repeating myself repeating myself, but I see the policy(ies) as static entities, that are unlikely to change during a product release. Architecture is much simpler if the policies are expected to sit on the node and not transfered by ovirt-engine when it senses that the host came up.
Dan
On Tue, Oct 18, 2011 at 11:33:40AM +0200, Dan Kenigsberg wrote:
On Mon, Oct 17, 2011 at 09:03:41AM -0500, Adam Litke wrote:
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP> > I find it quite awkward for ovirt-engine to send code to MOM. It is not like the > programmers of ovirt-engine are smarter than you and could devise a better > policy. It *is* important to have "ballooning profiles" attached to guests (e.g. > "this VM is holy, do not balloon unless host is in deep manure") or hosts ("keep > 100Mb free at all costs"). I think that defining the available profiles is the > business of MOM - no one can do it better. > > Dan.
I disagree, for one reason - the engine has an overall view of the system, not just of a single node. However, I haven't found yet where it can be advantage. I'm sure there are such cases, though. The knowledge of an operating system, its properties (to which user it belongs, its priority, is it related to another VM, ...) probably has some value. Y.
One example I might put forth is a policy that cooperates with node fencing. If a user plans to fence a node (in order to perform a hardware upgrade or for some other reason), we may want to push a different policy to that node while virtual machines are migrated away from it.
Some may argue that this could be implemented in a permanent policy through manipulation of simple variables. I don't tend to agree. The current policy is pretty basic but as we add additional control points and dependencies, the system will become more complex. It will become increasingly more difficult to write a comprehensive policy that can expose enough simple 'control knobs' to achieve the desired results for all of the potential use cases.
It makes perfect sense to expose a knob to choose a policy. I'm repeating myself repeating myself, but I see the policy(ies) as static entities, that are unlikely to change during a product release. Architecture is much simpler if the policies are expected to sit on the node and not transfered by ovirt-engine when it senses that the host came up.
Sure. That is a sensible deployment strategy.
On 10/18/2011 05:05 PM, Adam Litke wrote:
On Tue, Oct 18, 2011 at 11:33:40AM +0200, Dan Kenigsberg wrote:
On Mon, Oct 17, 2011 at 09:03:41AM -0500, Adam Litke wrote:
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP> > I find it quite awkward for ovirt-engine to send code to MOM. It is not like the > programmers of ovirt-engine are smarter than you and could devise a better > policy. It *is* important to have "ballooning profiles" attached to guests (e.g. > "this VM is holy, do not balloon unless host is in deep manure") or hosts ("keep > 100Mb free at all costs"). I think that defining the available profiles is the > business of MOM - no one can do it better. > > Dan.
I disagree, for one reason - the engine has an overall view of the system, not just of a single node. However, I haven't found yet where it can be advantage. I'm sure there are such cases, though. The knowledge of an operating system, its properties (to which user it belongs, its priority, is it related to another VM, ...) probably has some value. Y.
One example I might put forth is a policy that cooperates with node fencing. If a user plans to fence a node (in order to perform a hardware upgrade or for some other reason), we may want to push a different policy to that node while virtual machines are migrated away from it.
Some may argue that this could be implemented in a permanent policy through manipulation of simple variables. I don't tend to agree. The current policy is pretty basic but as we add additional control points and dependencies, the system will become more complex. It will become increasingly more difficult to write a comprehensive policy that can expose enough simple 'control knobs' to achieve the desired results for all of the potential use cases.
It makes perfect sense to expose a knob to choose a policy. I'm repeating myself repeating myself, but I see the policy(ies) as static entities, that are unlikely to change during a product release. Architecture is much simpler if the policies are expected to sit on the node and not transfered by ovirt-engine when it senses that the host came up.
Sure. That is a sensible deployment strategy.
Another deployment strategy is:
1. The data-center admin defines a customized policy. He does that within the core-engine as this is the centralized management tool he uses for managing the data-center.
2. When adding a host to the data-center it pulls the policies from the core-engine.
3. The host is following/executing the policies (might also get fine tuned parameters for specific host).
4. If a policy needs some editing - we can edit the policy in the engine-core and have it notify the hosts in the data-center about the need to pull the new policies.
One obvious advantage in the above approach is that it can be done once and the system get's the policy distributed - no need for per host action by the administrator. Another advantage - the admin can set a more complicated policy that optimizes behavior based on info of more than one host.
We had previous discussions about policy-engines and pacemaker was one option.
If i understand correctly MOM is also executing a policy and not only defining the policy so most likely there is a place for both applications in the oVirt stack but it is better have them in sync early in the process. I think it might be interesting to have someone from the pacemaker team on this thread. Adding Andrew Beekhof IIRC he was on previous threads on this issue.
The way i see it we need a policy engine in the oVirt stack, I think that integration for defining the policy should be at the oVirt-engine level, the integration for the execution of the policy should be in host level but we better have a "common-language".
Livnat
----- Original Message -----
On 10/18/2011 05:05 PM, Adam Litke wrote:
On Tue, Oct 18, 2011 at 11:33:40AM +0200, Dan Kenigsberg wrote:
On Mon, Oct 17, 2011 at 09:03:41AM -0500, Adam Litke wrote:
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP> > I find it quite awkward for ovirt-engine to send code to MOM. > It is not like the > programmers of ovirt-engine are smarter than you and could > devise a better > policy. It *is* important to have "ballooning profiles" > attached to guests (e.g. > "this VM is holy, do not balloon unless host is in deep > manure") or hosts ("keep > 100Mb free at all costs"). I think that defining the available > profiles is the > business of MOM - no one can do it better. > > Dan.
I disagree, for one reason - the engine has an overall view of the system, not just of a single node. However, I haven't found yet where it can be advantage. I'm sure there are such cases, though. The knowledge of an operating system, its properties (to which user it belongs, its priority, is it related to another VM, ...) probably has some value. Y.
One example I might put forth is a policy that cooperates with node fencing. If a user plans to fence a node (in order to perform a hardware upgrade or for some other reason), we may want to push a different policy to that node while virtual machines are migrated away from it.
Some may argue that this could be implemented in a permanent policy through manipulation of simple variables. I don't tend to agree. The current policy is pretty basic but as we add additional control points and dependencies, the system will become more complex. It will become increasingly more difficult to write a comprehensive policy that can expose enough simple 'control knobs' to achieve the desired results for all of the potential use cases.
It makes perfect sense to expose a knob to choose a policy. I'm repeating myself repeating myself, but I see the policy(ies) as static entities, that are unlikely to change during a product release. Architecture is much simpler if the policies are expected to sit on the node and not transfered by ovirt-engine when it senses that the host came up.
Sure. That is a sensible deployment strategy.
Another deployment strategy is:
- The data-center admin defines a customized policy.
He does that within the core-engine as this is the centralized management tool he uses for managing the data-center.
- When adding a host to the data-center it pulls the policies from
the core-engine.
- The host is following/executing the policies (might also get fine
tuned parameters for specific host).
- If a policy needs some editing - we can edit the policy in the
engine-core and have it notify the hosts in the data-center about the need to pull the new policies.
One obvious advantage in the above approach is that it can be done once and the system get's the policy distributed - no need for per host action by the administrator.
1-4 Are fine and supported with Dan's approach (vdsm exposes a set of parameters, engine sets policy).
Another advantage - the admin can set a more complicated policy that optimizes behavior based on info of more than one host.
throttles memory usage across hosts? cpu? Also, if you push the policy down to the host, how would the host get the info from other hosts?
We had previous discussions about policy-engines and pacemaker was one option.
If i understand correctly MOM is also executing a policy and not only defining the policy so most likely there is a place for both applications in the oVirt stack but it is better have them in sync early in the process. I think it might be interesting to have someone from the pacemaker team on this thread. Adding Andrew Beekhof IIRC he was on previous threads on this issue.
We've already discussed this with Andrew and Pacemaker does not support throttling, only on-off. Introducing throttling in pacemaker (which uses lisp for the rule language) for 5K loc in python doesn't seem like the way to go.
The way i see it we need a policy engine in the oVirt stack, I think that integration for defining the policy should be at the oVirt-engine level, the integration for the execution of the policy should be in host level but we better have a "common-language".
Livnat
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On 10/24/2011 08:03 AM, Ayal Baron wrote:
----- Original Message -----
On 10/18/2011 05:05 PM, Adam Litke wrote:
On Tue, Oct 18, 2011 at 11:33:40AM +0200, Dan Kenigsberg wrote:
On Mon, Oct 17, 2011 at 09:03:41AM -0500, Adam Litke wrote:
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP> > I find it quite awkward for ovirt-engine to send code to MOM. > It is not like the > programmers of ovirt-engine are smarter than you and could > devise a better > policy. It *is* important to have "ballooning profiles" > attached to guests (e.g. > "this VM is holy, do not balloon unless host is in deep > manure") or hosts ("keep > 100Mb free at all costs"). I think that defining the available > profiles is the > business of MOM - no one can do it better. > > Dan.
I disagree, for one reason - the engine has an overall view of the system, not just of a single node. However, I haven't found yet where it can be advantage. I'm sure there are such cases, though. The knowledge of an operating system, its properties (to which user it belongs, its priority, is it related to another VM, ...) probably has some value. Y.
One example I might put forth is a policy that cooperates with node fencing. If a user plans to fence a node (in order to perform a hardware upgrade or for some other reason), we may want to push a different policy to that node while virtual machines are migrated away from it.
Some may argue that this could be implemented in a permanent policy through manipulation of simple variables. I don't tend to agree. The current policy is pretty basic but as we add additional control points and dependencies, the system will become more complex. It will become increasingly more difficult to write a comprehensive policy that can expose enough simple 'control knobs' to achieve the desired results for all of the potential use cases.
It makes perfect sense to expose a knob to choose a policy. I'm repeating myself repeating myself, but I see the policy(ies) as static entities, that are unlikely to change during a product release. Architecture is much simpler if the policies are expected to sit on the node and not transfered by ovirt-engine when it senses that the host came up.
Sure. That is a sensible deployment strategy.
Another deployment strategy is:
- The data-center admin defines a customized policy.
He does that within the core-engine as this is the centralized management tool he uses for managing the data-center.
- When adding a host to the data-center it pulls the policies from
the core-engine.
- The host is following/executing the policies (might also get fine
tuned parameters for specific host).
- If a policy needs some editing - we can edit the policy in the
engine-core and have it notify the hosts in the data-center about the need to pull the new policies.
One obvious advantage in the above approach is that it can be done once and the system get's the policy distributed - no need for per host action by the administrator.
1-4 Are fine and supported with Dan's approach (vdsm exposes a set of parameters, engine sets policy).
Another advantage - the admin can set a more complicated policy that optimizes behavior based on info of more than one host.
throttles memory usage across hosts? cpu? Also, if you push the policy down to the host, how would the host get the info from other hosts?
We had previous discussions about policy-engines and pacemaker was one option.
If i understand correctly MOM is also executing a policy and not only defining the policy so most likely there is a place for both applications in the oVirt stack but it is better have them in sync early in the process. I think it might be interesting to have someone from the pacemaker team on this thread. Adding Andrew Beekhof IIRC he was on previous threads on this issue.
We've already discussed this with Andrew and Pacemaker does not support throttling, only on-off. Introducing throttling in pacemaker (which uses lisp for the rule language) for 5K loc in python doesn't seem like the way to go.
Ah, yes I recall that conversation. Agreed.
The way i see it we need a policy engine in the oVirt stack, I think that integration for defining the policy should be at the oVirt-engine level, the integration for the execution of the policy should be in host level but we better have a "common-language".
Livnat
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On 10/24/2011 05:40 AM, Livnat Peer wrote:
On 10/18/2011 05:05 PM, Adam Litke wrote:
On Tue, Oct 18, 2011 at 11:33:40AM +0200, Dan Kenigsberg wrote:
On Mon, Oct 17, 2011 at 09:03:41AM -0500, Adam Litke wrote:
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP> > I find it quite awkward for ovirt-engine to send code to MOM. It is not like the > programmers of ovirt-engine are smarter than you and could devise a better > policy. It *is* important to have "ballooning profiles" attached to guests (e.g. > "this VM is holy, do not balloon unless host is in deep manure") or hosts ("keep > 100Mb free at all costs"). I think that defining the available profiles is the > business of MOM - no one can do it better. > > Dan.
I disagree, for one reason - the engine has an overall view of the system, not just of a single node. However, I haven't found yet where it can be advantage. I'm sure there are such cases, though. The knowledge of an operating system, its properties (to which user it belongs, its priority, is it related to another VM, ...) probably has some value. Y.
One example I might put forth is a policy that cooperates with node fencing. If a user plans to fence a node (in order to perform a hardware upgrade or for some other reason), we may want to push a different policy to that node while virtual machines are migrated away from it.
Some may argue that this could be implemented in a permanent policy through manipulation of simple variables. I don't tend to agree. The current policy is pretty basic but as we add additional control points and dependencies, the system will become more complex. It will become increasingly more difficult to write a comprehensive policy that can expose enough simple 'control knobs' to achieve the desired results for all of the potential use cases.
It makes perfect sense to expose a knob to choose a policy. I'm repeating myself repeating myself, but I see the policy(ies) as static entities, that are unlikely to change during a product release. Architecture is much simpler if the policies are expected to sit on the node and not transfered by ovirt-engine when it senses that the host came up.
Sure. That is a sensible deployment strategy.
Another deployment strategy is:
- The data-center admin defines a customized policy.
He does that within the core-engine as this is the centralized management tool he uses for managing the data-center.
- When adding a host to the data-center it pulls the policies from the
core-engine.
- The host is following/executing the policies (might also get fine
tuned parameters for specific host).
- If a policy needs some editing - we can edit the policy in the
engine-core and have it notify the hosts in the data-center about the need to pull the new policies.
One obvious advantage in the above approach is that it can be done once and the system get's the policy distributed - no need for per host action by the administrator. Another advantage - the admin can set a more complicated policy that optimizes behavior based on info of more than one host.
We had previous discussions about policy-engines and pacemaker was one option.
If i understand correctly MOM is also executing a policy and not only defining the policy so most likely there is a place for both applications in the oVirt stack but it is better have them in sync early in the process. I think it might be interesting to have someone from the pacemaker team on this thread. Adding Andrew Beekhof IIRC he was on previous threads on this issue.
If I'm reading the above properly, you're after something that figures out which objects (guests) go into which buckets (hosts) and in what order (including fencing and other types of recovery), so the the PE could well be a good fit.
All the interesting stuff is in a C library but we also recently added Python bindings. What language is being used here?
The way i see it we need a policy engine in the oVirt stack, I think that integration for defining the policy should be at the oVirt-engine level, the integration for the execution of the policy should be in host level but we better have a "common-language".
Livnat
On Mon, Oct 31, 2011 at 09:59:56PM +1100, Andrew Beekhof wrote:
On 10/24/2011 05:40 AM, Livnat Peer wrote:
On 10/18/2011 05:05 PM, Adam Litke wrote:
On Tue, Oct 18, 2011 at 11:33:40AM +0200, Dan Kenigsberg wrote:
On Mon, Oct 17, 2011 at 09:03:41AM -0500, Adam Litke wrote:
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP> >I find it quite awkward for ovirt-engine to send code to MOM. It is not like the >programmers of ovirt-engine are smarter than you and could devise a better >policy. It *is* important to have "ballooning profiles" attached to guests (e.g. >"this VM is holy, do not balloon unless host is in deep manure") or hosts ("keep >100Mb free at all costs"). I think that defining the available profiles is the >business of MOM - no one can do it better. > >Dan.
I disagree, for one reason - the engine has an overall view of the system, not just of a single node. However, I haven't found yet where it can be advantage. I'm sure there are such cases, though. The knowledge of an operating system, its properties (to which user it belongs, its priority, is it related to another VM, ...) probably has some value. Y.
One example I might put forth is a policy that cooperates with node fencing. If a user plans to fence a node (in order to perform a hardware upgrade or for some other reason), we may want to push a different policy to that node while virtual machines are migrated away from it.
Some may argue that this could be implemented in a permanent policy through manipulation of simple variables. I don't tend to agree. The current policy is pretty basic but as we add additional control points and dependencies, the system will become more complex. It will become increasingly more difficult to write a comprehensive policy that can expose enough simple 'control knobs' to achieve the desired results for all of the potential use cases.
It makes perfect sense to expose a knob to choose a policy. I'm repeating myself repeating myself, but I see the policy(ies) as static entities, that are unlikely to change during a product release. Architecture is much simpler if the policies are expected to sit on the node and not transfered by ovirt-engine when it senses that the host came up.
Sure. That is a sensible deployment strategy.
Another deployment strategy is:
- The data-center admin defines a customized policy.
He does that within the core-engine as this is the centralized management tool he uses for managing the data-center.
- When adding a host to the data-center it pulls the policies from the
core-engine.
- The host is following/executing the policies (might also get fine
tuned parameters for specific host).
- If a policy needs some editing - we can edit the policy in the
engine-core and have it notify the hosts in the data-center about the need to pull the new policies.
One obvious advantage in the above approach is that it can be done once and the system get's the policy distributed - no need for per host action by the administrator. Another advantage - the admin can set a more complicated policy that optimizes behavior based on info of more than one host.
We had previous discussions about policy-engines and pacemaker was one option.
If i understand correctly MOM is also executing a policy and not only defining the policy so most likely there is a place for both applications in the oVirt stack but it is better have them in sync early in the process. I think it might be interesting to have someone from the pacemaker team on this thread. Adding Andrew Beekhof IIRC he was on previous threads on this issue.
If I'm reading the above properly, you're after something that figures out which objects (guests) go into which buckets (hosts) and in what order (including fencing and other types of recovery), so the the PE could well be a good fit.
Actually, I haven't tackled VM placement with MOM yet. MOM is more about dynamic optimization of a single host by adjusting tunables in response to the observed conditions. One way that MOM could "plug in" to a placement manager is through a signaling mechanism. If (for example) MOM determines that a host is overloaded, it could use its policy to select a guest to evict and signal this intention to a placement manager. Then, the placement manager could actually perform a live migration of the targeted guest.
More to the point, MOM optimizes a single host. It sounds like Pacemaker might be complimentary at the cluster level.
All the interesting stuff is in a C library but we also recently added Python bindings. What language is being used here?
MOM is written in Python.
vdsm-devel@lists.stg.fedorahosted.org
-
abaron＠redhat.com -
agl＠us.ibm.com -
Andrew Beekhof -
Dan Kenigsberg -
Daniel P. Berrange -
lpeer＠redhat.com -
ykaul＠redhat.com