Jaroslav,
This is an attempt to firm up the rather vague discussion I had with
you on IRC. Hope it makes sense.
Currently, tuned manages system-wide tuning parameters for realtime
and NFV systems, mainly cpu isolation. That's fine for simple
deployments where we boot up, isolate some cpus, then start an
application that runs on those isolated cpus. It's not so good when we
start dealing with subsystems like Open VirtualSwitch (ovs) or stacks
like OpenStack. Those components have their own startup logic and
could use application specific tuning directions, such as cpu
placement, scheduling policy/priority, IRQ placement, etc.
Essentially, I'm asking that we come up with a way for tuned to parse
a file (or files) that contain tuning keypairs, and then provide those
keypairs when asked. The tuning info would be in one place and tuned
could have some logic to verify that the tuning's don't conflict.
For example, let's say a system has 44 cores and using
tuned-profiles-nfv, we've specified that 8 of those cores should be
isolated (isolcpus=36-43). Additionally, we want to run ovs on four of
the cores and my-special-app on the other four, so we could have a
file somewhere that looks like this:
[ovs]
cpulist=36-39
policy=fifo
priority=20
[my-special-app]
cpulist=40-43
policy=fifo
priority=10
What I envision is the ovs startup logic querying tuned with the key
'ovs' and getting back the keypairs above, which it could then use
properly to place the ovs threads on the appropriate cpus at the
specified scheduling policy and priority. Ditto when the
my-special-app startup logic runs.
Additionally, we don't currently have a good way to add arbitrary
kernel parameters to the kernel command line. We can get isolcpus=
added via the tuned-profiles-{realtime,nfv} packages, but if you want
to add intel_idle.max_cstate= or rcu_nocb_poll, or a host of others,
you're off editing /etc/default/grub. It might be nice to have a file
section called 'boot', where you can add kernel command line
parameters and possibly IRQ affinity placement. Not a high priority
for me, but it might be nice to collect all this tuning info into one
place:
[boot]
isolcpus=36-43
irq_affinity= !${isolcpus}
params='debug intel_idle.max_cstate=1'
[ovs]
cpulist=36-39
policy=fifo
priority=20
[my-special-app]
cpulist=40-43
policy=fifo
priority=10
Bottom line is that I think I'm talking about a tuned plugin that
becomes a clearinghouse for tuning hints. The plugin parses the info,
checks to make sure that things like cpu lists and irqs all make
sense, possibly updates the grub config with info from the [boot]
section and then just replies when someone queries for info using a
particular key (e.g. 'ovs') with a list of the keypairs.
Thoughts?
Clark
