The other day I was running the stock fedora kernel on my ip forwarding setup, to see what the performance was, and the performance wasn't very good.
system is S5520HC dual socket 2.93GHz Xeon 5570 (Nehalem) with 3 quad port 82580 adapters (12 ports). Traffic is bidirectional 64 byte packets being forwarded and received on each port, basically port to port routing. I am only using 12 flows currently.
The driver is igb, and I am using an affinity script that lines up each pair of ports that are forwarding traffic into optimal configurations for cache locality. I am also disabling remote_node_defrag_ratio to stop cross node traffic.
With the fedora default kernel from F14 it appears that CONFIG_NETFILTER=y means that I cannot unload all of netfilter even if I stop iptables service.
perf showed netfilter being prominent, and removing it gives me much higher throughput. Is there a reason CONFIG_NETFILTER=y ? Isn't it a good thing to be able to disable netfilter if you want to?
Jesse
From: Jesse Brandeburg jesse.brandeburg@gmail.com Date: Wed, 6 Apr 2011 11:51:06 -0700
perf showed netfilter being prominent, and removing it gives me much higher throughput. Is there a reason CONFIG_NETFILTER=y ? Isn't it a good thing to be able to disable netfilter if you want to?
CONFIG_NETFILTER=y is the only possible non-disabled setting, as it's a boolean.
All that is enabling is the hooks, and it's not possible to provide netfilter at all without the hooks being compiled into the kernel image.
Fedora by default is also probably installing several defauly netfilter rules, make sure to remove them.
Em Wed, Apr 06, 2011 at 12:12:08PM -0700, David Miller escreveu:
From: Jesse Brandeburg jesse.brandeburg@gmail.com Date: Wed, 6 Apr 2011 11:51:06 -0700
perf showed netfilter being prominent, and removing it gives me much higher throughput. Is there a reason CONFIG_NETFILTER=y ? Isn't it a good thing to be able to disable netfilter if you want to?
CONFIG_NETFILTER=y is the only possible non-disabled setting, as it's a boolean.
All that is enabling is the hooks, and it's not possible to provide netfilter at all without the hooks being compiled into the kernel image.
Fedora by default is also probably installing several defauly netfilter rules, make sure to remove them.
Was there ever any attempt to try to have netfilter built-in but not causing the performance impact?
Something like ftrace code changing when the user inserts the first rule?
People wanting top performance disable it in the build, but thos wanting to stick to vendor provided kernels don't have that choice :)
- Arnaldo
From: Arnaldo Carvalho de Melo acme@ghostprotocols.net Date: Wed, 6 Apr 2011 16:57:19 -0300
Something like ftrace code changing when the user inserts the first rule?
People wanting top performance disable it in the build, but thos wanting to stick to vendor provided kernels don't have that choice :)
Using ftrace-like stubs would be an interesting idea, and I highly encourage people to work on something like that.
However I want to reiterate that I think that real rules are installed in Jesse's case, and once he removes those the majority of the overhead will disappear. The FC14 workstation I'm using right now, on which I've made no modifications to the installer's netfilter settings, has the following rules:
-------------------- [root@ilbolle davem]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT icmp -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh ACCEPT udp -- anywhere anywhere state NEW udp dpt:ipp ACCEPT udp -- anywhere 224.0.0.251 state NEW udp dpt:mdns ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ipp ACCEPT udp -- anywhere anywhere state NEW udp dpt:ipp REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT) target prot opt source destination REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@ilbolle davem]# --------------------
I suspect Jesse has something similar on his test box.
When no rules are loaded, all the stubs make happen is a function call plus a list_empty() check. Nothing more. I really can't see that, all by itself, obliterating routing performance.
In fact I've done udp flood tests, as recently as a month ago, with just NETFILTER=y and no rules installed, and the impact was minimal.
And that was on sparc64 where function calls are expensive :)
Em Wed, Apr 06, 2011 at 01:02:39PM -0700, David Miller escreveu:
From: Arnaldo Carvalho de Melo acme@ghostprotocols.net Date: Wed, 6 Apr 2011 16:57:19 -0300
Something like ftrace code changing when the user inserts the first rule?
People wanting top performance disable it in the build, but thos wanting to stick to vendor provided kernels don't have that choice :)
Using ftrace-like stubs would be an interesting idea, and I highly encourage people to work on something like that.
cool, these code modification and JIT mechanizms open up a lot of possibilities indeed ;-)
However I want to reiterate that I think that real rules are installed in Jesse's case, and once he removes those the majority of the overhead will disappear. The FC14 workstation I'm using right now, on
which I've made no modifications to the installer's netfilter settings, has the following rules:
<SNIP>
I suspect Jesse has something similar on his test box.
When no rules are loaded, all the stubs make happen is a function call plus a list_empty() check. Nothing more. I really can't see that, all by itself, obliterating routing performance.
Yeah, would be nice, since he is playing with it, for him to post numbers about the overheads.
In fact I've done udp flood tests, as recently as a month ago, with just NETFILTER=y and no rules installed, and the impact was minimal.
And that was on sparc64 where function calls are expensive :)
:-)
- Arnaldo
Le mercredi 06 avril 2011 à 13:02 -0700, David Miller a écrit :
From: Arnaldo Carvalho de Melo acme@ghostprotocols.net Date: Wed, 6 Apr 2011 16:57:19 -0300
Something like ftrace code changing when the user inserts the first rule?
People wanting top performance disable it in the build, but thos wanting to stick to vendor provided kernels don't have that choice :)
Using ftrace-like stubs would be an interesting idea, and I highly encourage people to work on something like that.
However I want to reiterate that I think that real rules are installed in Jesse's case, and once he removes those the majority of the overhead will disappear. The FC14 workstation I'm using right now, on which I've made no modifications to the installer's netfilter settings, has the following rules:
[root@ilbolle davem]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT icmp -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh ACCEPT udp -- anywhere anywhere state NEW udp dpt:ipp ACCEPT udp -- anywhere 224.0.0.251 state NEW udp dpt:mdns ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ipp ACCEPT udp -- anywhere anywhere state NEW udp dpt:ipp REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT) target prot opt source destination REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@ilbolle davem]#
I suspect Jesse has something similar on his test box.
I suspect problem is worse than that.
I remember last time I work on a fedora kernel, it had conntrack enabled
And yes, conntrack can really slowdown a router, because of default parameters.
cat /proc/sys/net/nf_conntrack_max
From: Eric Dumazet eric.dumazet@gmail.com Date: Wed, 06 Apr 2011 22:18:56 +0200
I remember last time I work on a fedora kernel, it had conntrack enabled
And yes, conntrack can really slowdown a router, because of default parameters.
Yes, if conntrack is enabled performance will indeed tank.
Le mercredi 06 avril 2011 à 13:29 -0700, David Miller a écrit :
From: Eric Dumazet eric.dumazet@gmail.com Date: Wed, 06 Apr 2011 22:18:56 +0200
I remember last time I work on a fedora kernel, it had conntrack enabled
And yes, conntrack can really slowdown a router, because of default parameters.
Yes, if conntrack is enabled performance will indeed tank.
I just check on latest fedora14 (yum updated) :
- conntrack is statically included in kernel, you cannot remove it.
Hmm...
On Wed, 6 Apr 2011, Eric Dumazet wrote:
Le mercredi 06 avril 2011 à 13:29 -0700, David Miller a écrit :
From: Eric Dumazet eric.dumazet@gmail.com Date: Wed, 06 Apr 2011 22:18:56 +0200
I remember last time I work on a fedora kernel, it had conntrack enabled
And yes, conntrack can really slowdown a router, because of default parameters.
Yes, if conntrack is enabled performance will indeed tank.
I just check on latest fedora14 (yum updated) :
- conntrack is statically included in kernel, you cannot remove it.
Hmm...
Thanks for the replies, Yes indeed, I bet that conntrack was the item that caused the grief I saw. In fact turning off CONFIG_NETFILTER disables conntrack and solved my problem. :-)
Now the question is how to get the netfiltery/conntracky goodness without impacting those who want to go fast, maybe without a helmet.
Le mercredi 06 avril 2011 à 14:08 -0700, Brandeburg, Jesse a écrit :
Thanks for the replies, Yes indeed, I bet that conntrack was the item that caused the grief I saw. In fact turning off CONFIG_NETFILTER disables conntrack and solved my problem. :-)
Now the question is how to get the netfiltery/conntracky goodness without impacting those who want to go fast, maybe without a helmet.
Thats pretty simple, just load netfilter modules if necessary.
If not loaded, cost is pretty small, being predicted branches.
kernel@lists.fedoraproject.org
-
Arnaldo Carvalho de Melo -
Brandeburg, Jesse -
David Miller -
Eric Dumazet -
Jesse Brandeburg