We're running Fedora 19 with the 3.11.10-200.fc19.x86_64 kernel (just the normal RPM) on large servers (128GB RAM over two NUMA regions, each with one hex-core processor) with a large number of processes (more than 700, a couple hundred of which are active fairly frequently). We encounter situations where there system gets overwhelmed with migrate/N tasks from the kernel, based on what we've seen in top.
Here's what we've tried: * tuned-adm on latency-performance and virtual-host profiles * kernel.sched_migration_cost_ns=5000000 (which tuned will do for those profiles in v3.3/Fedora 20) * numad
Here's what we've used for analysis: * powertop * top/htop * perf record -a -g * SystemTap with code to print out migrations occurring * numatop
All we know is that the migration storms correlate with concurrent Chef runs verifying/configuring containers on the system.
Obviously, Chef invokes many things. But, most of the migrations we see are for Chef, the Ruby interpreter, the sh interpreter, and Munin. Responsiveness returns to normal after SystemTap reports a large set of chef-solo migrations, presumably at their completion.
David Strauss Pantheon Systems Fedora Server Working Group
Here's what we see in htop:
1 [||||||||||||||||||||||100.0%] 7 [|||||||||||||||||||||||99.9%] 13 [||||||||||||||||||||||100.0%] 19 [|||||||||||||||||||||||99.8%] 2 [|||||||||||||||||||||||99.7%] 8 [|||||||||||||||||||||||99.5%] 14 [|||||||||||||||||||||||99.2%] 20 [||||||||||||||||||||||100.0%] 3 [||||||||||||||||||||||100.0%] 9 [||||||||||||||||||||||100.0%] 15 [|||||||||||||||||||||||84.0%] 21 [|||||||||||||||||||||||99.3%] 4 [||||||||||||||||||||||100.0%] 10 [||||||||||||||||||||||100.0%] 16 [|||||||||||||||||||||||84.8%] 22 [|||||||||||||||||||||||98.5%] 5 [|||||||||||||||||||||||99.9%] 11 [||||||||||||||||||||||100.0%] 17 [||||||||||||||||||||||100.0%] 23 [|||||||||||||||||||||||91.2%] 6 [||||||||||||||||||||||100.0%] 12 [||||||||||||||||||||||100.0%] 18 [|||||||||||||||||||||||98.1%] 24 [||||||||||||||||||||||100.0%] Mem[||||||||||||||||||||||||||||||||||||||||||||||||||||28449/64386MB] Tasks: 693, 439 thr, 206 kthr; 226 running Swp[|| 882/32263MB] Load average: 173.90 88.38 44.13
Uptime: 3 days, 23:06:28
CPU PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 1 11 root RT 0 0 0 0 S 98.7 0.0 20:59.88 watchdog/0 6 34 root RT 0 0 0 0 R 96.1 0.0 1h29:17 migration/5 14 74 root RT 0 0 0 0 S 92.3 0.0 1h17:16 migration/13 18 94 root RT 0 0 0 0 R 94.0 0.0 1h15:50 migration/17 4 24 root RT 0 0 0 0 R 94.4 0.0 1h26:53 migration/3 24 124 root RT 0 0 0 0 R 90.3 0.0 1h16:31 migration/23 23 119 root RT 0 0 0 0 S 0.0 0.0 1h24:46 migration/22 7 39 root RT 0 0 0 0 R 0.0 0.0 1h35:30 migration/6 5 29 root RT 0 0 0 0 R 0.0 0.0 1h37:23 migration/4 17 89 root RT 0 0 0 0 R 0.0 0.0 1h25:49 migration/16 9 49 root RT 0 0 0 0 S 90.2 0.0 2h03:21 migration/8 8 44 root RT 0 0 0 0 R 88.4 0.0 1h26:33 migration/7 22 114 root RT 0 0 0 0 S 95.8 0.0 1h14:00 migration/21 19 99 root RT 0 0 0 0 R 93.3 0.0 1h24:25 migration/18 12 64 root RT 0 0 0 0 S 91.6 0.0 1h24:13 migration/11 15 79 root RT 0 0 0 0 S 98.9 0.0 1h27:46 migration/14 11 59 root RT 0 0 0 0 R 81.0 0.0 1h39:03 migration/10 13 69 root RT 0 0 0 0 S 0.0 0.0 1h22:20 migration/12 3 18 root RT 0 0 0 0 R 0.0 0.0 15:55.93 watchdog/2 2 13 root RT 0 0 0 0 R 90.1 0.0 1h31:53 migration/1 2 12 root RT 0 0 0 0 S 0.0 0.0 13:05.37 watchdog/1 16 84 root RT 0 0 0 0 S 88.3 0.0 1h22:50 migration/15 F1Help F2Setup F3SearchF4FilterF5Tree F6SortByF7Nice -F8Nice +F9Kill F10Quit
kernel@lists.fedoraproject.org
-
David Timothy Strauss