Greetings.
Over this month so far, we have had mailman01 alert 27 times about swap space being low. When we look, it is indeed low, but there is still lots of memory available, so a swapoff -a && swapon -a "fixes" it.
My theory as to what is happening is that from time to time there's a Django thread that takes up lots and lots of memory, causing the machine to swap out other things, then it completes, leaving lots of things swapped out, but memory available. I have in fact seen the web-ui take up a ton of memory.
Right now we have the webui set to:
maximum-requests=1000 processes=4 threads=4
We may want to adjust that? Perhaps less processes and more threads?
However, one simple thing we can do right now is to just throw more memory at it. The virthost it's on has a lot of memory to spare, so it would be trivial to just double it's memory and see if the issue goes away. (from 16GB to 32GB). This would require a quick reboot, but downtime should be very short.
Thoughts? +1s?
kevin
On Mon, May 9, 2016 at 2:57 PM, Kevin Fenzi kevin@scrye.com wrote:
Greetings.
Over this month so far, we have had mailman01 alert 27 times about swap space being low. When we look, it is indeed low, but there is still lots of memory available, so a swapoff -a && swapon -a "fixes" it.
My theory as to what is happening is that from time to time there's a Django thread that takes up lots and lots of memory, causing the machine to swap out other things, then it completes, leaving lots of things swapped out, but memory available. I have in fact seen the web-ui take up a ton of memory.
Right now we have the webui set to:
maximum-requests=1000 processes=4 threads=4
I would say to consider moving this down to 100 max requests.
We may want to adjust that? Perhaps less processes and more threads?
However, one simple thing we can do right now is to just throw more memory at it. The virthost it's on has a lot of memory to spare, so it would be trivial to just double it's memory and see if the issue goes away. (from 16GB to 32GB). This would require a quick reboot, but downtime should be very short.
Thoughts? +1s?
+1 for memory bump.
kevin
infrastructure mailing list infrastructure@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraprojec...
On Mon, May 09, 2016 at 03:04:00PM +0000, Patrick Uiterwijk wrote:
On Mon, May 9, 2016 at 2:57 PM, Kevin Fenzi kevin@scrye.com wrote:
Greetings.
Over this month so far, we have had mailman01 alert 27 times about swap space being low. When we look, it is indeed low, but there is still lots of memory available, so a swapoff -a && swapon -a "fixes" it.
My theory as to what is happening is that from time to time there's a Django thread that takes up lots and lots of memory, causing the machine to swap out other things, then it completes, leaving lots of things swapped out, but memory available. I have in fact seen the web-ui take up a ton of memory.
Right now we have the webui set to:
maximum-requests=1000 processes=4 threads=4
I would say to consider moving this down to 100 max requests.
We may want to adjust that? Perhaps less processes and more threads?
However, one simple thing we can do right now is to just throw more memory at it. The virthost it's on has a lot of memory to spare, so it would be trivial to just double it's memory and see if the issue goes away. (from 16GB to 32GB). This would require a quick reboot, but downtime should be very short.
Thoughts? +1s?
+1 for memory bump.
+1 for me as well, although figuring out which page is causing the issue would be nice (maybe something for minimot?)
Pierre
+1 for me as well, although figuring out which page is causing the issue would be nice (maybe something for minimot?)
I believe it's the fulltext indexing process that causes these memory feasts. In my understanding, it eats memory until the swap is full and it's OOM-killed, and afterwards the swap is not emptied back into the RAM. I need to look into optimizing that, but it's a bit tricky.
Aurélien
infrastructure@lists.fedoraproject.org