Can I get two +1's for:
commit 863cbad2434f5bf612a59cce7cfcb9b242177ad7 Author: Adrian Reber adrian@lisas.de Date: Wed Jul 29 19:20:19 2015 +0000
Reduce the number of parallel crawled hosts to 29.
There has been a report that the MM database was not updated correctly and dmesg on mm-crawler01 shows three OOM killed crawlers.
https://fedorahosted.org/fedora-infrastructure/ticket/4845
diff --git a/roles/mirrormanager/crawler/files/crawler.cron b/roles/mirrormanager/crawler/files/crawler.cron index 43f85f7..f77d61d 100644 --- a/roles/mirrormanager/crawler/files/crawler.cron +++ b/roles/mirrormanager/crawler/files/crawler.cron @@ -1,8 +1,8 @@ # run the crawler twice a day # logs sent to /var/log/mirrormanager/crawler.log and crawl/* by default -# 30 threads with 32GB of RAM seems to work so far +# 29 threads with 32GB of RAM seems to work so far # # [ "`hostname -s`" == "mm-crawler02" ] && sleep 2h is used to start the crawl # later on the second crawler to reduce the number of parallel accesses to # the database -0 */12 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 2h; /usr/bin/mm2_crawler --timeout-minutes 180 --threads 30 `/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1 +0 */12 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 2h; /usr/bin/mm2_crawler --timeout-minutes 180 --threads 29 `/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1
On Wed, 29 Jul 2015 21:24:02 +0200 Adrian Reber adrian@lisas.de wrote:
Can I get two +1's for:
+1.
Also, if we want, I could also give them another 32GB memory...
kevin
On Wed, Jul 29, 2015 at 01:48:57PM -0600, Kevin Fenzi wrote:
On Wed, 29 Jul 2015 21:24:02 +0200 Adrian Reber adrian@lisas.de wrote:
Can I get two +1's for:
+1.
Thanks Kevin and Pierre. Pushed.
Also, if we want, I could also give them another 32GB memory...
Either that or we can setup a crawler in Europe to crawl European mirrors from there. The crawler has the necessary functionality and we could try continent based crawls. I was aiming for after alpha freeze.
Adrian
On Wed, Jul 29, 2015 at 09:24:02PM +0200, Adrian Reber wrote:
Can I get two +1's for:
commit 863cbad2434f5bf612a59cce7cfcb9b242177ad7 Author: Adrian Reber adrian@lisas.de Date: Wed Jul 29 19:20:19 2015 +0000
Reduce the number of parallel crawled hosts to 29. There has been a report that the MM database was not updated correctly and dmesg on mm-crawler01 shows three OOM killed crawlers. https://fedorahosted.org/fedora-infrastructure/ticket/4845
diff --git a/roles/mirrormanager/crawler/files/crawler.cron b/roles/mirrormanager/crawler/files/crawler.cron index 43f85f7..f77d61d 100644 --- a/roles/mirrormanager/crawler/files/crawler.cron +++ b/roles/mirrormanager/crawler/files/crawler.cron @@ -1,8 +1,8 @@ # run the crawler twice a day # logs sent to /var/log/mirrormanager/crawler.log and crawl/* by default -# 30 threads with 32GB of RAM seems to work so far +# 29 threads with 32GB of RAM seems to work so far # # [ "`hostname -s`" == "mm-crawler02" ] && sleep 2h is used to start the crawl # later on the second crawler to reduce the number of parallel accesses to # the database -0 */12 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 2h; /usr/bin/mm2_crawler --timeout-minutes 180 --threads 30 `/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1 +0 */12 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 2h; /usr/bin/mm2_crawler --timeout-minutes 180 --threads 29 `/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1
+1 here as well
Pierre
infrastructure@lists.fedoraproject.org