Hi all,
I would like to revert greenwave's gunicorn timeout to the default 30s. Changing it to 10+min was useful for our investigation but it now slows down bodhi cron job. Also the greenwave team is working on fix for this so long term we should not need to have a specific timeout.
I am leaving in place the longer OpenShift timeout on the route, so that it is easier to know where the timeout comes from.
diff --git a/roles/openshift-apps/greenwave/templates/buildconfig.yml b/roles/openshift-apps/greenwave/templates/buildconfig.yml index d836b482d..7bb5420ba 100644 --- a/roles/openshift-apps/greenwave/templates/buildconfig.yml +++ b/roles/openshift-apps/greenwave/templates/buildconfig.yml @@ -30,7 +30,6 @@ spec:
# Become non-root again USER 1001 - ENTRYPOINT docker/install-ca.sh && gunicorn-3 --workers 8 --timeout 660 --graceful-timeout 300 --bind 0.0.0.0:8080 --access-logfile=- --enable-stdio-inheritance greenwave.wsgi:app strategy: type: Docker dockerStrategy:
On Wed, Mar 20, 2019 at 08:22:09AM +0100, Clement Verna wrote:
Hi all,
I would like to revert greenwave's gunicorn timeout to the default 30s. Changing it to 10+min was useful for our investigation but it now slows down bodhi cron job. Also the greenwave team is working on fix for this so long term we should not need to have a specific timeout.
I am leaving in place the longer OpenShift timeout on the route, so that it is easier to know where the timeout comes from.
diff --git a/roles/openshift-apps/greenwave/templates/buildconfig.yml b/roles/openshift-apps/greenwave/templates/buildconfig.yml index d836b482d..7bb5420ba 100644 --- a/roles/openshift-apps/greenwave/templates/buildconfig.yml +++ b/roles/openshift-apps/greenwave/templates/buildconfig.yml @@ -30,7 +30,6 @@ spec:
# Become non-root again USER 1001
ENTRYPOINT docker/install-ca.sh && gunicorn-3 --workers 8
--timeout 660 --graceful-timeout 300 --bind 0.0.0.0:8080 --access-logfile=- --enable-stdio-inheritance greenwave.wsgi:app strategy: type: Docker dockerStrategy:
Our investigation has shown that increasing the timeout does reduce the error rate. I am +1 to reducing it because 10 minutes is definitely way too big, but -1 to put it back to 30 seconds.
Pierre
On Wed, 20 Mar 2019 at 09:57, Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Wed, Mar 20, 2019 at 08:22:09AM +0100, Clement Verna wrote:
Hi all,
I would like to revert greenwave's gunicorn timeout to the default 30s. Changing it to 10+min was useful for our investigation but it now slows down bodhi cron job. Also the greenwave team is working on fix for this so long term we should not need to have a specific timeout.
I am leaving in place the longer OpenShift timeout on the route, so that it is easier to know where the timeout comes from.
diff --git a/roles/openshift-apps/greenwave/templates/buildconfig.yml b/roles/openshift-apps/greenwave/templates/buildconfig.yml index d836b482d..7bb5420ba 100644 --- a/roles/openshift-apps/greenwave/templates/buildconfig.yml +++ b/roles/openshift-apps/greenwave/templates/buildconfig.yml @@ -30,7 +30,6 @@ spec:
# Become non-root again USER 1001
ENTRYPOINT docker/install-ca.sh && gunicorn-3 --workers 8
--timeout 660 --graceful-timeout 300 --bind 0.0.0.0:8080 --access-logfile=- --enable-stdio-inheritance greenwave.wsgi:app strategy: type: Docker dockerStrategy:
Our investigation has shown that increasing the timeout does reduce the error rate. I am +1 to reducing it because 10 minutes is definitely way too big, but -1 to put it back to 30 seconds.
Sure which value would you suggests ?
Pierre _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Wed, Mar 20, 2019 at 10:00:26AM +0100, Clement Verna wrote:
On Wed, 20 Mar 2019 at 09:57, Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Wed, Mar 20, 2019 at 08:22:09AM +0100, Clement Verna wrote:
Hi all,
I would like to revert greenwave's gunicorn timeout to the default 30s. Changing it to 10+min was useful for our investigation but it now slows down bodhi cron job. Also the greenwave team is working on fix for this so long term we should not need to have a specific timeout.
I am leaving in place the longer OpenShift timeout on the route, so that it is easier to know where the timeout comes from.
diff --git a/roles/openshift-apps/greenwave/templates/buildconfig.yml b/roles/openshift-apps/greenwave/templates/buildconfig.yml index d836b482d..7bb5420ba 100644 --- a/roles/openshift-apps/greenwave/templates/buildconfig.yml +++ b/roles/openshift-apps/greenwave/templates/buildconfig.yml @@ -30,7 +30,6 @@ spec:
# Become non-root again USER 1001
ENTRYPOINT docker/install-ca.sh && gunicorn-3 --workers 8
--timeout 660 --graceful-timeout 300 --bind 0.0.0.0:8080 --access-logfile=- --enable-stdio-inheritance greenwave.wsgi:app strategy: type: Docker dockerStrategy:
Our investigation has shown that increasing the timeout does reduce the error rate. I am +1 to reducing it because 10 minutes is definitely way too big, but -1 to put it back to 30 seconds.
Sure which value would you suggests ?
We have the data*, we should just look at it :)
* We have stored the time it took to query greenwave for each of the 1500+ updates from bodhi we've tested (83 of which have more than one build), so we should be able to look at it and see if there something in there to help us make a reasonable decision.
Pierre
On Wed, 20 Mar 2019 at 05:15, Pierre-Yves Chibon pingou@pingoured.fr wrote:
Sure which value would you suggests ?
We have the data*, we should just look at it :)
- We have stored the time it took to query greenwave for each of the 1500+
updates from bodhi we've tested (83 of which have more than one build), so we should be able to look at it and see if there something in there to help us make a reasonable decision.
If you aim me at the data I can help come up with some valid numbers.. and I would suggest that we look at a 80% fit. If we find a number which meets 80% of queries then we are good.
Pierre _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Wed, 20 Mar 2019 at 11:52, Stephen John Smoogen smooge@gmail.com wrote:
On Wed, 20 Mar 2019 at 05:15, Pierre-Yves Chibon pingou@pingoured.fr wrote:
Sure which value would you suggests ?
We have the data*, we should just look at it :)
- We have stored the time it took to query greenwave for each of the 1500+
updates from bodhi we've tested (83 of which have more than one build), so we should be able to look at it and see if there something in there to help us make a reasonable decision.
If you aim me at the data I can help come up with some valid numbers.. and I would suggest that we look at a 80% fit. If we find a number which meets 80% of queries then we are good.
Thanks smooge.
For the record the data location was shared on IRC.
Pierre _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
-- Stephen J Smoogen. _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
After looking at the data it seems that a timeout of about 2 minutes would improve the situation without making bodhi's cron job too long.
This is my revised patch
diff --git a/roles/openshift-apps/greenwave/templates/buildconfig.yml b/roles/openshift-apps/greenwave/templates/buildconfig.yml index d836b482d..3ed86258f 100644 --- a/roles/openshift-apps/greenwave/templates/buildconfig.yml +++ b/roles/openshift-apps/greenwave/templates/buildconfig.yml @@ -30,7 +30,7 @@ spec:
# Become non-root again USER 1001 - ENTRYPOINT docker/install-ca.sh && gunicorn-3 --workers 8 --timeout 660 --graceful-timeout 300 --bind 0.0.0.0:8080 --access-logfile=- --enable-stdio-inheritance greenwave.wsgi:app + ENTRYPOINT docker/install-ca.sh && gunicorn-3 --workers 8 --timeout 127 --bind 0.0.0.0:8080 --access-logfile=- --enable-stdio-inheritance greenwave.wsgi:app strategy: type: Docker dockerStrategy:
On Wed, Mar 20, 2019 at 01:32:09PM +0100, Clement Verna wrote:
After looking at the data it seems that a timeout of about 2 minutes would improve the situation without making bodhi's cron job too long.
This is my revised patch
diff --git a/roles/openshift-apps/greenwave/templates/buildconfig.yml b/roles/openshift-apps/greenwave/templates/buildconfig.yml index d836b482d..3ed86258f 100644 --- a/roles/openshift-apps/greenwave/templates/buildconfig.yml +++ b/roles/openshift-apps/greenwave/templates/buildconfig.yml @@ -30,7 +30,7 @@ spec:
# Become non-root again USER 1001
ENTRYPOINT docker/install-ca.sh && gunicorn-3 --workers 8
--timeout 660 --graceful-timeout 300 --bind 0.0.0.0:8080 --access-logfile=- --enable-stdio-inheritance greenwave.wsgi:app
ENTRYPOINT docker/install-ca.sh && gunicorn-3 --workers 8
--timeout 127 --bind 0.0.0.0:8080 --access-logfile=- --enable-stdio-inheritance greenwave.wsgi:app strategy: type: Docker dockerStrategy: -- 2.20.1
+1s ? -1s ?
+1 for me :)
Pierre
+1
On Wed, 20 Mar 2019 at 09:40, Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Wed, Mar 20, 2019 at 01:32:09PM +0100, Clement Verna wrote:
After looking at the data it seems that a timeout of about 2 minutes would improve the situation without making bodhi's cron job too long.
This is my revised patch
diff --git a/roles/openshift-apps/greenwave/templates/buildconfig.yml b/roles/openshift-apps/greenwave/templates/buildconfig.yml index d836b482d..3ed86258f 100644 --- a/roles/openshift-apps/greenwave/templates/buildconfig.yml +++ b/roles/openshift-apps/greenwave/templates/buildconfig.yml @@ -30,7 +30,7 @@ spec:
# Become non-root again USER 1001
ENTRYPOINT docker/install-ca.sh && gunicorn-3 --workers 8
--timeout 660 --graceful-timeout 300 --bind 0.0.0.0:8080 --access-logfile=- --enable-stdio-inheritance greenwave.wsgi:app
ENTRYPOINT docker/install-ca.sh && gunicorn-3 --workers 8
--timeout 127 --bind 0.0.0.0:8080 --access-logfile=- --enable-stdio-inheritance greenwave.wsgi:app strategy: type: Docker dockerStrategy: -- 2.20.1
+1s ? -1s ?
+1 for me :)
Pierre _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
Thanks this was pushed, I now need someone to run the following playbook
playbooks/openshift-apps/greenwave.yml -t apply-buildconfig
On Wed, 20 Mar 2019 at 14:52, Stephen John Smoogen smooge@gmail.com wrote:
+1
On Wed, 20 Mar 2019 at 09:40, Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Wed, Mar 20, 2019 at 01:32:09PM +0100, Clement Verna wrote:
After looking at the data it seems that a timeout of about 2 minutes would improve the situation without making bodhi's cron job too long.
This is my revised patch
diff --git a/roles/openshift-apps/greenwave/templates/buildconfig.yml b/roles/openshift-apps/greenwave/templates/buildconfig.yml index d836b482d..3ed86258f 100644 --- a/roles/openshift-apps/greenwave/templates/buildconfig.yml +++ b/roles/openshift-apps/greenwave/templates/buildconfig.yml @@ -30,7 +30,7 @@ spec:
# Become non-root again USER 1001
ENTRYPOINT docker/install-ca.sh && gunicorn-3 --workers 8
--timeout 660 --graceful-timeout 300 --bind 0.0.0.0:8080 --access-logfile=- --enable-stdio-inheritance greenwave.wsgi:app
ENTRYPOINT docker/install-ca.sh && gunicorn-3 --workers 8
--timeout 127 --bind 0.0.0.0:8080 --access-logfile=- --enable-stdio-inheritance greenwave.wsgi:app strategy: type: Docker dockerStrategy: -- 2.20.1
+1s ? -1s ?
+1 for me :)
Pierre _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
-- Stephen J Smoogen. _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Wed, Mar 20, 2019 at 11:57:22AM +0100, Clement Verna wrote:
On Wed, 20 Mar 2019 at 11:52, Stephen John Smoogen smooge@gmail.com wrote:
On Wed, 20 Mar 2019 at 05:15, Pierre-Yves Chibon pingou@pingoured.fr wrote:
Sure which value would you suggests ?
We have the data*, we should just look at it :)
- We have stored the time it took to query greenwave for each of the 1500+
updates from bodhi we've tested (83 of which have more than one build), so we should be able to look at it and see if there something in there to help us make a reasonable decision.
If you aim me at the data I can help come up with some valid numbers.. and I would suggest that we look at a 80% fit. If we find a number which meets 80% of queries then we are good.
Thanks smooge.
For the record the data location was shared on IRC.
The location shared on IRC isn't accessible to everyone, but the data is also publicly accessible in: https://pagure.io/cpe/bodhi_greenwave_investigation/ and more precisely, all the results are in: https://pagure.io/cpe/bodhi_greenwave_investigation/blob/master/f/results
Pierre
infrastructure@lists.fedoraproject.org