So in looking at a problem with greenwave getting hit with various web-bots I found that very few of our configured robots.txt for sites work because we normally proxypass everything to the backend. This causes extra work on each backend needing to either add this to their codebase (if they are a flask containers) and could be slow for other reasons. However this will change all robots.conf for all websites so I don't want to look at a FBR without discussion.
diff --git a/roles/httpd/website/templates/robots.conf b/roles/httpd/website/templates/robots.conf index f442128..a59eb68 100644 --- a/roles/httpd/website/templates/robots.conf +++ b/roles/httpd/website/templates/robots.conf @@ -1 +1,7 @@ +## Make sure that we don't skip this because we proxy pass it to a +## slow backend +<Location "/robots.txt"> + ProxyPass ! +</Location> + Alias /robots.txt /srv/web/{{site_name}}-robots.txt
On 3/18/19 7:43 AM, Stephen John Smoogen wrote:
So in looking at a problem with greenwave getting hit with various web-bots I found that very few of our configured robots.txt for sites work because we normally proxypass everything to the backend. This causes extra work on each backend needing to either add this to their codebase (if they are a flask containers) and could be slow for other reasons. However this will change all robots.conf for all websites so I don't want to look at a FBR without discussion.
diff --git a/roles/httpd/website/templates/robots.conf b/roles/httpd/website/templates/robots.conf index f442128..a59eb68 100644 --- a/roles/httpd/website/templates/robots.conf +++ b/roles/httpd/website/templates/robots.conf @@ -1 +1,7 @@ +## Make sure that we don't skip this because we proxy pass it to a +## slow backend +<Location "/robots.txt">
- ProxyPass !
+</Location>
Alias /robots.txt /srv/web/{{site_name}}-robots.txt
+1 here.
kevin
Anyone else have anything on this or should I move this to FBR for tomorrow?
On Mon, 18 Mar 2019 at 11:34, Kevin Fenzi kevin@scrye.com wrote:
On 3/18/19 7:43 AM, Stephen John Smoogen wrote:
So in looking at a problem with greenwave getting hit with various web-bots I found that very few of our configured robots.txt for sites work because we normally proxypass everything to the backend. This causes extra work on each backend needing to either add this to their codebase (if they are a flask containers) and could be slow for other reasons. However this will change all robots.conf for all websites so I don't want to look at a FBR without discussion.
diff --git a/roles/httpd/website/templates/robots.conf b/roles/httpd/website/templates/robots.conf index f442128..a59eb68 100644 --- a/roles/httpd/website/templates/robots.conf +++ b/roles/httpd/website/templates/robots.conf @@ -1 +1,7 @@ +## Make sure that we don't skip this because we proxy pass it to a +## slow backend +<Location "/robots.txt">
- ProxyPass !
+</Location>
Alias /robots.txt /srv/web/{{site_name}}-robots.txt
+1 here.
kevin
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
infrastructure@lists.fedoraproject.org