Hey all,
So I wrote a little script this morning to analyze all of bodhi's logs across app1-7. The results were quite interesting. Below are some of the top offenders, in the format of (# of tracebacks : exception)
# Database issues 134 : InterfaceError: connection already closed 1773 : OperationalError: terminating connection due to administrator command server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. 2566 : OperationalError: FATAL: connection limit exceeded for non-superusers
# FAS issues 115 : error: 35 126 : error: 52 236 : error: 6 1155 : ServerError: ServerError(https://admin.fedoraproject.org/accounts/, 500, Unknown HTTP Server Response) 1320 : error: 7 2758 : ServerError: ServerError(https://admin.fedoraproject.org/accounts/, 503, Unknown HTTP Server Response) 3539 : error: 18 5399 : ServerError: ServerError(https://admin.fedoraproject.org/accounts/, 502, Unknown HTTP Server Response)
The "error: #" messages are from pycurl within the TurboGears JSONFas identity/visit layer. PyCurl error descriptions can be found here: http://curl.haxx.se/libcurl/c/libcurl-errors.html
luke
On Thu, 5 Nov 2009, Luke Macken wrote:
Hey all,
So I wrote a little script this morning to analyze all of bodhi's logs across app1-7. The results were quite interesting. Below are some of the top offenders, in the format of (# of tracebacks : exception)
# Database issues 134 : InterfaceError: connection already closed 1773 : OperationalError: terminating connection due to administrator command server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. 2566 : OperationalError: FATAL: connection limit exceeded for non-superusers
# FAS issues 115 : error: 35 126 : error: 52 236 : error: 6 1155 : ServerError: ServerError(https://admin.fedoraproject.org/accounts/, 500, Unknown HTTP Server Response) 1320 : error: 7 2758 : ServerError: ServerError(https://admin.fedoraproject.org/accounts/, 503, Unknown HTTP Server Response) 3539 : error: 18 5399 : ServerError: ServerError(https://admin.fedoraproject.org/accounts/, 502, Unknown HTTP Server Response)
The "error: #" messages are from pycurl within the TurboGears JSONFas identity/visit layer. PyCurl error descriptions can be found here: http://curl.haxx.se/libcurl/c/libcurl-errors.html
out of how many total requests did we see these errors?
-Mike
On Thu, Nov 05, 2009 at 09:25:49AM -0600, Mike McGrath wrote:
On Thu, 5 Nov 2009, Luke Macken wrote:
Hey all,
So I wrote a little script this morning to analyze all of bodhi's logs across app1-7. The results were quite interesting. Below are some of the top offenders, in the format of (# of tracebacks : exception)
# Database issues 134 : InterfaceError: connection already closed 1773 : OperationalError: terminating connection due to administrator command server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. 2566 : OperationalError: FATAL: connection limit exceeded for non-superusers
# FAS issues 115 : error: 35 126 : error: 52 236 : error: 6 1155 : ServerError: ServerError(https://admin.fedoraproject.org/accounts/, 500, Unknown HTTP Server Response) 1320 : error: 7 2758 : ServerError: ServerError(https://admin.fedoraproject.org/accounts/, 503, Unknown HTTP Server Response) 3539 : error: 18 5399 : ServerError: ServerError(https://admin.fedoraproject.org/accounts/, 502, Unknown HTTP Server Response)
The "error: #" messages are from pycurl within the TurboGears JSONFas identity/visit layer. PyCurl error descriptions can be found here: http://curl.haxx.se/libcurl/c/libcurl-errors.html
out of how many total requests did we see these errors?
Out of ~18264206 requests.
luke
On Thu, Nov 05, 2009 at 10:19:02AM -0500, Luke Macken wrote:
# FAS issues 115 : error: 35 126 : error: 52 236 : error: 6 1155 : ServerError: ServerError(https://admin.fedoraproject.org/accounts/, 500, Unknown HTTP Server Response) 1320 : error: 7 2758 : ServerError: ServerError(https://admin.fedoraproject.org/accounts/, 503, Unknown HTTP Server Response) 3539 : error: 18 5399 : ServerError: ServerError(https://admin.fedoraproject.org/accounts/, 502, Unknown HTTP Server Response)
To elaborate on the FAS/curl related exceptions a bit more...
24 : CURLE_OPERATION_TIMEDOUT (28)
Operation timeout. The specified time-out period was reached according to the conditions.
115 : CURLE_SSL_CONNECT_ERROR (35)
A problem occurred somewhere in the SSL/TLS handshake. You really want the error buffer and read the message there as it pinpoints the problem slightly more. Could be certificates (file formats, paths, permissions), passwords, and others.
126 : CURLE_GOT_NOTHING (52)
Nothing was returned from the server, and under the circumstances, getting nothing is considered an error.
236 : CURLM_UNKNOWN_OPTION (6)
curl_multi_setopt() with unsupported option (Added in 7.15.4)
1155 : ServerError: ServerError(https://admin.fedoraproject.org/accounts/, 500, Unknown HTTP Server Response)
1320 : CURLE_COULDNT_CONNECT (7)
Failed to connect() to host or proxy.
2758 : ServerError: ServerError(https://admin.fedoraproject.org/accounts/, 503, Unknown HTTP Server Response)
3539 : CURLE_PARTIAL_FILE (18)
A file transfer was shorter or larger than expected. This happens when the server first reports an expected transfer size, and then delivers data that doesn't match the previously given size.
5399 : ServerError: ServerError(https://admin.fedoraproject.org/accounts/, 502, Unknown HTTP Server Response)
infrastructure@lists.fedoraproject.org