Since ansiblefest I've been kinda looking at these every morning because I've been thinking about making a push based playbook for it. But everytime i've looked it seems as though they occur at 9pm my time. Does that give us a clue about what might be causing it?
-Toshio ---------- Forwarded message ---------- From: "Cron Daemon" root@fedoraproject.org Date: Nov 1, 2013 9:02 PM Subject: Cron root@proxy05 /usr/local/bin/lock-wrapper fasClient "/bin/sleep $(($RANDOM % 100)); /usr/bin/fasClient -i | /usr/local/bin/nag-once fassync 1d 2>&1" To: root@fedoraproject.org Cc:
Traceback (most recent call last): File "/usr/bin/fasClient", line 776, in <module> users = fas.filter_users(valid_groups=valid_groups, restricted_groups=restricted_groups) File "/usr/bin/fasClient", line 350, in filter_users if uid not in self.users: File "/usr/bin/fasClient", line 222, in _refresh_users self._users = self.user_data() File "/usr/lib/python2.6/site-packages/fedora/client/fas2.py", line 794, in user_data request = self.send_request('json/fas_client/user_data', auth=True) File "/usr/lib/python2.6/site-packages/fedora/client/baseclient.py", line 355, in send_request auth_params=auth_params, retries=retries, timeout=timeout) File "/usr/lib/python2.6/site-packages/fedora/client/proxyclient.py", line 479, in send_request {'url': to_bytes(url), 'err': to_bytes(e)}) fedora.client.ServerError: ServerError( https://admin.fedoraproject.org/accounts/json/fas_client/user_data, 200, Error returned from json module while processing https://admin.fedoraproject.org/accounts/json/fas_client/user_data: Expecting object: line 1 column 2059567 (char 2059567))
On Fri, 1 Nov 2013 21:41:47 -0700 Toshio Kuratomi a.badger@gmail.com wrote:
Since ansiblefest I've been kinda looking at these every morning because I've been thinking about making a push based playbook for it. But everytime i've looked it seems as though they occur at 9pm my time. Does that give us a clue about what might be causing it?
Yes, these are all at 4am UTC, when the daily cron job runs.
Its fas01/02/03's httpd's restarting after logrotate, and breaking any in progress fasClient connections.
At least I am pretty sure this is the case.
kevin
On Sat, Nov 02, 2013 at 11:03:47AM -0600, Kevin Fenzi wrote:
Yes, these are all at 4am UTC, when the daily cron job runs.
Its fas01/02/03's httpd's restarting after logrotate, and breaking any in progress fasClient connections.
At least I am pretty sure this is the case.
ah... so if they did the equivalent of reload instead of restart, we should stop getting these?
-Toshio
On Sat, 2 Nov 2013 16:28:25 -0700 Toshio Kuratomi a.badger@gmail.com wrote:
On Sat, Nov 02, 2013 at 11:03:47AM -0600, Kevin Fenzi wrote:
Yes, these are all at 4am UTC, when the daily cron job runs.
Its fas01/02/03's httpd's restarting after logrotate, and breaking any in progress fasClient connections.
At least I am pretty sure this is the case.
ah... so if they did the equivalent of reload instead of restart, we should stop getting these?
Yeah, but the reason if I recall right that the RHEL6 httpd logrotate doesn't do this is that if you reload/graceful, there may well be running children writing to the logs after it. So, you can't move the log as it still might have processes writing to it, so they do a HUP/restart to make sure logs are not being used before rotate.
I guess we could try switching it and see if we see any issues... if we compress logs they could get corrupted, but if we don't they might be ok.
kevin
On Sat, Nov 02, 2013 at 05:48:37PM -0600, Kevin Fenzi wrote:
running children writing to the logs after it. So, you can't move the log as it still might have processes writing to it, so they do a HUP/restart to make sure logs are not being used before rotate.
I guess we could try switching it and see if we see any issues... if we compress logs they could get corrupted, but if we don't they might be ok.
The workaround here is to use `delaycompress` in logrotate.
In the past I've used copy & truncate log rotate feature. Kinda racy though... some log bits might get lost between the copy & truncate parts. (on extremely loaded httpd systems)
The perk is the inode number stays the same. So there is even less reason to HUP Apache.
-Jon Disnard
On Sat, Nov 2, 2013 at 9:46 PM, Matthew Miller mattdm@fedoraproject.org wrote:
On Sat, Nov 02, 2013 at 05:48:37PM -0600, Kevin Fenzi wrote:
running children writing to the logs after it. So, you can't move the log as it still might have processes writing to it, so they do a HUP/restart to make sure logs are not being used before rotate.
I guess we could try switching it and see if we see any issues... if we compress logs they could get corrupted, but if we don't they might be ok.
The workaround here is to use `delaycompress` in logrotate.
-- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ mattdm@fedoraproject.org _______________________________________________ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
ok, lets try this thing:
diff --git a/modules/httpd/files/httpd.logrotate b/modules/httpd/files/httpd.logrotate index 3a6e6cb..e55f078 100644 --- a/modules/httpd/files/httpd.logrotate +++ b/modules/httpd/files/httpd.logrotate @@ -9,8 +9,9 @@ compressext .bz2 dateext sharedscripts + delaycompress postrotate - /sbin/service httpd reload > /dev/null 2>/dev/null || true + /sbin/service httpd graceful > /dev/null 2>/dev/null || true endscript }
kevin
If you go with: copytruncate
Then you can avoid the apache reload or graceful all together.
The log file inode never changes, so it need no reload.
just fyi -Jon
On Wed, Nov 13, 2013 at 11:08 AM, Kevin Fenzi kevin@scrye.com wrote:
ok, lets try this thing:
diff --git a/modules/httpd/files/httpd.logrotate b/modules/httpd/files/httpd.logrotate index 3a6e6cb..e55f078 100644 --- a/modules/httpd/files/httpd.logrotate +++ b/modules/httpd/files/httpd.logrotate @@ -9,8 +9,9 @@ compressext .bz2 dateext sharedscripts
- delaycompress postrotate
/sbin/service httpd reload > /dev/null 2>/dev/null || true
endscript/sbin/service httpd graceful > /dev/null 2>/dev/null || true
}
kevin
infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
On Wed, 13 Nov 2013 13:03:52 -0600 Jon jdisnard@gmail.com wrote:
If you go with: copytruncate
Then you can avoid the apache reload or graceful all together.
The log file inode never changes, so it need no reload.
Right. I'm going to give this a try now...
My main concern is that we have some things that are leaky and the reload clears them out, but if thats the case it will be good to know that too. ;)
kevin
On Sat, Nov 02, 2013 at 04:28:25PM -0700, Toshio Kuratomi wrote:
Yes, these are all at 4am UTC, when the daily cron job runs. Its fas01/02/03's httpd's restarting after logrotate, and breaking any in progress fasClient connections. At least I am pretty sure this is the case.
ah... so if they did the equivalent of reload instead of restart, we should stop getting these?
They should probably do 'graceful':
https://bugzilla.redhat.com/show_bug.cgi?id=221073
infrastructure@lists.fedoraproject.org