I'm not sure if we have any memcached experience on the list but I thought I'd ask. Can anyone explain this:
Notice how memcached1 has a much higher hit rate and memcached2 has a much lower hit rate?
-Mike
On Wed, Jul 1, 2009 at 3:33 PM, Mike McGrathmmcgrath@redhat.com wrote:
I'm not sure if we have any memcached experience on the list but I thought I'd ask. Can anyone explain this:
Notice how memcached1 has a much higher hit rate and memcached2 has a much lower hit rate?
The time for memcached1 is 5x less than memcached2 being up. That can have an effect on caching as right after a system comes up its rates are usually much higher and then over time fall off (iirc). I think it would take bringing both up at the same time to figure out if there is a true disparity over caching.
-Mike
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
On Wed, Jul 1, 2009 at 3:33 PM, Mike McGrathmmcgrath@redhat.com wrote:
I'm not sure if we have any memcached experience on the list but I thought I'd ask. Can anyone explain this:
Notice how memcached1 has a much higher hit rate and memcached2 has a much lower hit rate?
The time for memcached1 is 5x less than memcached2 being up. That can have an effect on caching as right after a system comes up its rates are usually much higher and then over time fall off (iirc). I think it would take bringing both up at the same time to figure out if there is a true disparity over caching.
I thought that exact same thing :)
memcahed1: STAT uptime 9143 STAT get_hits 311736 STAT get_misses 11255
memcached2: STAT uptime 9144 STAT get_hits 49679 STAT get_misses 11116
-Mike
On Wed, Jul 1, 2009 at 6:08 PM, Mike McGrathmmcgrath@redhat.com wrote:
On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
On Wed, Jul 1, 2009 at 3:33 PM, Mike McGrathmmcgrath@redhat.com wrote:
I'm not sure if we have any memcached experience on the list but I thought I'd ask. Can anyone explain this:
Notice how memcached1 has a much higher hit rate and memcached2 has a much lower hit rate?
The time for memcached1 is 5x less than memcached2 being up. That can have an effect on caching as right after a system comes up its rates are usually much higher and then over time fall off (iirc). I think it would take bringing both up at the same time to figure out if there is a true disparity over caching.
I thought that exact same thing :)
memcahed1: STAT uptime 9143 STAT get_hits 311736 STAT get_misses 11255
memcached2: STAT uptime 9144 STAT get_hits 49679 STAT get_misses 11116
Now that shows something not kosher. My guess is some app is not talking to both? What apps use memcached for what?
On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
On Wed, Jul 1, 2009 at 6:08 PM, Mike McGrathmmcgrath@redhat.com wrote:
On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
On Wed, Jul 1, 2009 at 3:33 PM, Mike McGrathmmcgrath@redhat.com wrote:
I'm not sure if we have any memcached experience on the list but I thought I'd ask. Can anyone explain this:
Notice how memcached1 has a much higher hit rate and memcached2 has a much lower hit rate?
The time for memcached1 is 5x less than memcached2 being up. That can have an effect on caching as right after a system comes up its rates are usually much higher and then over time fall off (iirc). I think it would take bringing both up at the same time to figure out if there is a true disparity over caching.
I thought that exact same thing :)
memcahed1: STAT uptime 9143 STAT get_hits 311736 STAT get_misses 11255
memcached2: STAT uptime 9144 STAT get_hits 49679 STAT get_misses 11116
Now that shows something not kosher. My guess is some app is not talking to both? What apps use memcached for what?
I was just talking to ricky about this a bit in IRC. So here's the scoop.
We've got mediawiki using memcached for a couple of things, including session data (which is weird and wrong but fast).
The recent addition to the group is Fedora community, specifically in it's implementation of beaker. I'm going to get ahold of luke tomorrow to verify and test some stuff but I think this line in the config:
beaker.cache.url = memcached1;memcached
I'm not sure how beaker reads that, but I suspect it might be only sending information to memcached1 and ignoring memcached2 altogether. If this theory holds it'd explain why memcached1 not only has a higher request rate but also a higher hit rate because I suspect fedoracommunity requests some of the same info over and over again compared to the wiki which probably has a broader data pool it pulls from.
-Mike
On 07/01/2009 05:39 PM, Mike McGrath wrote:
On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
On Wed, Jul 1, 2009 at 6:08 PM, Mike McGrathmmcgrath@redhat.com wrote:
On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
On Wed, Jul 1, 2009 at 3:33 PM, Mike McGrathmmcgrath@redhat.com wrote:
I'm not sure if we have any memcached experience on the list but I thought I'd ask. Can anyone explain this:
Notice how memcached1 has a much higher hit rate and memcached2 has a much lower hit rate?
The time for memcached1 is 5x less than memcached2 being up. That can have an effect on caching as right after a system comes up its rates are usually much higher and then over time fall off (iirc). I think it would take bringing both up at the same time to figure out if there is a true disparity over caching.
I thought that exact same thing :)
memcahed1: STAT uptime 9143 STAT get_hits 311736 STAT get_misses 11255
memcached2: STAT uptime 9144 STAT get_hits 49679 STAT get_misses 11116
Now that shows something not kosher. My guess is some app is not talking to both? What apps use memcached for what?
I was just talking to ricky about this a bit in IRC. So here's the scoop.
We've got mediawiki using memcached for a couple of things, including session data (which is weird and wrong but fast).
The recent addition to the group is Fedora community, specifically in it's implementation of beaker. I'm going to get ahold of luke tomorrow to verify and test some stuff but I think this line in the config:
beaker.cache.url = memcached1;memcached
I'm not sure how beaker reads that, but I suspect it might be only sending information to memcached1 and ignoring memcached2 altogether. If this theory holds it'd explain why memcached1 not only has a higher request rate but also a higher hit rate because I suspect fedoracommunity requests some of the same info over and over again compared to the wiki which probably has a broader data pool it pulls from.
easy test: reverse that:
beaker.cache.url = memcached2;memcached1
and see if the cache hit ratio reverses itself.
-Toshio
On Wed, Jul 01, 2009 at 05:50:12PM -0700, Toshio Kuratomi wrote:
On 07/01/2009 05:39 PM, Mike McGrath wrote:
On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
On Wed, Jul 1, 2009 at 6:08 PM, Mike McGrathmmcgrath@redhat.com wrote:
On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
On Wed, Jul 1, 2009 at 3:33 PM, Mike McGrathmmcgrath@redhat.com wrote:
I'm not sure if we have any memcached experience on the list but I thought I'd ask. Can anyone explain this:
Notice how memcached1 has a much higher hit rate and memcached2 has a much lower hit rate?
The time for memcached1 is 5x less than memcached2 being up. That can have an effect on caching as right after a system comes up its rates are usually much higher and then over time fall off (iirc). I think it would take bringing both up at the same time to figure out if there is a true disparity over caching.
I thought that exact same thing :)
memcahed1: STAT uptime 9143 STAT get_hits 311736 STAT get_misses 11255
memcached2: STAT uptime 9144 STAT get_hits 49679 STAT get_misses 11116
Now that shows something not kosher. My guess is some app is not talking to both? What apps use memcached for what?
I was just talking to ricky about this a bit in IRC. So here's the scoop.
We've got mediawiki using memcached for a couple of things, including session data (which is weird and wrong but fast).
The recent addition to the group is Fedora community, specifically in it's implementation of beaker. I'm going to get ahold of luke tomorrow to verify and test some stuff but I think this line in the config:
beaker.cache.url = memcached1;memcached
I'm not sure how beaker reads that, but I suspect it might be only sending information to memcached1 and ignoring memcached2 altogether. If this theory holds it'd explain why memcached1 not only has a higher request rate but also a higher hit rate because I suspect fedoracommunity requests some of the same info over and over again compared to the wiki which probably has a broader data pool it pulls from.
easy test: reverse that:
beaker.cache.url = memcached2;memcached1
and see if the cache hit ratio reverses itself.
-Toshio
So, I've been poking at this a little bit lately, and I'm thinking there are some problems with the way Fedora Community is utilizing it's Beaker cache & memcached.
The fact that something is wrong with the caching setup becomes obvious when playing with the Bugzilla grid *should* cache the first 5 pages, and be very snappy, as it is in my local instance. However, that is not the case, which makes me think it's not hitting our memcached servers.
I wrote up a little test script on app1::
from beaker.cache import CacheManager
memcached1 = CacheManager(type='ext:memcached', url='memcached1', lock_dir='.') memcached2 = CacheManager(type='ext:memcached', url='memcached2', lock_dir='.')
def createfunc(*args): print "createfunc(%s)" % (args,)
def get_value(cache, value): cache_1 = memcached1.get_cache(cache) cache_2 = memcached2.get_cache(cache) result1 = cache_1.get_value(key=value, createfunc=createfunc) result2 = cache_2.get_value(key=value, createfunc=createfunc) print "memcached1[%s][%s] = %s" % (cache, value, result1) print "memcached2[%s][%s] = %s" % (cache, value, result2) return result1, result2
get_value('fedoracommunity_alerts_global', 'today') get_value('bodhi', 'dashboard_None')
Which produces::
memcached1[fedoracommunity_alerts_global][today] = None memcached2[fedoracommunity_alerts_global][today] = None memcached1[bodhi][dashboard_None] = None memcached2[bodhi][dashboard_None] = None
I also tried using netcat to query for these by hand, to no avail.
So, it looks like we need to look a bit deeper into what is going on here. Either I'm Doing It Wrong with Beaker, or we're hitting a bug somewhere.
luke
On Wed, Sep 09, 2009 at 04:33:19PM -0400, Luke Macken wrote:
On Wed, Jul 01, 2009 at 05:50:12PM -0700, Toshio Kuratomi wrote:
On 07/01/2009 05:39 PM, Mike McGrath wrote:
On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
On Wed, Jul 1, 2009 at 6:08 PM, Mike McGrathmmcgrath@redhat.com wrote:
On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
On Wed, Jul 1, 2009 at 3:33 PM, Mike McGrathmmcgrath@redhat.com wrote: > I'm not sure if we have any memcached experience on the list but I thought > I'd ask. Can anyone explain this: > > http://pastebin.ca/1481219 > > Notice how memcached1 has a much higher hit rate and memcached2 has a much > lower hit rate? >
The time for memcached1 is 5x less than memcached2 being up. That can have an effect on caching as right after a system comes up its rates are usually much higher and then over time fall off (iirc). I think it would take bringing both up at the same time to figure out if there is a true disparity over caching.
I thought that exact same thing :)
memcahed1: STAT uptime 9143 STAT get_hits 311736 STAT get_misses 11255
memcached2: STAT uptime 9144 STAT get_hits 49679 STAT get_misses 11116
Now that shows something not kosher. My guess is some app is not talking to both? What apps use memcached for what?
I was just talking to ricky about this a bit in IRC. So here's the scoop.
We've got mediawiki using memcached for a couple of things, including session data (which is weird and wrong but fast).
The recent addition to the group is Fedora community, specifically in it's implementation of beaker. I'm going to get ahold of luke tomorrow to verify and test some stuff but I think this line in the config:
beaker.cache.url = memcached1;memcached
I'm not sure how beaker reads that, but I suspect it might be only sending information to memcached1 and ignoring memcached2 altogether. If this theory holds it'd explain why memcached1 not only has a higher request rate but also a higher hit rate because I suspect fedoracommunity requests some of the same info over and over again compared to the wiki which probably has a broader data pool it pulls from.
easy test: reverse that:
beaker.cache.url = memcached2;memcached1
and see if the cache hit ratio reverses itself.
-Toshio
So, I've been poking at this a little bit lately, and I'm thinking there are some problems with the way Fedora Community is utilizing it's Beaker cache & memcached.
The fact that something is wrong with the caching setup becomes obvious when playing with the Bugzilla grid *should* cache the first 5 pages, and be very snappy, as it is in my local instance. However, that is not the case, which makes me think it's not hitting our memcached servers.
I wrote up a little test script on app1::
from beaker.cache import CacheManager memcached1 = CacheManager(type='ext:memcached', url='memcached1', lock_dir='.') memcached2 = CacheManager(type='ext:memcached', url='memcached2', lock_dir='.') def createfunc(*args): print "createfunc(%s)" % (args,) def get_value(cache, value): cache_1 = memcached1.get_cache(cache) cache_2 = memcached2.get_cache(cache) result1 = cache_1.get_value(key=value, createfunc=createfunc) result2 = cache_2.get_value(key=value, createfunc=createfunc) print "memcached1[%s][%s] = %s" % (cache, value, result1) print "memcached2[%s][%s] = %s" % (cache, value, result2) return result1, result2 get_value('fedoracommunity_alerts_global', 'today') get_value('bodhi', 'dashboard_None')Which produces::
memcached1[fedoracommunity_alerts_global][today] = None memcached2[fedoracommunity_alerts_global][today] = None memcached1[bodhi][dashboard_None] = None memcached2[bodhi][dashboard_None] = NoneI also tried using netcat to query for these by hand, to no avail.
So, it looks like we need to look a bit deeper into what is going on here. Either I'm Doing It Wrong with Beaker, or we're hitting a bug somewhere.
I did a little more testing with this today.
On app1 I was unable to get fedoracommunity to put any values in memcached, and I was not seeing any errors either. I tried this with both the mod_wsgi deployment, and running paster locally. I then ran `curl http://localhost:8080/apps/fedoracommunity.alerts/%60 to trigger the cache, and then ran the script I mentioned earlier to see if I could retrieve those values.
I was able to reproduce this failure on a local RHEL5 VM. However, after upgrading to python-beaker-1.4 *and* restarting memcached, my script *worked*. (the app just silently returned None for all requests if I didn't restart memcached after upgrading Beaker)
Looking at the ChangeLog for Beaker 1.4, I'm thinking this could have done the trick:
* Fixed bug with CacheMiddleware overwriting configuration with default arguments despite prior setting.
It'll be tough to test this new package in staging, as earlier today I couldn't get any of the cached apps to load due to koji/fas problems in staging. When we deploy this production, we'll need to make sure to restart our memcache daemons.
luke
infrastructure@lists.fedoraproject.org
-
Luke Macken -
Mike McGrath -
Stephen John Smoogen -
Toshio Kuratomi