Are you getting memcached interruptions in your application logs? Does memcached say its been running but your app says something else?
###Diagnosing the Problem
The first step is to check memcached stats with Telnet. If you’re running on Engine Yard (we use Gentoo) you can run
emerge -g netkit-telnetd on your memcached server. You can then
telnet localhost 11211 and then type
stats. You should see something like this:
stats STAT pid 27631 STAT uptime 104 STAT time 1401932723 STAT version 1.4.5 STAT pointer_size 64 STAT rusage_user 0.004999 STAT rusage_system 0.010998 STAT curr_connections 10 [...] STAT listen_disabled_num 0 STAT threads 4 STAT conn_yields 4354 STAT bytes 0 STAT curr_items 0 STAT total_items 0 STAT evictions 23420009 STAT reclaimed 0 END
The two stats that we will be focusing on are
###Do you have conn_yields in your stats?
Connection yields are the amount of times the memcached server has throttled a specific client. If you have
conn_yields you are likely using large multi-get requests. It is possible to lower the sizes of your multi-get operations or increase the -R value, or both.
The default value of -R is 50 connections from a client before throttling. Note that not all memcached clients respond the same way to the connection yield throttling. Some clients reconnect without an issue, while others may report that the memcached server is unavailable. You will likely want to increase these numbers until you no longer see the
conn_yield statistic being throttled.
You can verify you have enough space on the server and aren’t over allocating memory. Running
free -m will make sure you have memory to spare.
Likely your slabs aren’t in ideal allocations for the objects you’re storing. You can verify the status of your slab memory allocation by running
telnet localhost 11211 and then
stats slabs. This command allows you to view how your slabs are allocated. For more on slab statistics, please check out this article.
The breakdown of those statistics are per slab and as a whole at the bottom. If there are large amounts of active slabs, but very few active items in those slab classes, you will likely want to alter the default growth factor.
If you have large amounts of active slabs all with very high active usage and high eviction count you may wish to increase the amount of slabs you have available.
The default growth factor for memcached is 1.25. There isn’t a magical best number for every application. If you need more available slabs you would lower the growth factor (-f) number from 1.25. If you needed less slabs you would raise the number.
Memcached is a great option for caching some objects. But out of the box, it isn’t going to be fine tuned for every application’s data set. You’ll want to alter your growth factor (-f) and -R settings to reduce conn_yields and evictions. These will better tune memcached for your application’s needs.
For more on memcached memory usage, check out this article. Management of your memcached memory is the key to having a better, more smooth running environment on Engine Yard.
P.S. Have you fine-tuned your memcached setup in the past? What tips or tricks do you have? Any gotchas to share?