Nginx and Memcached - connections timeout under high load

Just a little post to summurize my experience after resolving a performance issue between nginx and memcached.

After successfully connect Nginx to fetch some binary data from Memcached, we experienced a problem in production under moderate load  (10K of users online). The logs of nginx were plainly of the following error messages:

In the same time the CPU load was extremely high - about 100% for each core. On the top of the most busy processes were the Nginx workers.

So Nginx was unable to open new connections to memcached.
Before I found the reason and the fix for this problem I had the following nginx configuration.

The keyword to solve this problem is - persistent connection. I figured out that Nginx tried to open a new TCP connection for every new request. It's a huge waste, as opening a new TCP connection can take a lot of time and ressources. 

So to fix the problem I declared an upstream with one single memcached server and set the directive keepalive to 1024:

More information about this parameter you can found in the official Nginx documentation: here

After reloading nginx config, the connection timeout errors gone. And the CPU load was quite small and generated in this case only by memcached.

So now our production memcached instance serves up to 6500 images per second. Quite impressive and reliable (upstart is more then 3 months and I'm sure that it will continue to rock )