Nginx and Memcached - connections timeout under high load

Just a little post to summurize my experience after resolving a performance issue between nginx and memcached.

After successfully connect Nginx to fetch some binary data from Memcached, we experienced a problem in production under moderate load  (10K of users online). The logs of nginx were plainly of the following error messages:

In the same time the CPU load was extremely high - about 100% for each core. On the top of the most busy processes were the Nginx workers.

So Nginx was unable to open new connections to memcached.
Before I found the reason and the fix for this problem I had the following nginx configuration.

The keyword to solve this problem is - persistent connection. I figured out that Nginx tried to open a new TCP connection for every new request. It's a huge waste, as opening a new TCP connection can take a lot of time and ressources. 

So to fix the problem I declared an upstream with one single memcached server and set the directive keepalive to 1024:

More information about this parameter you can found in the official Nginx documentation: here

After reloading nginx config, the connection timeout errors gone. And the CPU load was quite small and generated in this case only by memcached.

So now our production memcached instance serves up to 6500 images per second. Quite impressive and reliable (upstart is more then 3 months and I'm sure that it will continue to rock )

MacBook Pro WiFi hardware config problems

Recently, I have experienced the problem with my Mac Book Pro mid 2009.
The WiFi module couldn't detect any WiFi network.

I have tried all naive solutions, like recreating WiFi config, restart the computer. Nothing helps.


Fortunately, I have found this article - http://www.cnet.com/news/tackling-the-no-wi-fi-hardware-installed-error-in-os-x/

And the second solution - reset the PRAM - helped me. After reseting the hardware configuration, my Mac saw all WiFi networks.

I hope this little note will be useful for somebody.

java-ffmpeg-pipe-memcache

Just a cross link https://github.com/yev/java-ffmpeg-pipe for github repo with simple code example which demonstrates how to push snapshot grabbed by ffmpeg from a video stream to a memchache server without intermediate File System calls.


The schema describing the interactions between all components:

Actually, the RTMP stream can be replaced with a whatever source which can be read by ffmpeg.

Best java client for swift (OpenStack storage) - JOSS

Recently, I had a task to choose a java client for interacting with OpenStack Storage service aka Swift.

I recommend you JOSS

Actually, I found 3 open source projects on GitHub:

1. https://github.com/dkocher/java-openstack-swift
2. https://github.com/woorea/openstack-java-sdk
3. https://github.com/javaswift/joss

After testing and working for a bit with each library I ended up with the following conclusion - the best one is JOSS - http://joss.javaswift.org

So, now the list of advantages:

1. openstack-java-sdk is a huge library for almost all OpenStack services. I needed only swift driver so I wasn't interested to add useless dependencies. Always use KISS principle.
2. java-openstack-swift from dkocher - I didn't manage to make a connection to my swift proxy. Maybe this library is very good and written by smart people, but the documentation is poor. If a developer should dig into the code source to figure out how to make the basic simple step, this will increase the learning curve. One more little disadventage is absence of the project in maven central repo. 
3. And finally the winner - JOSS - https://github.com/javaswift/joss by Robert Bor. 

Why JOSS:

1. Very simple API. 
2. The principle of modern fluid API building is used. The API is readable and comprehensible.
3. Good documentation for each common task (authentication, uploading, downloading, etc)
4. Available at standard maven repo.
5. Excellent support organised by the project author - Robert Bor. He helped me within few hours to find an error in my configuration and pleased me with his reaction and quality of responses.