Memcached Security aka Don't Attack GitHub 

GitHub recently experienced the largest attack we've seen to date. At the peak, they received 1.35 Tbps via 126.9 million packets per second. We don't know who launched the attack but we know how they did it. The attackers used an amplification attack using memcached servers that were exposed to the internet.

An amplification attack allows the attacker to send only a small request but still generate a large attack to the target. In the case of a memcached attack, a byte sent by the attacker can result up to 51KB sent to the target. The attacker doesn't need to control a botnet or have a big capacity, the vulnerable memcached servers will do the attack for him.

We'll not talk about how to respond to an amplification attack on this post. Instead, I will outline steps to secure your memcached server so that you don't participate in this type of attacks. Teach your servers to be good citizens on the internet.

Memcached is used to speed up your application. Response time can go down if your Rails application can get data from memcached instead of running a slow database query. A typical setup is using multiple servers running Rails which connect to one or more servers running memcached. With this in mind, here are the steps to secure your servers.

Use a firewall

Memcached needs to be accessible from your other servers but there's no reason to expose it to the internet. At Engine Yard, we use an AWS security group to prevent access to all ports by default (except for a few ones). Memcached uses port 11211 which is only exposed to the servers belonging to the same Engine Yard environment. In short, only your other production servers have access to your production memcached servers.

This alone would prevent your server from being used in an attack. A better reason to do this is to protect your data. Memcached out of the box doesn't use authentication so anyone who can connect to your server will be able to read your data.

While this is enough to secure your memcached server, there are a few more best practices you can follow.

Listen on a private interface

If you're running one server for your Rails application and memcached, you should listen on 127.0.0.1. For availability reasons, you shouldn't have 1 server in production anyway. For staging and test environments, follow this rule.

For production setups where you have multiple Rails servers that need to connect to memcached, use the private IP of the server. This is something like 192.168.0.1, 172.16.0.1, or 10.0.0.1.

When you start memcached, use --listen 127.0.0.1 or --listen 192.168.0.1.

Disable UDP

The amplification attack on GitHub used UDP. I like many people, was surprised that memcached uses UDP. We run a lot of memcached servers for Engine Yard customers and we don't set it to use UDP. But it is enabled by default. If you leave the settings as is, you are using UDP.

To disable UDP, use -U 0 when starting memcached.

Summary

That's it. Follow these simple rules and your data will be safe and your memcached servers won't attack anyone. Remember, if you don't need to expose your server to the internet, don't. If you do, make sure you use authentication. Otherwise, attackers will find you and can use you for attacks.

Resources

GitHub DDOS Incident Report
Cloudflare blog on memcached amplification attack