Memcached Security aka Don't Attack GitHub 

  

GitHub recently experienced the largest attack we've seen to date. At the peak, they received 1.35 Tbps via 126.9 million packets per second. We don't know who launched the attack but we know how they did it. The attackers used an amplification attack using memcached servers that were exposed to the internet.

An amplification attack allows the attacker to send only a small request but still generate a large attack to the target. In the case of a memcached attack, a byte sent by the attacker can result up to 51KB sent to the target. The attacker doesn't need to control a botnet or have a big capacity, the vulnerable memcached servers will do the attack for him.

We'll not talk about how to respond to an amplification attack on this post. Instead, I will outline steps to secure your memcached server so that you don't participate in this type of attacks. Teach your servers to be good citizens on the internet.

Memcached is used to speed up your application. Response time can go down if your Rails application can get data from memcached instead of running a slow database query. A typical setup is using multiple servers running Rails which connect to one or more servers running memcached. With this in mind, here are the steps to secure your servers.

Use a firewall

Memcached needs to be accessible from your other servers but there's no reason to expose it to the internet. At Engine Yard, we use an AWS security group to prevent access to all ports by default (except for a few ones). Memcached uses port 11211 which is only exposed to the servers belonging to the same Engine Yard environment. In short, only your other production servers have access to your production memcached servers.

This alone would prevent your server from being used in an attack. A better reason to do this is to protect your data. Memcached out of the box doesn't use authentication so anyone who can connect to your server will be able to read your data.

While this is enough to secure your memcached server, there are a few more best practices you can follow.

Listen on a private interface

If you're running one server for your Rails application and memcached, you should listen on 127.0.0.1. For availability reasons, you shouldn't have 1 server in production anyway. For staging and test environments, follow this rule.

For production setups where you have multiple Rails servers that need to connect to memcached, use the private IP of the server. This is something like 192.168.0.1, 172.16.0.1, or 10.0.0.1.

When you start memcached, use --listen 127.0.0.1 or --listen 192.168.0.1.

Disable UDP

The amplification attack on GitHub used UDP. I like many people, was surprised that memcached uses UDP. We run a lot of memcached servers for Engine Yard customers and we don't set it to use UDP. But it is enabled by default. If you leave the settings as is, you are using UDP.

To disable UDP, use -U 0 when starting memcached.

Summary

That's it. Follow these simple rules and your data will be safe and your memcached servers won't attack anyone. Remember, if you don't need to expose your server to the internet, don't. If you do, make sure you use authentication. Otherwise, attackers will find you and can use you for attacks.

Resources

GitHub DDOS Incident Report
Cloudflare blog on memcached amplification attack

Free Ebook: PaaS Is Dead

Platform as a Service (PaaS) is experiencing a digital transformation, and despite what some may argue, it’s far from dead. Learn why PaaS continues to prove it has a promising future for DevOps.

Download Ebook
PaaS Is Dead

Related posts

RubyConf Round Up

November 27, 2018

The City: Los Angeles, California

Read More

It's Almost Time For Ruby Conf 2018!

November 6, 2018

It’s autumn and November is right around the corner! We all know what this means…

Read More

Jekyll and Engine Yard: A Match Made in The Clouds

October 29, 2018

Blogging is without a doubt the most important way to get your ideas out to the world, whether

Read More

Christopher Rigor

 
Christopher Rigor is a Senior Technical Evangelist at Engine Yard. He’s a long time Rails user, system administrator, and recently became a contributor of RailsInstaller. Previously, he was the DevOps Support Manager for Asia-Pacific at Engine Yard.
Find me on:

Comments

Subscribe Here!