I have been in IT for over 20 years, but have never had any hands on experience with containers. Conceptually, I understand what they are and how they work, but since I’ve never had to implement them, I wasn’t sure how it worked. In addition, I had no idea how to actually secure them. Again, the concept sounds great, but the old adage of “as security increases, usability decreases” sat in my head, and with how easy everything container-related sounded, it also seemed like security was going to be an afterthought.
Turns out I was right.
To get up to speed, I signed up for the Black Hat 2020 session entitled, “From Zero to Hero: Pentesting and Securing Docker Swarm and Kubernetes Environments." The course, taught by Sheila A. Berta and Sol Ozzan, literally started with a description of how Docker containers worked and went all the way through a Kubernetes deployment. It was entirely hands on - students were required to install Docker and microk8s on their own machine before the class - and was a great way to see how the tools communicate, where the weak points are, and most importantly, how to try locking it down.
Unfortunately, while the course says you’ll be a “hero” at the end of two days, I feel like I am just starting down the road and have a lot more to learn. That said, I wasn’t going into it expecting to be an expert after only 16 hours of training.
Before getting into my high level observations, it's important to explain what a container is. In the development world, it's common to write code on your own machine, have it work perfectly, but then when you try running it on a server somewhere, it just doesn't work. Containers try to overcome that issue by providing self-contained machines that you can easily port from one server to another and you know it will always work. Like the name implies, it contains all of the code, libraries, and software needed to run. Kubernetes, on the other hand, is an orchestration platform for containers. Basically, it lets you manage hundreds or thousands of different containers in a seamless manner.
Below are some of my takeaways from both a Red Team and Blue team perspective.
Red Team
Most containers run things as root: This means if you compromise a host, you’ll probably have full control of the container. That makes the following steps a lot easier.
The docker.sock Bind Mount is dangerous: If you’re root in a container and install Docker in a container that has a Docker socket (var/run/docker.sock), you’ll be able to inspect the entire cluster, including getting a shell inside of any other container. At that point, none of the network segmentation or other controls will prevent access.
Environment variables often contain secrets: In many cases, people send passwords into the container using simple environment variables. So if you have access to the account, you can see the environment variables and further expand your powers.
The Docker API can give a lot of information: The Docker API, by default, is unauthenticated and can expose a lot of information. Using Shodan, you can easily find a list of open ports and from there, can get details about the cluster and pivot to taking it over completely. Trend Micro has an interesting blog post about the topic.
Blue Team
Don’t run things in the container as root: While it’s easy to run things as root, you shouldn’t do it. Instead, run applications with lower permissions by mapping the UID by either using the --user command-line option or by using the USER directive in the dockerfile.
Don’t let containers install things: Almost every attack started with us installing something. From nmap to ifconfig to Docker itself (within the container), installation of something in the container was a common thread. This is why you should always block any unused ports at your boundary. This also helps prevent Command and Control communication if someone does infect your machine. In addition to preventing the installation of tools, you should also make sure the container itself has only the minimum number of applications installed to still do its job.
Protect the Docker Socket: The Docker socket is how communication is handled between container and cluster, so it must be protected. While I won’t go into detail here, the Docker webpage has what appears to be a great article detailing how to lock it down.
Use Docker Secrets instead of environment variables: Docker Secrets has been around since 2017. While not fool-proof, it’s still better than using environment variables to pass secrets into the container.
If this has peaked your curiosity on containers, you can install Docker and/or Microk8s (a small version of Kubernetes) fairly easily. Here are the installation instructions for installing Docker on Linux or MacOS, and here are the installation instructions for installing Microk8s on Windows, Linux, or MacOS.
Once the installation is complete, the Docker has a nice "Quick Start" guide on their website, and MicroK8s has one for their product as well.
If you would be interested in reading more technical articles like this in the future, let us know so that we can continue bringing you quality content at Neowin.
10 Comments - Add comment