Security and Kubernetes

As someone who specialises in developing in Kubernetes, I’ve seen plenty of examples where security has been overlooked and ends up being an issue. That’s why, over two posts, I’ll be exploring Kubernetes cluster security.

First up, I want to talk about the basic isolation mechanisms you can use to protect your clusters. The traditional processes of threat modelling (like STRIDE) no longer apply when it comes to the holistic approach of Kubernetes cluster security.

In a multi-tenancy environment, security is all about isolating groups of tenants with different level of trusts. Kubernetes introduces a new abstraction level, and you may be tempted to set the security boundaries just around Kubernetes artifacts. This is definitely the starting point and I will explore that in a little more detail later in this post.

There is also a dark side of high-level abstraction. The actual elements of the cluster (with their vulnerabilities) are under the horizon but can still be explored when a high-level isolation is broken. In fact, they’re also the main attack surfaces, because compromising them may lead to gaining full supervisor privileges over the cluster. I’ll explore this problem in more detail in my second post.

Before I go into the details, it’s important to understand that a single security boundary is not enough. In the Kubernetes world, one vulnerability can be “game over” for the whole cluster. So what are the basic isolation mechanisms? The most obvious is the container implemented on top of Linux namespaces and cgroups. Namespaces and cgroups provide some level of isolation, but without conscious configuration it’s very weak.

Pod and container security context

The next layer of isolation is provided by Kubernetes pods, (which are the focus of this article). A pod encapsulates a service, as an element of microservices architecture, so it can be orchestrated by the cluster and wraps one or more explicitly-defined containers. The encapsulation also introduces tools for isolation hardening. How can we make use of these mechanisms?

Security contexts set various security rules associated with a pod (all containers in a pod) or a particular container. Both can be configured in the pod’s template in the general spec section (PodSecurityContext) or container spec section (SecurityContext); however each has different options.

By default, the main container in a pod runs as root. You can use runAsUser and fsGroup (on pod level) to control the main process owner. It’s good practice to also set the allowPrivilegeEscalation flag (on container level) to false (default for non-privileged pods or pods without SYS_ADMIN capability). This will disable the possibility of a process gaining more privileges than its parent. A container’s readOnlyRootFilesystem flag makes a container able to write only to its mounted volume.

Even if you need a container process to run as root, you can use the capabilities drop feature for limiting root’s privileges. Linux Capabilities is a mechanism that slices root’s privileges into unit permissions that can be attributed to or dropped from a process. You can create a list of forbidden actions and add it directly to your container spec: this would reduce the damage if the container was compromised. If your Kubernetes node supports AppArmor or SELinux modules, these can also be configured using security context (SELinux options can be added directly in pod or container spec sections and AppArmor via annotations). The configuration of both modules is rather complex and out of the scope of this article.

Security context also supports Seccomp. Seccomp is a Linux kernel feature for filtering system calls, and needs a profile which is a whitelist of allowed system calls. The default Docker profile is available and can be used as a base for modification. The pod Seccomp setup is controlled by:

annotations:seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default

Pod security policies and other security-related admission controllers

Security context works very well with PodSecurityPolicies. This is an admission controller that helps enforce security context on a cluster level. Admission controllers work on the Kubernetes API server as the third phase of request processing, modifying the resource just before it’s persisted. If the security context of a pod that is being created or modified doesn’t fulfil security policy, its creation or modification is rejected. Aside from PodSecurityPolicies, there are a few more admission controllers that help to enforce security rules on a cluster level, such as:

• DenyEscalatingExec. This admission controller will deny the exec and attach commands to pods that run with escalated privileges that allow host access.
• NodeRestriction. This admission controller limits the Node and Pod objects a kubelet can modify.
• SecurityContextDeny. This admission controller will deny any pod that attempts to set certain escalating SecurityContext fields. This should be enabled if a cluster doesn’t utilise pod security policies to restrict the set of values a security context can take.

(The above descriptions are taken straight from Kubernetes’ documentation)

Network security policies

As mentioned before, pods provide a common network stack for their containers, and this allows a segmentation mechanism on the networking level. By default, all pods are able to communicate with each other, but the goal of Network Policies is to restrict this communication to groups of pods determined by a selector and a namespace. The constraints can be applied to inbound and outbound traffic. A pod with assigned Network Policy will reject any traffic that is not explicitly allowed by the policy.

The policies are defined as whitelists, which makes some configurations challenging to achieve ( “I need to connect to all but this pod…”). The other drawback is that some combinations of Kubernetes versions and network plugins have bugs and undocumented behaviour. I recommend always verifying if the policy really works (also after any cluster update). If the Kubernetes Network Policy doesn’t work for you, but you feel like network segregation would solve your problem, you may dig deeper and use security policies of the network plugin. For example, security configuration of the Calico plugin is much more flexible and helps to solve problems where Kubernetes Network Policies fails.

In this article, I described the basic isolation mechanisms of Kubernetes. In the next part, I take a closer look at underlying Kubernetes architectural elements and the ways of protecting them.

Useful resources

• Pod Security Context
• Container Security Context
• List of all root capabilities for Docker containers
• Linux Capabilities list
• PodSecurityPolicy admission controller documentation
• Network Policies documentation