Security practices in Kubernetes

Kubernetes offers great solutions for container orchestration. It facilitates automated deployment, scaling, and management with ease, along with which there are increased security concerns. According to the Kubernetes adoption, security, and market trends report by RedHat in 2021, 94% of the respondents experienced at least 1 security incident in their Kubernetes environment in the last 12 months. In this blog post, we will be talking about the security best practices for Kubernetes which can be implemented at each phase of the SDLC.

BUILD PHASE

Adopting the “shift left” approach when it comes to the implementation of security practices ensures optimization of the Kubernetes environment. This usually begins with securing your container images.

1. Usage of Minimal Base Images: Opt for minimal base images like distroless images. Refrain from using images with OS package managers or shells as they contain unknown vulnerabilities. Even if you do so, make sure that the package manager is removed in the later steps.
2. Avoid Unnecessary Components and Update Your Images: Remove debugging tools from the container images in the production stage as they might be beneficial to attackers. Always update your images and all the third-party tools to avoid vulnerabilities.
3. Scan the Images for Vulnerabilities: Use an image scanner to identify image layer-wise vulnerabilities. The image scanner must be able to segregate the vulnerabilities as fixable and non-fixable. These vulnerabilities must then be labeled accordingly. Some vulnerabilities are non-critical and don’t need immediate fixes. They can be added to an allow list or the scanner output can be filtered. This ensures that it doesn’t interrupt the development team’s workflow.
4. Policy Checks and Remediation Workflows: In case of a security issue unearthed in a container image or a running deployment, carry out a policy check immediately. This helps us to detect and fix faulty images.

DEPLOY PHASE

To implement security policies in the deployment phase, we need to gain visibility into details like:

1. Information about the images used, the pods which are deployed, and their locations(clusters, namespaces, and nodes).
2. The privilege of the pods and the other deployments it communicates with.
3. Accesses granted to the secrets, volumes, and other infrastructure components.

With the help of these details, we can implement the following security practices accordingly.

• Leveraging Namespaces To Perform Isolation of Sensitive Workloads: Namespaces provide a reference for network policies, access control restrictions, and other security controls. Segregating workloads into namespaces helps in containing attacks and limits exposure to mistakes by authorized users.
• Kubernetes Network Policies To Manage Traffic Between Pods and Clusters: In Kubernetes, every pod is in contact with the other pods, which results in traffic. Network segmentation policies can help in preventing lateral movement across containers in case of an attack. There are two types of traffic namely:

1. Ingress Traffic: incoming traffic to the pod.
2. Egress Traffic: outgoing traffic from the pod.

There are different network policies for each type that can be implemented accordingly.

• Assessing Container Privileges and Image Provenance: The container should always be provided with minimum privileges and capabilities that allow it to perform the intended function. Pod security policies can be used to control security-related attributes of pods and container privilege levels. The policies specify the following points:

1. Refrain from running application processes as root and use a read-only root file system.
2. Do not allow privilege escalation.
3. Drop unnecessary Linux capabilities. SELinux can be used to gain more fine-grained process controls.
4. Do not use host network and process spaces.
5. If the container doesn’t need access to the Kubernetes API, don’t mount the service account credentials in it.
6. Always stick to images from known registries and allow-lists. Enforce image scanning even in the deploy phase.

Use Role-Based Access Control(RBAC)

The teams in the organization require security policies that solely depend upon their function and the respective time frames. No team would have the same security privileges and hence it is necessary to implement Role-Based Access Control. Here is an insight into the security policy requirements for each team:

1. Security/Infosec Team: The security policies should comply with the organizational security requirements such as multi-tenancy, external communication compliance, and threat defense.
2. Platform: hardening and securing the control and management planes must be performed.
3. Developers/DevOps Team: application security such as implementing micro-segmentation of microservices communication must be carried out.

RUNTIME PHASE

This is a dynamic phase that introduces new security challenges in the containerized environment. The security practices implemented in the previous phases pay off in the runtime phase as it reduces the liability towards security issues at runtime. It also reduces the effort needed to curb them.

As a first step towards run-time security implementation, we should monitor activities such as process activity, communication amongst containerized services, external clients, and servers. The next step would be to implement the following security practices in the run-time environment:

1. Leverage Contextual Information and Built-In Controls: The build and deploy time information in Kubernetes can be used to evaluate the observed versus expected activity during runtime. The runtime activity in the pods of the same deployment must be compared and evaluated. Since replicas behave nearly identical must be compared and evaluated. Since replicas behave nearly identical, we can shortlist the contrast replicas for further investigation. This helps in the detection of malicious activity in the Kubernetes environment. Leverage the built-in controls such as the configuration of security context for pods, read-only root file systems, etc. They can curb the entire class of attacks which depend on privileged accesses. In case of a security breach, use the native controls to scale the suspicious pods to zero and then restart instances of the breached applications.
2. Integrate External Systems with the Kubernetes Security Tool: Tools like email, PagerDuty, Slack, Google Cloud Security Command Center, SIEMs(Security Information and Event Management), and labeled deployment tools can be used to alert the concerned team when a potential threat is detected. Vulnerability scanning should be carried out on the running deployments.

• Monitoring Network Traffic: Monitor active network trafficking to understand the interaction patterns and as a result, identify unexpected communication. This helps us to tighten the allowed network policies so that it removes superfluous connections and decreases attack surface.
• Use Allow-Lists: Generate lists by identifying processes in the normal course of application behavior, which can later be used as allow-lists. These allow-lists can be used to compare against future application processes to detect malicious behavior.

WRAPPING-UP

Apart from the build, deploy and runtime phases, there are a few overall security practices to be followed. This helps to protect the entire Kubernetes environment including the cluster infrastructure. These include updating the Kubernetes to its latest version, configuring the Kubernetes API server, securing the kubelets, using secure etcd , and much more. The security practices must be tailored according to the organization's needs. One must always be prepared for recovering from an attack, even if top-notch security policies are implemented in the system.