Kubernetes Security – Introduction to Attack Vectors

Kubernetes helps the enterprises to automate their application deployment for the business benefits. Now-a-days Kubernetes security can be considered as a critical component for all deployments because the new deployments might be vulnerable to attacks and exploits from hackers or insiders. Different kind of attacks will be launched against new container based virtualized environments. (in both private and public clouds.)

Kubernetes is an open source tool that automates the deployment, and for the management of containerized applications. All major container management and cloud platforms such as Red Hat OpenShift, Docker EE, Rancher, IBM Cloud, AWS EKS, Azure, SUSE CaaS, and Google Cloud are supported by Kubernetes.

Here are some of the key things to know about Kubernetes:

• Master Node: The server which manages the Kubernetes worker node cluster and the deployment of pods on nodes.
• Worker Node: These slave or minion servers run the application containers and other components of kubernetes such as agents and proxies.
• Pod: The most basic deployable unit within a cluster. A pod has its own IP address and can contain one or more containers.
• Service: A service functions as a proxy to its underlying pods and requests can be load balanced across replicated pods.
• System Components: Key components for Kubernetes cluster managemnt includes the API Server, Kubelet, and etcd. Any of these components can be a targets for attacks.

How Kubernetes Works

The basic idea of Kubernetes networking is that every pod has its own routable IP address and its network plug-in takes care of routing all requests internally between hosts to its corresponding pod. External access to Kubernetes pods can be provided through a service, load balancer, or ingress controller, which Kubernetes routes to the appropriate pod.

Using network overlay and load balancing, Pods communicate with each other. DNAT takes place to get the connections to the appropriate pod. Packets may be encapsulated with appropriate headers to get them to the appropriate destination. it is very difficult to monitor network traffic with the overlay networking which being dynamically handled by Kubernetes.

The dynamic nature of containers creates some Kubernetes security challenges.They are:

• Containers can be dynamically deployed across hosts or even clouds, increasing rapidly the east-west, or internal, traffic that must be monitored for attacks.
• Different attack surface and vulnerabilities can be exploited for each container.
• Old models and tools for security will not be able to keep up in a constantly changing container environment.

Kubernetes Vulnerabilities and Attack Vectors

Attacks on Kubernetes containers running in pods can originate externally through the network or internally by insiders. Some examples are:

1. An application misconfiguration or vulnerability leads the attacker to get access into a container to start exploit the network weaknesses, process controls, or file system.

2. Attempt to connect with other running pods on the same or other hosts to probe or launch an attack using the compromised container.

3. Data Theft can be done using a combination of reverse shell in a pod connecting to a command/control server and network tunneling to hide confidential data.

‘Kill Chain’ Attacks

To achieve the attacker’s goal, the attacker involve a series of malicious activities which results in most damaging attacks . These events can occur dramatically, within a span of seconds, or can be spread out over days, weeks or even months. Different resources needs to be used in multiple layers of security monitoring to Detect kill chain events. The most critical vectors to monitor includes:

• Network inspection: Attackers usually enter through a network connection and expand the attack via the network. The network offers initially the opportunity to an attack, subsequent opportunities to detect next movements, and end with the opportunity to catch data stealing activity.
• Container monitoring: The exploit can be detected by monitoring the process, syscall, and file system activity in each container. This is to determine whether a suspicious process have started or attempts are being made to escalate privileges and break out of the container.
• Host security: Endpoint security can be useful to detect exploits against the kernel or system resources.

Attacks on the Kubernetes Infrastructure Itself

Attackers can attempt to compromise Kubernetes resources like API Server or Kubelets to disable or disrupt applications or to gain access to the underlying resources used to run containers. There are some published privilege escalation mechanisms in Kubernetes, via the Kubelet, access to etcd or service tokens. This can enable an attacker to gain cluster admin privilege rights from a compromised container.

Pre-Deployment Kubernetes Security Steps

It is recommended to lock down the Kubernetes worker nodes, before deploying any application containers. Some effective steps to lock down the hosts are:

• Use namespaces
• Restrict Linux capabilities
• Enable SELinux
• Utilize Seccomp
• Configure Cgroups
• Use R/O Mounts
• Use a minimal Host OS
• Update system patches
• Run CIS Benchmark security tests

Real-Time & Run-Time Kubernetes Security

The three critical security vectors for protecting running containers in production are network filtering, container inspection, and host security.

Inspect and Secure the Network

Container Firewall feature provides a possibility to control the nodes availability both from inside and outside of the Platform. It analyzes various parameters like incoming request source, protocol, target node port and so on to flexibly manage access to the containers through necessary connection rules. Different approaches to secure a container network with a firewall are:

• The approach Layer 3/4 filtering includes Kubernetes network policy to update rules in a dynamic manner, protecting deployments as they change and scale. It is based on IP addresses and ports. Simple network segmentation rules are not enough to provide the robust monitoring, logging, and threat detection required for business critical container deployments. But it can provide some protection against unauthorized connections.

• WAF attack detection can protect the web facing containers using HTTP methods that detect common attacks. However, the protection is limited to external attacks over HTTP, and lacks the multi-protocol filtering needed for internal traffic.

• A container firewall with Layer 7 filtering and DPI of inter-pod traffic secures containers using network application protocols. Protection is based on application protocol whitelists as well as built-in detection of common network based application attacks such as DDoS, DNS, and SQL injection. Container firewalls also are in a unique position to incorporate container process monitoring and host security into the threat vectors monitored. Deep packet inspection (DPI) techniques. Exploits usually use predictable attack vectors: malicious HTTP requests with a malformed header, or inclusion of an executable shell command within the XML object. Layer 7 Deep Packet Inspection for in-depth network security in a container firewall can look for and recognize these methods. Container firewalls using these techniques can determine whether each pod connection should be allowed to go through, or if they are a possible attack which should be blocked.

Host Security

If the Kubernetes worker node on which containers run is compromised, it is difficult to predict the after effect, all kinds negative things can happen. These include:

• Privilege escalations to root
• Stealing of secrets used for secure application or infrastructure access
• Changing of cluster admin privileges
• Host resource damage or hijacking
• Stopping of critical orchestration tool infrastructure such as the API Server or the Docker daemon
• Starting of suspicious processes

The host system are needed to be monitored for these kind of suspicious activities.. Together, the combination of network inspection, container inspection, and host security offer the best way to detect a kill chain from several vectors.

Container Inspection

Attacks utilize privilege escalations and malicious processes to carry out an attack or spread it. Exploits in the Linux kernel, packages, libraries or applications can result in suspicious activity within a container.

The critical element of container security is inspecting all the container processes and file system activity to detect the suspicious behavior. Suspicious processes such as port scanning and reverse shells, or privilege escalations should all be detected. There are combination of built-in detection as well as a baseline behavioral learning process which can identify unusual processes.

The applications with containers which are designed with microservices principles can detect the suspicious processes and file system activity more easier and will be more accurate.

Securing the Kubernetes System and Resources

If not protected, orchestration tools like Kubernetes and the management platforms can be vulnerable to attacks. These expose a new attack surface for container deployments and thus will be attempted to be exploited by hackers.

It is critical to properly setup the RBACs for system resources to protect Kubernetes and management platforms . The areas to configure properly to have the access controls are:

1. Configure RBAC for the API Server or manually create firewall rules to prevent unauthorized access.

2. Restrict Kubelet Permissions and manage certificate rotation.

3. Require Authentication for All External Ports. For non-authenticated services, restrict access to a whitelist source.

4. Don’t allow console/proxy access unless properly configured for user login with strong passwords or two-factor authentication.

It is recommended to track access to infrastructure services to detect unauthorized connection attempts and potential attacks using the monitoring tools. Real-time, policy based monitoring of the containers, hosts, network and system resources would have detected suspicious processes as well as unauthorized external connections.

Auditing and Compliance for Kubernetes Environments

Kubernetes, enterprises are getting constantly updating, upgrading, and migrating the container environment. Running a set of security tests designed for Kubernetes environments will ensure that security does not regress with each change. As more enterprises migrate to containers, the changes in the infrastructure, tools, and topology may require re-certification for compliance standards like PCI.

Already we have a comprehensive set of Kubernetes security and Docker security health checks through the CIS Benchmarks for Kubernetes and the Docker Bench tests and confirms expected results are automated. These test focus on the below areas:

• Host security
• Kubernetes security
• Docker daemon security
• Container security
• Properly configured RBACs
• Securing data at rest and in transit

Vulnerability scanning of containers in production is a core component for preventing known exploits to achieve compliance. However, it is not enough to provide the multiple vectors of security needed to protect runtime container deployments.