In the world of container orchestration, Kubernetes has attained de-facto status and has become the most loved container orchestration platform. More and more companies are migrating their applications to Kubernetes. However, this application migration has inevitably attracted many cyber attackers, who have set Kubernetes as their main target.

Cyber attackers disrupt the application deployment and production stages by stealing Kubernetes secrets and injecting malware on unsecured clusters. The RedHat State of Kubernetes Security Report has shown that 94% of the survey respondents have experienced a Kubernetes runtime attack and 55% of the respondents had to slow down the deployment phase due to Kubernetes or container security issues.

29% percent of companies surveyed do not invest competently in container security. Poor security investments lead to developers having fewer resources and tools to secure the Kubernetes platform and cloud infrastructure. When Kubernetes security is weak cyber attackers can use simple techniques, such as stealing secrets from unencrypted etcd or plaintext files.

In this article, you will learn about analysis of Kubernetes runtime security threats and also learn how to detect and prevent them before a disaster happens.

What this article will cover

  1. What are Kubernetes runtime security threats?
  2. How to detect and prevent Kubernetes runtime security threats
  3. Conclusion

What are Kubernetes runtime security threats?

Security vulnerabilities that bypass the testing stage and disrupt running containers fall under the Kubernetes runtime security threats category. Kubernetes runtime security vulnerabilities make the biggest impact because they can hijack running applications or even steal critical data that can be used to impersonate you in order to steal data further down the line.

Usually, cyber attackers want to cause chaos in your systems and seize information such as:

  • Credentials
  • Intellectual property
  • Customer data and personally identifiable information

For you to be able to understand Kubernetes runtime security threats and also know how vulnerabilities reach the container runtime phase and disrupt it clearly, let us go through the steps followed by developers when working with containers:

  1. Collecting container resources: The first step begins when you start searching and downloading container images from DockerHub or other container image repositories. If you download insecure images acquired from un-vetted sources, and use untrusted components that will be used in the deployment and delivery stage of your application, these can cause trouble once running, if they get past your testing phase.
  2. Build: The building process involves assembling and using all the content acquired in the first stage. The images are used to build containers and applications are also built and configured in this stage. The main risk that lies in this stage is with misconfiguration. These misconfigurations can arise when configuring clusters inappropriately, and when writing manifests for your resources. For example, if a Kubernetes administrator turns --kubelet-https to false while configuring a cluster, this will lead to traffic between the Kubernetes API server and the kubelets being unencrypted.

Misconfigurations can escalate the impact of any potential security breach. As an example, when a container is given high privileges, attackers can have a bigger impact since the container has privileges that can be used to access the host.

  1. Delivery and deployment: this stage involves delivering the application into a production environment and making sure that it starts running after all requirements and checks have been met. If penetration testing or code analysis is done poorly, vulnerabilities can be missed and pass through this stage causing potential technical problems during container runtime.
  2. Running and maintaining: This stage involves debugging applications and improving application scalability. All the vulnerabilities that were not caught in the previous phases may be used to disrupt the container runtime or exploited by cyber attackers through the Kubernetes API. Getting application visibility and detecting logs is used in this stage to detect container runtime threats.

How to analyze and detect Kubernetes runtime security threats

Knowing what Kubernetes components and vulnerabilities are mainly targeted by attackers is important as it helps you make data-driven decisions. Commonly-targeted vulnerabilities include:

  • Containers running with privileged flags
  • Using insecure images
  • Container and Kubernetes misconfiguration
  • Unprotected communication between containers

Attackers may try to exploit vulnerabilities across the entire Kubernetes architecture, including the containers themselves, the pods that the containers are grouped in, or the Kubelet on the node itself.

How to detect and prevent Kubernetes runtime vulnerabilities

We will look at four ways, both reactive (analysing after an event) and pre-emptive (practices before deployment) to manage Kubernetes runtime vulnerabilities. SysDig produced a great guide that has more in-depth analysis of these methods and others.

Identifying errors and vulnerabilities in logs

Logs provide the best visibility on how your cluster and application performs. Regularly inspecting logs is a key way to detect and eliminate Kubernetes runtime threats.

Unfortunately, Kubernetes does not have a built-in logging mechanism that filters and shows you the root cause. But, there are third-party tools like ELK (Elasticsearch, Logstash, Kibana), Prometheus/Loki/Grafana, and others that filter and visualise logs. These mechanisms can help give you the root cause of a problem or bug instead of giving you a thousand lines of codes to sift through.

Tools that can help you eliminate logging complexity and provide management services include:

  • Sysdig Falco analyzes running applications. Falco will alert you when any socket that does not meet the specified criteria is listening.
  • ELK stack, as mentioned above, can be integrated with Kubernetes to aggregate and analyze logs.
  • Zebrium uses machine learning to analyze logs to find main causes of errors.
  • Grafana Loki can integrate with Kubernetes and the Prometheus time-series database natively to store logs and aggregate them.
  • Sematext monitors infrastructure and provides logging without the need to maintain ElasticSearch.

It's important to note that natively, container engine logging in Kubernetes only keeps logs around as long as the container runs. A container-level crash or failure means the logs can be lost and you can't retrospectively see the events that led to it. For this kind of after-event debugging a tool like Sloop is useful, as if you are running it you can visualize events even after pod eviction or container crash.

As you analyze logs compare previous performance and results, paying attention to the following information:

  1. API request history
  2. Performance metrics
  3. Deployments
  4. Resource consumption
  5. Operating system calls
  6. Protocols and permission changes
  7. Network traffic

When you spot errors during log inspection, aside from making sure that you correct them, it is imperative to keep on logging and analyzing the gathered data as these fixes may affect your application in other ways.

Scanning components and pods for misconfigurations

More than 1.2 billion containers have been downloaded. Not all container images downloaded from public repositories such as DockerHub are trusted and secure. It is therefore vital to scan all images you are going to use, and only use images that are official if you are not building them yourself.

Using free tools such as Checkov and Kube-hunter which either run as a container outside your cluster or as a pod within is useful this kind of static analysis. These tools allow you to scan any misconfigurations and vulnerabilities. Kube--hunter will also inspect your Kubernetes setup, and generate a report which details how your cluster is vulnerable to known explots.

Penetration testing

Penetration testing should be done regularly to ensure that the Kubernetes cluster is secured. Penetration testing helps you simulate the event of a cyber attacker seizing a cluster, and get an insight on how to alleviate vulnerable points and known issues that an attacker can use to exploit the cluster. Penetration testing also helps you identify things such as excessive service account privileges (RBAC), and issues such as unsecured Kubelet API.

Security compliance standards such as PCI DSS require penetration testing to be carried out after making significant changes. These compliance standards boost your organization's trustworthiness, allow your brand or application to be considered as secure when confirmed to be compliant.

Setting a remedial plan

Be prepared as if you have been told that you will experience a Kubernetes security breach tomorrow. Create a plan and a playbook that will guide you on what to do during and after a security breach. The playbook should state what remedial actions should be taken in a variety of situations, such as when:

  • Malware has been activated
  • When a node has been attacked
  • Secrets have been stolen
  • Container privileges have been escalated

Conclusion

Securing Kubernetes may seem like a daunting task since Kubernetes is a dynamic and fast-developing platform with many components. But, consistently scanning containers and auditing component manifests and logs is the only way you can detect and eliminate vulnerabilities for your particular application. It is also important to adhere to best practice around handling secrets such as secure peer to peer communication. Of course, other components of your infrastructure should make use of well-configured firewalls and strong authentication to limit the potential attack surface to areas outside your Kubernetes setup.