Kubernetes has emerged as the industry standard for container orchestration, empowering developers to streamline the deployment and management of applications. However, misconfigurations in Kubernetes configuration files can lead to operational issues, security vulnerabilities, scalability challenges, inefficient resource allocation, and application downtime. For this reason, this tutorial will delve into the significance of validating Kubernetes configuration files to avoid these pitfalls and provide a practical demonstration of how to do all this using Civo's managed Kubernetes service.
Pre-requisites
To follow along with this tutorial, you will need the following:
- A Civo account: Sign up for a Civo account if you don't already have one.
- Kubernetes cluster launched: Learn how to launch a cluster on Civo here and install the Civo CLI from here.
- Install Helm, the Kubernetes package manager.
- Familiarity with Kubernetes configuration files.
Validating Kubernetes configuration files on Civo
Validation plays a crucial role in ensuring the reliability and stability of software projects. By validating configuration files, codebases, and other critical files, you can catch errors, enforce best practices, and prevent issues from arising in your applications. This validation process helps maintain consistency, reduces downtime, and enhances overall software quality.
Datree is a powerful tool that fits seamlessly into your development workflow to validate files and enforce best practices. It integrates with your code repository and automatically scans your files, identifying potential problems, misconfigurations, and security risks. With Datree, you can establish and enforce coding standards, adhere to compliance requirements, and mitigate misconfigurations risks.
1. Installing the Datree CLI
Install the Datree CLI for your operating system by following the instructions in their documentation.
On Linux and MacOS, you can use the following command:
curl https://get.datree.io | /bin/bash
With the following command, we can test an example Kubernetes manifest within the installed datree repository.
datree test ~/.datree/k8s-demo.yaml
This command will compare the Kubernetes manifest to the policies that have been configured.
2. Configuring Datree for your cluster
Once the Datree CLI is installed, configure it to connect with your running Civo cluster.
a. Add the Datree Helm repository
helm repo add datree-webhook https://datreeio.github.io/admission-webhook-datree
helm repo update
b. Install Datree on your Civo cluster
Replace
helm install -n datree datree-webhook datree-webhook/datree-admission-webhook --debug \
--create-namespace \
--set datree.token=<DATREE_TOKEN> \
--set datree.clusterName=$(kubectl config current-context)
Datree will now run in the background, scanning your clusters for misconfigurations.
3. Run validation checks
Use the Datree CLI to run validation checks on your uploaded Kubernetes configuration files on the Civo cluster. Datree will analyze the files against predefined best practices and policies, providing feedback on potential misconfigurations.
4. Interpret the validation results
Upon reviewing the validation results furnished by Datree, you'll encounter a variety of findings, encompassing misconfigurations, warnings, and enhancement suggestions. Let's explore an example of a frequently encountered misconfiguration, how Datree reports it, and its role in addressing the issue.
Example: Misconfiguration of Pod Resource Limits
Misconfigurations pertaining to Kubernetes Pods' resource limits are prevalent issues that can precipitate performance bottlenecks and resource depletion. Datree plays a crucial role in pinpointing these misconfigurations and offering rectification suggestions.
As Datree scans your files, it scrutinizes the resource limits set for Pods in your Kubernetes manifests. If a misconfiguration, such as a Pod devoid of defined resource limits, is detected, it will be flagged as a problem in the validation results.
Datree's validation results typically comprise a detailed breakdown of the misconfiguration, spotlighting the exact file and line number where the issue resides. This might also include further context and explanations to facilitate better comprehension.
To rectify this misconfiguration, Datree proposes defining suitable resource limits for your Pods. Suggestions might encompass setting limits for CPU and memory utilization to ensure balanced resource distribution and to avoid any single Pod from monopolizing resources, which could detrimentally affect the stability and performance of other Pods within the cluster.
By implementing these recommendations, you can minimize the risk associated with resource-related problems and boost the efficiency and reliability of your Kubernetes cluster.
Preventing future Kubernetes misconfigurations
Datree provides a demo file allowing you to validate a deployment object for those that don’t have one. Here is the content of the demo deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: rss-site
labels:
environment: prod
app: web
spec:
replicas: 2
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: front-end
image: nginx:latest
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
resources:
requests:
memory: "64Mi"
cpu: "64m"
limits:
cpu: "500m"
ports:
- containerPort: 80
- name: rss-reader
image: datree/nginx@sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2
livenessProbe:
httpGet:
path: /healthz
port: 8080
httpHeaders:
- name: Custom-Header
value: Awesome
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
resources:
requests:
cpu: "64m"
memory: "128Mi"
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 88
Use the following command to verify the deployment YAML file:
datree test <path_of_your_kubernetes_manifest_YAML_NAME>
So in our case, it will be:
datree test ~/.datree/k8s-demo.yaml
You will get similar output to this:
>> File: .datree/k8s-demo.yaml
[V] YAML validation [V] Kubernetes schema validation
[X] Policy check
`❌ Ensure seccomp profile is set to docker/default or runtime/default [1 occurrence]
metadata.name: rss-site (kind: Deployment)
key: spec.template.metadata (line: 17:7)`
💡 Invalid value for key seccomp.security.alpha.kubernetes.io/pod - set to docker/default or runtime/default to ensure restricted privileges
`❌ Ensure containers and pods have a configured security context [1 occurrence]
metadata.name: rss-site (kind: Deployment)`
💡 Missing key securityContext - set to enforce your containers' security and stability
`❌ Prevent the admission of containers with the NET_RAW capability [2 occurrences]
metadata.name: rss-site (kind: Deployment)
key: spec.template.spec.containers.0 (line: 22:11) key: spec.template.spec.containers.1 (line: 37:11)`
💡 Invalid value for key drop - prohibit the potentially dangerous NET_RAW capability
`❌ Ensure each container has a read-only root filesystem [2 occurrences]
metadata.name: rss-site (kind: Deployment)
key: spec.template.spec.containers.0 (line: 22:11) key: spec.template.spec.containers.1 (line: 37:11)`
💡 Incorrect value for key readOnlyRootFilesystem - set to 'true' to protect filesystem from potential attacks
`❌ Prevent container from running with root privileges [2 occurrences]
metadata.name: rss-site (kind: Deployment)
key: spec.template.spec.containers.0 (line: 22:11) key: spec.template.spec.containers.1 (line: 37:11)`
💡 Invalid value for key runAsNonRoot - must be set to true to prevent unnecessary privileges
`❌ Ensure each container image has a pinned (tag) version [1 occurrence]
metadata.name: rss-site (kind: Deployment)
key: spec.template.spec.containers.0.image (line: 23:18)`
💡 Incorrect value for key image - specify an image version to avoid unpleasant "version surprises" in the future
`❌ Prevent containers from escalating privileges [2 occurrences]
metadata.name: rss-site (kind: Deployment)
key: spec.template.spec.containers.0 (line: 22:11) key: spec.template.spec.containers.1 (line: 37:11)`
💡 Missing key allowPrivilegeEscalation - set to false to prevent attackers from exploiting escalated container privileges
`❌ Ensure each container has a configured liveness probe [1 occurrence]
metadata.name: rss-site (kind: Deployment)
key: spec.template.spec.containers.0 (line: 22:11)`
💡 Missing property object livenessProbe - add a properly configured livenessProbe to catch possible deadlocks
`❌ Ensure each container has a configured memory limit [1 occurrence]
metadata.name: rss-site (kind: Deployment)
key: spec.template.spec.containers.0.resources.limits (line: 34:15)`
💡 Missing property object limits.memory - value should be within the accepted boundaries recommended by the organization
`❌ Ensure workload has valid label values [1 occurrence]
metadata.name: rss-site (kind: Deployment)
key: metadata.labels.owner (line: 7:12)`
💡 Incorrect value for key(s) under labels - the vales syntax is not valid so the Kubernetes engine will not accept it
(Summary)
Passing YAML validation: 1/1
Passing Kubernetes (1.20.0) schema validation: 1/1
Passing policy check: 0/1
+-----------------------------------+------------------------------------------------------+ | Enabled rules in policy "Starter" | 67 | | Configs tested against policy | 1 | | Total rules evaluated | 67 | | Total rules skipped | 0 | | Total rules failed | 10 | | Total rules passed | 57 | | See all rules in policy | <https://app.datree.io/login?t=3Kar3X39tUkQRzFx68huqT> |
In a meetup with Shimon Tolts and Kunal Kushwaha, they discussed preventing Kubernetes misconfigurations and deprecations. You can watch the full “Preventing Kubernetes misconfigurations and deprecations with Datree” video here:
Conclusion
This article discussed the importance of validating Kubernetes configuration files and preventing misconfigurations in order to ensure the stability, security, and efficient operation of your Kubernetes clusters. By utilizing Civo's Kubernetes expertise and the powerful validation capabilities of Datree, you can identify and prevent misconfigurations that could lead to operational issues and security vulnerabilities. Incorporating these practices into your deployment pipeline ensures a smooth and efficient Kubernetes experience, reducing the risk of application downtime and ensuring a robust application infrastructure.
