Ensuring YAML best practices using KubeLinter

In the Kubernetes world, we all play with YAML files, deploy them in order to create various Kubernetes objects, but the challenge is whether we are following best practices when writing them? Are we using the right set of standard configurations? Can we have YAML checked prior to the deployment of applications or even Helm charts? The answer to all these is yes, we can. On 28th October 2020, StackRox introduced a new open-source tool named KubeLinter aimed at identifying any misconfigurations in YAML files.

By definition, KubeLinter is a static analysis tool that checks Kubernetes YAML files and Helm charts to ensure the applications represented in them adhere to best practices. Once a YAML file is supplied to the tool, it will run through the built-in checks and then give a detailed report any errors, and the remediations for solving them. The best part about this tool is that it is configurable and extensible: The built-in checks can be enabled or disabled, and you can define and use your own custom checks.

Usage

I will be installing KubeLinter on a Mac, but the same instructions can be used for Linux as well by just downloading the appropriate release for your operating system:

Download the KubeLinter CLI from the releases page.

$ curl -LO https://github.com/stackrox/kube-linter/releases/download/0.1.1/kube-linter-darwin.zip
$ unzip kube-linter-darwin.zip
$ mv kube-linter /usr/local/bin

#check if it's working
$ kube-linter version
0.1.1

Now, in order to simply check a single YAML file, just supply the YAML filename. Say that you have the below deploy.yaml file that you want to check for best security and configuration practices saved in your current directory:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: portainer
  namespace: portainer
  labels:
    io.portainer.kubernetes.application.stack: portainer
    app.kubernetes.io/name: portainer
    app.kubernetes.io/instance: portainer
    app.kubernetes.io/version: "2.0.0"
spec:
  replicas: 1
  strategy:
    type: "Recreate"
  selector:
    matchLabels:
      app.kubernetes.io/name: portainer
      app.kubernetes.io/instance: portainer
  template:
    metadata:
      labels:
        app.kubernetes.io/name: portainer
        app.kubernetes.io/instance: portainer
    spec:
      serviceAccountName: portainer-sa-clusteradmin
      volumes:
         - name: "data"
           persistentVolumeClaim:
             claimName: portainer
      containers:
        - name: portainer
          image: "portainer/portainer-ce:latest"
          imagePullPolicy: IfNotPresent
          args:  [ '--tunnel-port','30776' ]
          volumeMounts:
            - name: data
              mountPath: /data
          ports:
            - name: http
              containerPort: 9000
              protocol: TCP
            - name: tcp-edge
              containerPort: 8000
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /
              port: 9000
          readinessProbe:
            httpGet:
              path: /
              port: 9000
          resources:
            {}

Let's run the test using kube-linter lint deploy.yaml. We should get output like the following:

deploy.yaml: (object: portainer/portainer apps/v1, Kind=Deployment) container "portainer" does not have a read-only root file system (check: no-read-only-root-fs, remediation: Set readOnlyRootFilesystem to true in your container's securityContext.)

deploy.yaml: (object: portainer/portainer apps/v1, Kind=Deployment) serviceAccount "portainer-sa-clusteradmin" not found (check: non-existent-service-account, remediation: Make sure to create the service account, or to refer to an existing service account.)

deploy.yaml: (object: portainer/portainer apps/v1, Kind=Deployment) container "portainer" is not set to runAsNonRoot (check: run-as-non-root, remediation: Set runAsUser to a non-zero number, and runAsNonRoot to true, in your pod or container securityContext. See https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ for more details.)

deploy.yaml: (object: portainer/portainer apps/v1, Kind=Deployment) container "portainer" has cpu request 0 (check: unset-cpu-requirements, remediation: Set your container's CPU requests and limits depending on its requirements. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits for more details.)

deploy.yaml: (object: portainer/portainer apps/v1, Kind=Deployment) container "portainer" has cpu limit 0 (check: unset-cpu-requirements, remediation: Set your container's CPU requests and limits depending on its requirements. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits for more details.)

deploy.yaml: (object: portainer/portainer apps/v1, Kind=Deployment) container "portainer" has memory request 0 (check: unset-memory-requirements, remediation: Set your container's memory requests and limits depending on its requirements. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits for more details.)

deploy.yaml: (object: portainer/portainer apps/v1, Kind=Deployment) container "portainer" has memory limit 0 (check: unset-memory-requirements, remediation: Set your container's memory requests and limits depending on its requirements. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits for more details.)

Error: found 7 lint errors

Types of checks

As you can see there are errors spotted in the YAML file with clear remediation steps for each.

If you want to have a look at the built-in checks, all of them are listed here – or you can have KubeLinter tell you the list:

$ kube-linter checks list
Name: dangling-service
Description: Alert on services that don't have any matching deployments
Remediation: Make sure your service's selector correctly matches the labels on one of your deployments.
Template: dangling-service
Parameters: map[]
Enabled by default: true

------------------------------

Name: default-service-account
Description: Alert on pods that use the default service account
Remediation: Create a dedicated service account for your pod. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/ for more details.
Template: service-account
Parameters: map[serviceAccount:^(|default)$]
Enabled by default: false
|
|
|
and many more

Ignoring a check

You can ignore a specific check for a YAML file, or groups of checks as per your need by using the following annotations:

ignore-check.kube-linter.io/<check-name>

for example:

ignore-check.kube-linter.io/unset-cpu-requirements

You can add a ignore-check rule to the above deployment file example like this:

metadata:
  name: portainer
  namespace: portainer
  labels:
    io.portainer.kubernetes.application.stack: portainer
    app.kubernetes.io/name: portainer
    app.kubernetes.io/instance: portainer
    app.kubernetes.io/version: "2.0.0"
  annotations:
    ignore-check.kube-linter.io/unset-cpu-requirements : "cpu requirements not required"

Now, when you run kube-linter lint deploy.yaml again you should see no lint errors related to CPU requirements.

Run with a configuration

KubeLinter can also run with a config where you can provide all the information on the checks to be included/excluded. Below is the example configuration from the repository:

# customChecks defines custom checks.
customChecks:
- name: "required-label-app"
  template: "required-label"
  params:
    key: "app"
checks:
  # if doNotAutoAddDefaults is true, default checks are not automatically added.
  doNotAutoAddDefaults: false

  # addAllBuiltIn, if set, adds all built-in checks. This allows users to
  # explicitly opt-out of checks that are not relevant using Exclude.
  # Takes precedence over doNotAutoAddDefaults, if both are set.
  addAllBuiltIn: false

  # include explicitly adds checks, by name. You can reference any of the built-in checks.
  # Note that customChecks defined above are included automatically.
  include:
  - "required-label-owner"
  # exclude explicitly excludes checks, by name. exclude has the highest priority: if a check is
  # in exclude, then it is not considered, even if it is in include as well.
  exclude:
    - "privileged"

If you run kubelinter --config config lint deploy.yaml where config is a file with the above-mentioned configuration, you should be able to see extra check added for "required-label":

deploy.yaml: (object: portainer/portainer apps/v1, Kind=Deployment) no label matching "owner=<any>" found (check: required-label-owner, remediation: Add an email annotation to your object with information about the object's owner.)

KubeLinter can prove to be a very useful tool when built into a CI pipeline. Code pushed into the repository can be checked and validated for best practices and security considerations, generating alerts if any are detected.

This way there can be healthy production-ready applications deployed to the cluster. This project is at an alpha stage but looks promising as a CI integration tool to validate any deployments before actually deploying to production.