Container Monitoring with cAdvisor, Prometheus and Grafana on Kubernetes

Containers as the preferred runtime environment for applications have soared in popularity as more companies have begun adopting DevOps as their applications lifecycle management culture. Through this tutorial, we will look at container monitoring with Prometheus and Grafana on Kubernetes whilst deploying two containerized applications to our Kubernetes cluster and scraping the CAdvisor exposed metrics with Prometheus.

Before we begin, let us explore the terminology that will be used throughout this tutorial:

• DevOps a work culture to enable fast-growing companies to speed up their software release lifecycle, allowing them to implement new features, make changes and integrate third-party libraries and APIs more effectively.

• Containers are the standard unit of executables. They consist of an entire runtime environment such as an application, its dependencies, configuration files, and libraries, which are needed to run it. Containers do this by virtualizing the operating system allowing for portability.

• Portable Containers mean moving applications from your developer’s laptop so updates on the host server can happen faster and more efficiently. This software agility helps us make changes and implement new features faster and safer.

• Kubernetes is a container orchestration platform. It is a portable, extensible, open-source platform for managing containerized applications. A Kubernetes cluster can handle thousands of microservices packaged and run as containers making it ideal for running services at scale.

Why monitor container performance?

Tracking a containerized application's health and environment status helps us make better service decisions. We can use the data from our monitoring system to set up automated responses to things such as when to scale up or down, start a task, etc. This can help improve our service availability and prevent system outages.

One of the significant challenges with monitoring containers comes from their ephemeral nature. In other words, because containers are designed to be compact and portable, systems like Kubernetes tend to easily discard them as changes happen. Additionally, an application can run across many container instances on different machines and even cloud services, so tracing events across them can be challenging.

Containers also share resources such as memory and CPU across the host machine(s). This provides some difficulty when it comes to monitoring resource consumption and usage.

A good container monitoring stack should cover metrics such as CPU usage, memory utilization, CPU limits, Read/Write operations, etc.

CAdvisor for Container Metrics

CAdvisor (Container Advisor) is a running daemon that can be installed on our host servers. It collects, aggregates, processes, and exports information about containers running on the host. Examples of container metrics collected by CAdvisor include:

container_cpu_load_average_10s This measures the value of the container CPU load average over the last 10 seconds.

container_memory_usage_bytes This measures current memory usage.

container_fs_io_time_seconds_total This measures the cumulative counts in seconds spent doing input/output (I/O) operations.

container_network_receive_errors_total This measures the cumulative count of errors encountered while receiving bytes over your network.

CAdvisor is open source and can collect data from all different types of containers, including Docker, containerd, and others.

You will need to have installed Docker on your operating system to follow along with the following.

You can install CAdvisor using its Docker image with the following command:

VERSION=v0.36.0
sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  --privileged \
  --device=/dev/kmsg \
  gcr.io/cadvisor/cadvisor:$VERSION

Additionally, you can clone the repository and build the binary yourself.

CAdvisor does an excellent job of collecting and processing container data as metrics. However, it is not designed to store this data long-term, so you would need a dedicated monitoring tool to work alongside.

Prometheus

Prometheus is an open-source monitoring and alerting toolkit which collects and stores metrics as time series data. It has a multidimensional data model which uses key/value pairs to identify data, a fast and efficient query language (PromQL), service discovery, and does not rely on distributed storage.

CAdvisor exports data collected from containers as Prometheus metrics and serves them at the /metrics endpoint. We can configure Prometheus using service discovery or Jobs to scrape the metrics from the endpoint.

Prometheus has a default storage duration of up to 15 days and can be configured to store metrics longer. We can also use PromQL to query the data and set alerts for our services.

Monitoring Kubernetes Managed Containerised Applications

As Kubernetes primarily deals with containers, it is no surprise that CAdvisor is integrated into the Kubelet binary. To deploy our applications, we will use Civo’s managed Kubernetes service. Civo’s cloud-native infrastructure services are powered by Kubernetes and use the lightweight Kubernetes distribution k3s for superfast launch times.

Prerequisites

To get started, we will need the following:

• Civo Account
• Civo Command Line Client
• Helm
• Kubectl

After setting up the Civo command line with our API key using the instructions in the repository, we can create our cluster using the following command. It will create a three-node cluster with a default size:

civo kubernetes create civo-cluster

You will be able to see on the dashboard that our cluster ‘civo-cluster’ is created:

Next, we will set up our monitoring stack in the cluster by installing Prometheus via the Prometheus Operator. In order to do this, make sure your KUBECONFIG is downloaded from the cluster page and set to be your default KUBECONFIG for kubectl.

We will start by creating a namespace called ‘monitoring’ where all our monitoring resources will reside:

kubectl create ns monitoring

We will now install Prometheus Operator with the manifests using the following command:

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml -n monitoring

The Prometheus operator deploys the following Custom Resource Definitions which allow us to describe how we want to deploy our Prometheus components declaratively:

• Prometheus
• Alert manager
• Service Monitor
• Pod Monitor
• Probes
• Prometheus rules

The following code deploys an instance of Prometheus. It defines the namespace from which we want to pull metrics, the service monitors to select, the service account to use, etc., for our Prometheus Deployment.

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  labels:
    app: prometheus
spec:
  serviceAccountName: prometheus
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  podMonitorSelector: {}
  resources:
    requests:
      memory: 400Mi

Save the above YAML into a file, prometheus.yaml and apply it to your cluster.

Our Prometheus deployment needs to operate undisturbed, and for that, we can create a service account and assign it some permissions. We will then use ClusterRoleBinding to link our service account to our permissions.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus




apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get"]
- nonResourceURLs: ["/metrics", "/metrics/cadvisor"]
  verbs: ["get"]



apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring

Save the above YAML as serviceaccount.yaml and create the resources using the following command:

kubectl apply -f serviceaccount.yaml

We can expose our Prometheus service using port-forwarding. To access the Prometheus User Interface from our web browser, use the following command:

kubectl port-forward {PODNAME} 9090:9090

Once the port-forward has been established, we can reach Prometheus at the following URL <a href="http://localhost:9090/" target="blank" rel="noopener">http://localhost:9090/:

Deploying Our Application

Our application, civoapp is a two-tier application. A client-side with three endpoints /home, /error, and /metrics. It processes its business logic on an ngnix webserver. The code below (civoappdep.yaml) deploys three instances of our client-side application:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: civoapp-deployment
  labels:
    app: civoapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: civoapp
  template:
    metadata:
      labels:
        app: civoapp
    spec:
      containers:
      - name: civoapp
        image: ehienabs/civoapp:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080




apiVersion: v1
kind: Service
metadata:
  name: civoapp-service
  labels:
    app: civoappsvc
spec:
  selector:
    app: civoapp
  ports:
    - name: http
      port: 80
      targetPort: 8080
  type: LoadBalancer

The following code (ngnixdep.yaml) deploys three replicas of our backend ngnix webserver:

---
apiVersion: apps/v1 
kind: Deployment
metadata:
  name: nginx
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nginx
  replicas: 3 
  template: 
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:



          - containerPort: 80



apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginxsvc
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Next, we will create a namespace for our application with the following command:

kubectl create ns civoapp

Deploy the application using the following command:

kubectl apply -f civoappdep.yaml ngnixdep.yaml

To view our container metrics we will first enable temporary access to our API server using the following command:

kubectl proxy

We can view our container metrics using the following URL: http://127.0.0.1:8001/api/v1/nodes/{NODE_NAME}/proxy/metrics/cadvisor

👉🏾N.B, where NODENAME is the name of one of the nodes in our Kubernetes cluster.

Service Discovery With Prometheus

Although we have our Kubernetes cluster ready for monitoring, and we have CAdvisor collecting metrics from our containers, Prometheus has no idea of our container metrics. We can still only view our container metrics through the browser, which is tedious and not convenient.

Service Discovery the way Prometheus finds endpoints to scrape. It allows Prometheus to adapt to dynamic environments such as Kubernetes. Prometheus has a Service Monitor object which we can use to configure scrape targets.

Using the following code, we will use a service monitor to configure our Prometheus to scrape the /metrics endpoint:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus
  labels:
spec:
  selector:
    matchLabels:
  namespaceSelector:
    any: true
  endpoints:
    - path: /metrics/cadvisor
    - Port: http-metrics

Once Prometheus is configured to pull metrics from CAdvisor, we can view a list of metrics from our Prometheus user interface:

Visualizing Container Metrics With Grafana

Visualizing metric data can help us keep abreast of the trends in our applications. We are better informed when making changes such as scaling, etc. Grafana is an open-source interactive data visualization platform helpful in visualizing metrics, logs, and traces collected from your applications. Grafana allows us to pull metrics from various sources, run queries against them, visualize them, and organize them into dashboards.

We can install Grafana with helm using the following code. First, we will add the repository to helm:

helm repo add grafana https://grafana.github.io/helm-charts

Then we can install the chart:

helm install grafana grafana/grafana

We can expose Grafana service using port-forwarding:

kubectl port-forward grafana-5874c8c6cc-h6q4s 3000:3000

Note: You will need to change the pod name to the one that is running on your cluster, which you can find with kubectl get pods -A

Users can access the Grafana User Interface via the following URL: http://localhost:3000/

To log in to our Grafana service, we will use the default username admin and retrieve the password using the following code:

kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

And now we are in our Grafana service:

To visualize data in Grafana, we must first add a data source. A data source is typically the output of our monitoring system. Grafana allows for a variety of data sources, including Prometheus.

To add a data source, from the settings button on the right of the dashboard, click on data source and add Prometheus.

👉🏾 N.B Prometheus is in the same cluster as our Grafana service, meaning they can communicate using their local DNS. Therefore we can add Prometheus using its local DNS name.

Now that we have added a data source, we can start creating dashboards and visualizing our data:

Grafana dashboards are a set of one or more panels arranged into one or more rows. Each panel is a visualization of a constructed query from a data source. Dashboards are a way to group various panels to provide a holistic look at our services.

Additionally, Grafana allows us to import templated dashboards in various ways. These templated dashboards are reusable and can be shared as JSON, URLs, or Grafana Dashboard ID. Grafana also supports a host of dashboards that can be easily imported with their IDs.

To import a Grafana dashboard, click on import from the dashboard menu at the right corner:

We will be provided with a list of import options. We can import Docker Host and Containers dashboard template using the ID 395:

We now have panels visualizing some of our preferred metrics, including CPU_Usage_per_Container, Memory_Usage_per_Container and we can see how our container services perform:

Wrapping Up

By following this guide, we have gained an understanding of monitoring containers in Kubernetes using CAdvisor. We deployed two containerized applications to our Kubernetes cluster and scraped the CAdvisor exposed metrics with Prometheus. Finally, we visualized the data using Grafana.