Creating a Prometheus Exporter Using Go

Metrics are a big slice of the observability pie, allowing teams to gain quantitative insights into the performance of their applications. Prometheus has established itself as a standard for generating and collecting time series data in the cloud-native world.

In this tutorial, we will take a look at how to generate custom metrics and understand Prometheus metric types. We’ll wrap up by making sense of the metrics generated by visualizing.

What’s Prometheus?

Originally developed at SoundCloud in 2012 and later donated to the CNCF in May of 2016, Prometheus is an open-source monitoring and alerting system that stores metrics as time-series data. What really sets Prometheus apart is its powerful query language called PromQL, which allows users to create simple and complex queries on the metrics they export, and the interesting “pull model,” which means the Prometheus server pulls metrics from targets as opposed to the traditional push-based system where applications “push” metrics to a central server.

Prometheus Metric Types

Before we can expose our metrics, it's important to understand the various metric types that Prometheus supports:

Hopefully, this will give you a clear picture of Prometheus core metric types. With that out of the way, let’s proceed to implementation.

Prerequisites

This tutorial assumes some familiarity with Golang and Kubernetes. Additionally, you will need the following in order to proceed:

• A Civo Account
• Kubectl installed
• Helm
• Go 1.21+

Creating a Cluster on Civo

To create a cluster using the Civo CLI, run the following command:

civo k3s create --create-firewall --nodes 1 -m --save --switch --wait exporter-demo

This would launch a one-node Kubernetes cluster in your Civo account, the -m flag would merge the kubeconfig for the new cluster with your existing kubeconfig, --switch points your kube-context to the newly created cluster.

Setting up the development environment

To make the setup process easier, we'll be working with a sample repository that already has some code in place. This will allow us to focus on the important concepts without getting bogged down in project setup.

Clone the sample repository:

git clone git@github.com:s1ntaxe770r/prometheus-http.git & cd prometheus-http

Install the dependencies:

go mod tidy

This will install the major dependencies we need, which is github.com/prometheus/client_golang

Exploring the CodeBase

Within the project directory, there is a single main.go file which serves as the entry point for the application:

package main


import (
   "encoding/json"
   "fmt"
   "math/rand"
   "net/http"
   "time"


   "log/slog"


   "github.com/prometheus/client_golang/prometheus"
   "github.com/prometheus/client_golang/prometheus/promhttp"
)


var (
   RequestCounter = prometheus.NewCounter(prometheus.CounterOpts{
       Name: "http_request_count",
       Help: "Total number of HTTP requests received",
   })


   RequestDuration = prometheus.NewHistogram(prometheus.HistogramOpts{
       Name:    "http_request_duration_seconds",
       Help:    "Duration of HTTP requests in seconds",
       Buckets: prometheus.LinearBuckets(0.001, 0.005, 10),
   })
)


type pingResponse struct {
   Message string `json:"message"`
}


func main() {
   // Seed the random number generator for better randomness


   http.HandleFunc("/ping", func(w http.ResponseWriter, r *http.Request) {
       slog.Info("handling new get request")
       startTime := time.Now()
       RequestCounter.Inc() // Increment request counter


       // Generate random number between 1 and 10
       randomNumber := rand.Intn(10) + 1


       var response pingResponse
       response.Message = "hello"


       if randomNumber < 5 {
           // Sleep for 3 seconds
           time.Sleep(3 * time.Second)
       }


       elapsed := time.Since(startTime)
       RequestDuration.Observe(elapsed.Seconds()) // Observe request duration


       // Set content type and encode response as JSON
       w.Header().Set("Content-Type", "application/json")
       err := json.NewEncoder(w).Encode(response)
       if err != nil {
           fmt.Println("Error encoding response:", err)
           return
       }
       slog.Info("processed request", "duration", elapsed)
   })


   // Register metrics handler
   prometheus.MustRegister(RequestCounter, RequestDuration)


   // Start HTTP server
   slog.Info("Starting server on port 8080")
   http.Handle("/metrics", promhttp.Handler())
   http.ListenAndServe(":8080", nil)
}

Here’s a quick rundown of what’s going on:

Defining Metrics

RequestCounter: This line creates a Prometheus counter named httprequestcount. Counters track the total number of events that have occurred and keep increasing. The Help string provides a description of what the metric represents.

RequestDuration: This line defines a Prometheus histogram named httprequestduration_seconds. Histograms track how many events fall within specific time ranges (buckets). Here, it measures the duration of HTTP requests in seconds. The Buckets option defines the range and number of buckets used for recording durations (0.001 seconds to 0.005 seconds with 10 buckets).

Handling Requests and Recording Metrics

The http.HandleFunc registers a function to handle requests for the ping endpoint.

Inside the handler function:

RequestCounter.Inc(): Whenever a request arrives, this line increments the httprequestcount counter by 1, signifying one more request received.

Simulating Request Duration: The snippet above generates a random number and sleeps for 3 seconds if the number is less than 5. This simulates requests with varying durations.

RequestDuration.Observe(elapsed.Seconds()): After processing the request, the elapsed time since the request started (startTime) is calculated. This duration is then passed to the Observe function of the RequestDuration histogram. This records the request duration in the appropriate bucket based on its value.

Exposing Metrics to Prometheus

prometheus.MustRegister(RequestCounter, RequestDuration): This line registers both the counter and histogram metrics with Prometheus. This makes them accessible for scraping by the Prometheus server.

Starting the Server

http.Handle("/metrics", promhttp.Handler()): This line tells the server to handle requests for the /metrics endpoint using the promhttp.Handler(). This handler is specifically designed by Prometheus to serve metrics data in the format Prometheus expects.

http.ListenAndServe(":8080", nil): Finally, the server starts listening for incoming connections on port 8080.

Containerizing the Exporter

Now that you understand what the code does, let's build a container image.

In this demo, we would be using ttl.sh, an ephemeral container registry that doesn’t require authentication, which makes it easy to use in demos such as these. In production, you’d probably want to use an internal registry or something like DockerHub to store your images.

Within the project directory, run the following commands to build and push the image:

Store the image name as an environment variable:

export IMAGE_NAME=ttl.sh/exporter-demo:1h

Build and Push the image:

docker build --push -t ttl.sh/${IMAGE_NAME}:1h .

export DOCKER_DEFAULT_PLATFORM="linux/amd64"

Deploying the Exporter

With the image built, we can finally deploy the exporter and run the following command to deploy the exporter:

cat <<EOF | kubectl apply -f -
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-exporter
spec:
  replicas: 3
  selector:
    matchLabels:
    app: custom-exporter
  template:
    metadata:
    labels:
        app: custom-exporter
    spec:
    containers:
    - name: custom-exporter
        image: ttl.sh/exporter-demo:1h
        ports:
        - name: web
        containerPort: 8080
---
kind: Service
apiVersion: v1
metadata:
  name: custom-exporter
  labels:
    app: custom-exporter
  annotations:
    prometheus.io/port: "web"
    prometheus.io/scrape: "true"
spec:
  selector:
    app: custom-exporter
  ports:
  - name: web
    port: 8080
EOF

For the most part, this is a standard Kubernetes deployment and service with a few key differences:

Deployment: The ports section within the container definition specifies a named port. Here, it's named "web" and maps the container port (8080) to a port on the pod (which might be dynamically assigned by Kubernetes).

In Service: The ports section within the Service definition also uses the same name, "web." This defines a Service port that maps to the container port through the pods selected by the Service.

The annotations on the Service tell Prometheus that the "web" port on the pods exposes the metrics endpoint, and Prometheus should scrape metrics from this Service.

Deploying The Prometheus Operator

The Prometheus operator was created to simplify the deployment of Prometheus on Kubernetes. For our use case, we can simplify the creation of Prometheus targets using the ServiceMonitor Custom resource definition(CRD).

Clone the Kube-Prometheus repo, which includes the manifests for the Prometheus operator, custom resource definitions, and Grafana for visualization.

git clone git@github.com:prometheus-operator/kube-prometheus.git  && cd kube-prometheus/

To use the version of the Prometheus operator used in this tutorial, run the following command checkout the following commit:

git checkout  65922b9fd8c3869c06686b44f5f3aa9f96560666

Create a monitoring namespace:

kubectl create ns monitoring

Deploy the Operator:

kubectl apply --server-side -f manifests/setup
kubectl wait \
    --for condition=Established \
    --all CustomResourceDefinition \
    --namespace=monitoring
kubectl apply -f manifests/

Verify the CRDs are installed:

k get crds | grep servicemonitor 

The output is similar to:

servicemonitors.monitoring.coreos.com      2024-05-26T17:40:30Z

Creating a ServiceMonitor

As mentioned earlier, ServiceMonitors ensures the Prometheus operator collects metrics from the target service. Apply the following manifest to create a service monitor that scrapes metrics from our custom exporter:

cat <<EOF | kubectl apply -f -
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: custom-exporter
  labels:
    app: custom-exporter
spec:
  selector:
    matchLabels:
    app: custom-exporter
  endpoints:
  - port: web
EOF

Using spec.selector, we tell the service monitor to scrape pods with the label app=custom-exporter.

Testing the exporter

Before querying the metrics we have, we’ll need to send some traffic to the exporter, beginning by port-forwarding the exporter using kubectl:

kubectl port-forward svc/custom-exporter 8080:8080

Next, use the following command to send a couple of requests to the service:

 for i in $(seq 50); do curl http://localhost:8080/ping; done

This will send 50 requests to the service, which is enough for the next phase, which is visualization.

You might also notice that making a request to /metrics would return a bunch of metrics that we didn’t define in the code.

Example Output:

# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 1308
promhttp_metric_handler_requests_total{code="500"} 0

The help text provides meaningful information on what each metric is for.

You can verify that Prometheus is scraping metrics from our app by viewing the targets section within the dashboard. To do this, expose Prometheus using kubectl:

kubectl port-forward svc/prometheus-k8s 9090:9090 -n monitoring

Navigate to localhost:9090/targets in your browser, and you should see the following:

Visualizing the Metrics

Now we have some metrics collected, and it’s time to visualize. Earlier on, we deployed the Prometheus operator, which also includes Grafana, which we can use to quickly visualize metrics.

Expose the Grafana dashboard:

kubectl port-forward svc/grafana 3000:3000 -n monitoring

Navigate to http://localhost:3000 in your browser, you should be greeted with a login page:

By default, the username and password is admin, soon after, you will be prompted to change it.

Creating a new dashboard

To speed things up, we will be using a dashboard that is created ahead of time. To do this, click on the hamburger menu in the top left corner of the dashboard:

Click on the dashboard, and you will be greeted with a new menu:

Click on New > Import. This should open up a new menu where you can paste or upload a JSON file:

There’s a dashboard.json file in the root of the project we cloned earlier. Run the following command to copy it to your clipboard:

On Mac:

cat dashboard.json | pbcopy 

For Linux users:

cat dashboard.json | xclip -selection clipboard

Now paste the contents of the JSON file and click on load. Your dashboard should look something like this:

Upon completing this tutorial, you might want to clean up some of the resources you provided.

Within the kube-prometheus directory, run the following command:

kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup

Deleting the cluster:

civo k3s rm exporter-demo

Wrapping Up

Metrics remain an indispensable part of the observability stack. In this tutorial, we went over how we can create custom Prometheus metrics and visualize them using Grafana and the Prometheus operator. Looking to learn more about Prometheus and Grafana? Here are some suggestions:

Check out this guide on node monitoring using Prometheus and Grafana. Grafana can also be used for end-to-end tracing. Learn how to do that in this guide.