Manage Kubernetes clusters using the Civo Pulumi provider

Pulumi is an open-source infrastructure-as-code tool that enables us to define our cloud infrastructure using modern programming languages. It supports many programming languages: JavaScript / TypeScript, Python, Go and the DotNet ecosystem (C#, F# and VB.NET). As I'm most familiar with TypeScript that is the language we will use throughout this guide.

The full source code for the examples below were added to the Civo Pulumi provider repository on GitHub. For easier readability only the important parts of the code will be shown in the code-snippets.

Prerequisites

For Pulumi to work we need to install two things: the language runtime of our choice and the Pulumi CLI. For TypeScript we need to install Node.js from its download page, or alternatively on Mac you can use Homebrew (for the current LTS version):

brew install node@12

Next we can install the Pulumi CLI as described on their installation page. On Mac with Homebrew:

brew install pulumi

Creating a new Pulumi project

The Pulumi CLI can be used to create a new Pulumi project - all the examples we will see were originally created with this feature. To create a new project run the following in an empty directory and answer the questions on screen:

pulumi new kubernetes-typescript

This will generate a Pulumi project for Kubernetes cluster creation, using TypeScript as the language. It also sets up a stack - an instance of our project (you can reuse the same project multiple times, each independent deployment will have a separate stack).

In the following examples I will point out the changes that are necessary to make to the project in order to use the Civo Pulumi provider.

Part I: Hello Civo-Pulumi!

The first example will show how to create a Kubernetes cluster with Civo Pulumi provider. The full project is available here.

There are two main changes from the CLI generated Kubernetes project: The Civo Pulumi provider needs to be added to the package.json file using npm install and the index.js file needs to be modified to create a new cluster with this provider, as follows:

npm install @pulumi/civo

# index.js

import * as k8s from "@pulumi/kubernetes";
import * as civo from "@pulumi/civo";

const cluster = new civo.KubernetesCluster("acc-test", {
    tags: "demo-kubernetes-typescript",
});

const k8sProvider = new k8s.Provider("acc-provider-test", {
    kubeconfig: cluster.kubeconfig,
});

const appLabels = { app: "nginx" };
const deployment = new k8s.apps.v1.Deployment("nginx", {
    spec: {
        selector: { matchLabels: appLabels },
        replicas: 1,
        template: {
            metadata: { labels: appLabels },
            spec: { containers: [{ name: "nginx", image: "nginx" }] }
        }
    }
}, {
    provider: k8sProvider,
});

export const clusterName = cluster.name;

Warning: Make sure that you use the provider we configured with the Civo cluster's kubeconfig field, when creating a Kubernetes resource. You can do this by specifying provider: k8sProvider in the options. Otherwise Pulumi will deploy the resource (in our case the Nginx Deployment) to a Kubernetes cluster by reading the local KUBECONFIG environment variable, instead of the new Civo cluster.

Tip: we use export to create an output - these will be displayed on the console and also will be saved into our stack so they can be retrieved later. In the example above we export the name of the cluster, which we will use with the Civo CLI.

This Pulumi application will create a Kubernetes cluster in Civo cloud with default parameters (3 small nodes (1 CPU, 2GB RAM) and the default Kubernetes version). The default applications: Traefik and Metrics Server will be also deployed.

Before we can actually create the cluster, we have to configure the API Key for Civo. You can find your API Key on your account security page:

Configure Civo API Key through Pulumi configuration

In the project's root directory, run:

pulumi config set civo:token --secret <YOUR_API_KEY_FROM_CIVO_PAGE>

This will save the api key under the configuration key civo:token. The --secret option ensures that the value is stored encrypted.

Configure Civo API Key through environment variable

The other way to configure the Civo API Key is to set the environment variable CIVO_TOKEN:

export CIVO_TOKEN=<YOUR_API_KEY_FROM_CIVO_PAGE>

Let's create our cluster

We can run Pulumi's preview feature to see what would be created for us:

pulumi preview

and then actually create the cluster:

pulumi up

In a couple of minutes we will have our brand new cluster with Nginx installed in it.

The name of the cluster was exported as an output, and now we can use the Civo CLI to interact with our cluster. For example we could save the kubeconfig file and start using it:

$ civo kubernetes config acc-test-cd059ce > ./kubeconfig.civo
$ export KUBECONFIG=./kubeconfig.civo

Once we are done experimenting with this cluster, it is time to delete it:

pulumi destroy

Part II: Deeper dive

In this second example we will dive a bit deeper into what is possible with Pulumi: we will see how we can reuse our already existing Kubernetes YAML files and how we can combine our programming language's libraries with Pulumi's state representation objects. The full project is available here. This is what it will do when applied:

We are going to use the Open Source Ambassador Api Gateway as our Ingress controller, and for this we will turn their YAML based installation description into a Pulumi program.

As a first step we copied the YAML files from the Ambassador guide linked above into a new ambasssador-yaml directory, and then - unlike in the first example - we set a specific configuration for the new Civo Kubernetes cluster:

const cluster = new civo.KubernetesCluster("acc-test", {
    applications: "-traefik",
    kubernetesVersion: "1.18.6+k3s1",
    targetNodesSize: "g2.medium",
    numTargetNodes: 4,
    tags: "demo-kubernetes-typescript"
});

const k8sProvider = new k8s.Provider("acc-provider-test", {
    kubeconfig: cluster.kubeconfig,
});

This configuration disables Traefik and uses 4 medium sized nodes. We also specify the exact version of Kubernetes we want to use.

Once the cluster is ready and the Kubernetes provider is configured, we will use Pulumi's k8s.yaml.ConfigGroup object to deploy Ambassador's CRDs, then we create a new namespace and finally deploy Ambassador into this namespace:

const ambassadorCrdCG = new k8s.yaml.ConfigGroup("ambassador-crd-manifests",
    {
        files: "ambassador-yaml/crd.yaml"
    },
    {
        provider: k8sProvider,
    },
);

const ambassadorNamespace = new k8s.core.v1.Namespace("ambassador-namespace",
    {
        metadata: {
            name: "ambassador"
        },
    },
    {
        provider: k8sProvider,
        dependsOn: [ ambassadorCrdCG ],
    },
);

const ambassadorCG = new k8s.yaml.ConfigGroup("ambassador-manifests",
    {
        files: [
            "ambassador-yaml/ambassador-rbac.yaml",
            "ambassador-yaml/ambassador-service.yaml",
            "ambassador-yaml/ambassador-config.yaml",
        ],
    },
    {
        provider: k8sProvider,
        dependsOn: [ ambassadorCrdCG, ambassadorNamespace ],
    }
);

Tip: we use the dependsOn option to ensure that Pulumi fully deploys a given resource before the dependents are deployed. While Pulumi can detect many dependencies on its own, in this case we have to explicitly tell that the main deployment can proceed only once all the CRDs and the namespace is created.

When everything is deployed, we want to show the user what the public IP address of the Ambassador ingress is. We are going to programmatically look for Ambassador's Service object and then transform this object into an output that will be displayed once we run pulumi up:

const ambassadorService = ambassadorCG.getResource("v1/Service", "ambassador/ambassador");

export const ingressIp = ambassadorService.apply(
    service => service.status.apply(
        status => status.loadBalancer.ingress.map(function (ingress) {
            return ingress.ip;
        })
    )
);

Tip: we have to use apply when we want to manipulate any Output<T> type object, because these values are only known at deployment time, once the cluster and our Kubernetes resources are created.

After the Ambassador deployment we are going to deploy an echo server from the Cilium project, and expose it to the /echo URL endpoint. To keep this article's length reasonable this part of the code will not be shown here, but it's part of the example project you downloaded earlier. Instead we will jump to the part where we automate the creation of the kubeconfig file, to demonstrate how easy it is to combine Pulumi objects with normal code libraries:

export const kubeConfigFileName = pulumi.all([cluster.kubeconfig, clusterName]).apply(([config, name]) => {
   const kubeConfigName = "kubeconfig." + pulumi.getStack();
   fs.writeFileSync(kubeConfigName, config, "utf-8");

   return kubeConfigName;
});

Tip: we can use pulumi.all when we want to manipulate more than one Output<T> objects at once.

Finally we will create a clickable link, in our output, that points to our echo service:

export const echoUrl = ingressIp.apply(ip => {
   return "http://" + ip + "/echo"
});

Further Exploration

Pulumi has a large number of features, and this post can't possibly cover even a fraction of them. The third example project demonstrates some of my favorites - here are a couple of those in no particular order.

Helm support

Besides working with Kubernetes YAML files directly, Pulumi supports both Helm v2 and v3.

let configValues = {
    accessKey: minioConfig.accessKey,
    secretKey: minioConfig.secretKey,
    ...
}

const minioChart = new k8s.helm.v2.Chart("minio",
    {
        chart: "minio",
        version: "6.0.2",
        namespace: "minio",
        values: configValues,
        fetchOpts: {
            repo: "https://helm.min.io/"
        }
    },
    {
        parent: this,
        dependsOn: [ minioNamespace ]
    },
);

This would install Minio, fetching the chart directly from the repository.

Transformations

This feature makes it possible to apply dynamic changes to Kubernetes YAML files, or even Helm charts. In our example below we use it to disable Pulumi's await logic on certain Kubernetes resources. It can be also used to fix problems with charts without the need to fork the chart (e.g.: missing namespace definitions on resources).

const minioChart = new k8s.helm.v2.Chart("minio",
    {
        chart: "minio",
        ...
        transformations: [
            (obj: any) => {
                if (!minioConfig.persistenceStorageClassInstalled) {
                    if ((obj.kind == "PersistentVolumeClaim") || (obj.kind == "Service") || (obj.kind == "Deployment")){
                        obj.metadata.annotations = {
                            "pulumi.com/skipAwait": "true"
                        };
                    }
                }
            }
        ]
    },
    {
        parent: this,
        dependsOn: [ minioNamespace ]
    },
);

Tip: transformations doesn't yet support the removal of a resource directly, however this small trick effectively removes a given - in this case Ingress - resource:

transformations: [
    (obj: any) => {
        if (obj.kind == "Ingress") {
            obj.apiVersion = "v1";
            obj.kind = "List";
            obj.items = [];
        }
    }
]

This idea originates from the Pulumi Slack that I can highly recommend.

Custom components

We can encapsulate our installation logic for certain tools / functionality into their standalone reusable components. This can serve as a nice alternative for Helm charts (and it can even use Helm charts internally!). In the third example both the Minio and the Ambassador installation was packaged in their own components, making the index.ts file nice and tidy.

Conclusion

I hope that this guide made you interested in trying out Pulumi and the Civo Pulumi provider when you are working on your next Kubernetes cluster setup. Thank you for your attention and have a great day exploring!