The State of Modern Backup
"We have a backup, right?"
This is a question that few people in IT want to hear and probably fewer want to answer! Backups are a cornerstone to any IT infrastructure. With the introduction of cloud computing and technologies like containerisation and Kubernetes it is easy to be lulled into a false sense of security with backups... "yeah it's in the cloud, it's backed up there".
Major incidents happen, even with cloud providers. We only have to look to the recent incident with OVH to see this. Also, not all businesses have the budget or skills to deploy high availability services and disaster recovery (DR) plans, and in fact for most companies, doing so would not be cost effective.
Therefore there is always a place in the IT infrastructure landscape for solid backup and recovery tools. Kubernetes has been one of the most important developments to IT infrastructure in recent years: the maturity of the product and the way it's delivered via cloud providers has made it an extremely accessible and attractive technology. However, due to the nature of Kubernetes and the concept of immutable containers along with the movement to paradigms such as infrastructure as code, it feels like backup solutions have come as an afterthought in the Kubernetes world, or are overly complicated. This is not a new experience in the industry: the move from physical server backups to virtual machine backup took a similar path, not to mention the process of moving backup to the cloud.
Kasten 10
When a solution comes along which is claiming to solve all these issues, and as a kicker comes with the weight of a known leader in the backup space (Veeam) it definitely gets my attention. Kasten K10 (by Veeam) is making big waves in Kubernetes circles and I thought I needed to get my hands on it! Plus it helps that they have a free version that we can try out!
I will be testing the product on the Civo Kubernetes platform. If you've not yet set up an account on Civo, you can do that here and get $250 free credit! I'll wait.
Setting up a cluster in Civo is beyond the scope of this article, there is an excellent doc on how to do this here, but because it's so quick i'll include some commands!
Tools required:
Create new cluster
This will create a new 1 node medium cluster, save the config and merge to my kubeconfig file, handy right! I have purposely only provisioned a single node, you can expand the number of nodes at a later date
civo k3s create -s g3.k3s.medium -n 1 --wait --save --merge
Creating a 3 node k3s cluster of g3.k3s.medium instances called ancient-deer... -
You should see something like this
Merged with main kubernetes config: /Users/keithhubner/.kube/config
Access your cluster with:
kubectl config use-context ancient-deer
kubectl get node
The cluster ancient-deer (c9c480bc-9b6e-4512-bb03-266df50db261) has been created in 2 min 24 sec
Let's quickly install a basic app from the civo marketplace, Ghost:
civo k3s app add Ghost:5GB --cluster ancient-deer
Switch to the new cluster:
kubectx ancient-deer
Check it has been installed properly:
kubectl get pods -n ghost
You will see something like:
NAME READY STATUS RESTARTS AGE
ghost-blog-868bc76558-kgs5v 1/1 Running 0 4m11s
Setting up and using Ghost is beyond the scope of this document, but I recommend you go through the setup and add a test page. For details on how to access the UI, refer to the quick start guide with the marketplace app.
Deploying Kasten10
Once you have an application installed it's time to deploy Kasten K10. The Kasten documentation is really simple to follow and there are various options available.
For this guide i've taken the snippets relevant to deploying it via helm onto Civo Kubernetes. For further instructions and deployment options please see the Kasten documentation.
Deploying with Helm:
helm repo add kasten https://charts.kasten.io/
kubectl create namespace kasten-io
Currently the storage platform on Civo does not support snapshots, therefore we must deploy K10 to use generic storage backup.
I've purposely installed Kasten K10 after installing the app as this is likely to be the position you are in with current workloads where you will want to add a backup solution you are already running.
As documented in the K10 guide, we will use the sidecar option to enable us to backup volumes in Civo. Other options are available so please review the documentation if this is not your setup.
I will also specify that the namespace must include the label selector, otherwise the sidecar will look to inject on all namespaces, something you may not want.
helm install k10 kasten/k10 --namespace=kasten-io --set injectKanisterSidecar.enabled=true \
--set-string injectKanisterSidecar.namespaceSelector.matchLabels.k10/injectKanisterSidecar=true
Keep en eye on the pod creation (will take a few mins so grab a tea/coffee/other):
kubectl get pods -n kasten-io
As we already have a running app, we need to label the namespace to ensure the sidecar is deployed:
kubectl label namespace ghost k10/injectKanisterSidecar=true
We also need to ensure that generic backup is enforced on the deployment, this will force the re-creation of the pod:
kubectl annotate deployment ghost-blog k10.kasten.io/forcegenericbackup="true" --namespace=ghost
This next part might not be needed, depending on your use case:
In order to inject the sidecar we need to scale the deployment down and back up again. There are other ways to inject the sidecar but because this is a new app and not being used for anything else at the moment, this is the fastest way. If you have running apps in production, please refer to the Kasten documentation on what the best route to take is.
scale deployment down:
kubectl scale deployment ghost-blog --replicas=0 -n ghost
scale deployment up:
kubectl scale deployment ghost-blog --replicas=1 -n ghost
This will then re-create the pods and include the sidecar container in the pod.
If you inspect the details on the pod you should see the sidecar has been injected.
Once all the pods are running, let's access the GUI. We do that by port-forwarding to our cluster:
kubectl --namespace kasten-io port-forward service/gateway 8080:8000
Then you can then open the GUI at http://127.0.0.1:8080/k10/#/
You will be taken to the dashboard and the first thing you will want to do is create a location profile. Go to settings and location profile:
You should now create a new location profile based on where you want to store your backups. For the purpose of this demo I have spun up a Minio endpoint:
It's so easy i'll just add it here:
civo k3s application add MinIO:5GB --cluster ancient-deer
For more information and help on setting up locations please see the official K10 documentation on locations
Once you have configured MinIO or another location, we can go and setup our policy.
Navigate back to the dashboard and click create policy.
You can now create a new policy to suit your backup requirements. In this example I am backing up every hour and including the default namespace. There are many more options here so again please review the official guides for this.
Once you have a policy you can kick off a manual backup and put your feet up!
Once the backup has complete and been running for a while, you should see some nice successful backups!
Breaking stuff
OK now let's cause some havoc!
kubectl delete ns ghost
Whoops! Let's do a restore: head back to the Kasten GUI, and click on applications. Because Kasten can no longer see the app you will need to use the filter to select removed applications:
You can then select restore on the ghost app.
It should be that simple to get your application back up and running!
I hope this guide has been useful and given you a chance explore the capability of Kasten K10. If you have any questions please don't hesiate to contact me on twitter and join the Civo community Slack.
