"Git" a Handle on it! A Scalable Approach to GitOps Configuration Patterns

Good morning, everybody. I am here today to talk about some GitOps. I am not the person that you came here to see. My buddy Fred was intended to be here today got sick at the last minute. So, uh, he called me up and he said, "Dietz, we got a problem. I'm not going to be able to make it. Is there anything GitOps can do about this?" I said, "What can't GitOps do? What can't GitOps do?" I said. So today, we're going to talk about a scalable approach to GitOps configuration patterns. I'm going to do the best I can, cut me some slack.

So, we don't have very much time and with the technical difficulties this morning, perhaps even a little bit less. But in that, I'm a substitute, A-Okay in my book, that sounds great. So let's talk about GitOps. GitOps is one of my favorite topics. It's one of my favorite disciplines that I've ever stumbled upon. First, you know, I've been in CI DevOps space forever and I thought that I knew everything until I stumbled upon Kubernetes.

Kubernetes was an obvious game-changer. Immediately, Kubernetes, the Kubernetes manifest, like we were talking about in the talk before having that stable declaration of what you want running in your Kubernetes cluster, and the ability to trust that that is what Kubernetes is going to make run in that cluster, and having that well-structured, manifest where you can establish your, your configurations, how you're going to scale, etc. was just extremely eye-opening.

And I thought to myself, "Well, that's such a powerful technology, albeit with a touch of a learning curve, but it's such a powerful technology, I'm going to build a company around it." So, I've founded this company, Kubefirst and about one year into building Kubefirst, this instant Kubernetes cloud-native platform that you run from a single command, about a year into this adventure, my co-founder and I stumbled upon GitOps, and it wrecked us. I mean, it destroyed us because we weren't building our platform on GitOps, and GitOps was clearly going to be the winner.

It clearly. What GitOps does is, it marries your Git provider with your Kubernetes engine in a way where you get to let go of the CD part of CI/CD in a really, really elegant way, using only the tools that your engineers are already using. And it's extremely simple from an architecture standpoint, it's extremely powerful from a scalability standpoint, sets you up very well for disaster recovery.

So let's get into it a little bit. This story is not new, right? You've got your application, runs on your laptop, you need it to run elsewhere. You're going to keep your application source code in a Git repository, and you're going to deliver it. In the old days, circa two years ago, it used to be that your pipeline was a CI/CD pipeline, right? We always lump those letters together, CI/CI/CD. There was never a strong separation between CI and CD.

So, you get your application code and before you know it, your application evolves a little bit. It needs some place to run. Maybe it needs a back-end RDS database, maybe needs an S3 bucket, whatever. So you have to establish some IAC, some infrastructure as code. That infrastructure as code is needed in order for your application to run successfully. So before you know it, you have a couple versions of your application, and then you need some configuration somewhere, right?

All of this is not a new problem. This is not new to GitOps, this is just comp-sci 101. And this source of truth that you have to establish in your Git repository has to drive the end result of what's delivered throughout your infrastructure ecosystem. GitOps doesn't change any of that. Well, GitOps changes is the separation of CI and CD, in such a way where you're now responsible as a, as a DevOps engineer, in establishing that source of truth, that desired state, that your application, the pipeline to build the application and package it, and define where you want it deployed, that's now your job.

CD, as a DevOps engineer, is now the responsibility of your GitOps engine. Now that could be Argo CD, that could be Flux CD, there's a couple others that are getting into the game, GitLab, etc. Um, what's important about this GitOps engine is, architecturally, it's not a part of your CI/CD pipeline. It's an engine that lives in a cluster, and it's always pulling the desired state from a Git repository.

So, from a security standpoint, it's very, very powerful because when you have a whole bunch of different clusters, and you have a CI ecosystem, probably in a management cluster, that CI ecosystem is an attack vector for bad actors. They want to get in, they want to be able to gain access to all the things that your management CI/CD ecosystem has access to, so that they can infiltrate your system. With GitOps, the game changes because it's a read-only connection to a Git repository in order to establish what's delivered everywhere, and because of that, the game changes to one where your new role, from a security standpoint, is to protect that main branch of that Git repository.

So, that's GitOps in a nutshell. There's a lot to like about GitOps. I love GitOps. I'm sure some of you in the room probably already do. Show of hands, is anybody using GitOps in production today? Yeah, awesome! GitOps is like a game-changer. Great security posture, great disaster recovery posture. If you have your applications, every single thing that's deployed everywhere, if you have that in a main branch in a GitOps repository, and your cluster goes belly up in the middle of the night, now you need a new cluster, and everything that was in that cluster needs to get back in this brand-new version of that cluster. Being able to bootstrap that cluster against the exact same Git repository, get entry point for the tree of applications that are supposed to be delivered to that cluster, is significantly easier than the script ops paradigm.

The script ops paradigm being that you had some CI/CD workflows, and in order to redeliver that application to your brand-new production cluster, you're gonna have to run a script, you're gonna have to run a job, you're going to have to click a go button, hopefully the developers haven't messed up the pipelines while they're working in development and they didn't realize that the production cluster is going to fall over in the middle of the night. GitOps sets you up so that you have those high nines.

It's not without its challenges. There are a handful of challenges that come with GitOps, but all things considered, they are significantly smaller than the challenges that you have without GitOps. The challenges that you'll run into, you know, interacting with Git. Shouldn't act like that's a given but most of your engineers are going to be familiar with using Git. If not, that's a bit of a learning curve for you. Infrastructure as code and making infrastructure code part of your application delivery workflow is a little bit of a shift from the old DevOps. You know, I have an Ops team, they're managing the infrastructure as code. Then we have a development team, they're managing applications. GitOps has this neat way of bridging those two ecosystems together in a very clean way because everything's predicated on that main branch of that GitOps repository.

There are no real standards, I guess you could say, in GitOps. There's a lot of patterns that you can follow. The closest thing to a standard that we have is the Open GitOps group. It's a working group with the CNCF, a bunch of great people working in that work group. And their responsibility is trying to, in a vendor-agnostic way, define the discipline of GitOps. So that when you're evaluating tools, when you're evaluating what your GitOps engine is going to be, you're looking at it with the mindset of a GitOps ecosystem and vendor-agnostic way.

The first principle is that it needs to be declarative. Now, generally speaking, this basically means that you just need to have the desired state, that source of truth of what you want, in a place that doesn't change. So, typically speaking, we're talking about a main branch of a GitOps repository.

Number two principle is that it needs to be versioned and immutable. Probably goes mostly without saying, Git takes care of a lot of that for you. What it doesn't take care of for you is the process of defining the versions of the packaged applications that you have. I'm sure a lot of you have Helm charts. Some of you are probably using manifests with customized overlays. One way or another, you're going to have to build a container, you're going to have to publish that container somewhere. That container has to have a version associated with it, and that version should never mutate. So concepts of older days, you know, you build a Docker container and you tag it with 'latest' was a very common standard. Do not use anything like 'latest' in a GitOps ecosystem. It's going to set you up for a bad time. Everything has to have a strong version, and that version of that application of that Helm chart must be immutable.

The GitOps engine, what makes GitOps so powerful and from a security standpoint, what gives it that nice security posture is the fact that there's a GitOps engine that's always pulling from your desired state and it never stops. Anybody who are already familiar with Kubernetes, understands the relentlessness of that loop of Kubernetes trying to make the desired state your actual state. You go through it every time you deploy an application, but you messed up the image and it can't resolve that image. It's going to keep trying. It'll be in that back-off loop for a while, but Kubernetes does not quit on you. And that coupling of that loop with the power of a Git provider, it's just an incredible combination of tools.

It's distributed very well, it comes with everything that Git gives you out of the box in terms of understanding who changed a pile, why the file changed. You know, these are basic pull request methodologies that we're all very familiar with. Super cool for GitOps.

And then, lastly, a GitOps engine has to continuously reconcile. So, much like Kubernetes, if there's divergence between your desired state and your actual state that's in Kubernetes, your GitOps engine needs to continue to try to work to make your desired state the actual state.

So, let's talk about some patterns. Mono repo versus poly repo, let's get these terms on the table real quick. So, mono repo is when you have one repository that has a whole bunch of stuff in it, a bunch of different applications is the typical pattern. You know, you've got an app that you, you've got a whole bunch of developers working on, and it is comprised of three different microservices. Think about a Git repository with three subfolders, one for each of those applications. That's all we're talking about with the mono repo. Typically speaking, that means that your application CI/CD is associated with that one Git repository. No big deal. Separate from that is the poly repo pattern where, instead of putting all three of those microservice applications into a single Git repository, you're going to break them up into micro repositories, and then each one's going to get its own CI/CD Pipeline. And that's poly repo pattern.

Now, with GitOps, there's a new repository that's in play. Historically, in your script ops paradigms, you've had your application source code and the instruction for how to build that source code and the instruction on how to deliver that source code in your CI/CD pipeline for that given application. In GitOps, generally speaking, it's best practice to push that source of truth about what's being deployed and sever that from all the other repos, separate from your application repos, and establish your GitOps repository that represents the system. Now, the question is, what is the system and, and at what point should you break that up? My experience has been that it's best to always, always try and fight to keep that repository, a single GitOps repository that everybody has to buy into. Now, some organizations that have multiple teams that work in isolation from each other, there could be use cases where it's relevant to have multiple GitOps repositories for multiple spaces of your organization. And that's case by case. There's no silver bullet there.

So, repository structures. Let's, uh, let's take a look real quick. I'm gonna, And let's, uh, bring up a monocle.

I'm gonna have to kill my double screen for a second. I don't think I can do that with my head turned. Sorry, I'm not sure what's going on there. Got something adds the screen recording. Give me one second. Yeah, could I, buddy?

So, here we're going to bring up Monocle. Monocle is an open source tool. I work for a company, Kubeshop. They've got a handful of open source projects. Monocle is a manifest IDE for Kubernetes that we're going to use to show a couple of things about the directory structure. And I'm going to open our GitOps repository. So, this environment that you're going to be looking at here is a Kubefirst local environment. You can, it's a free instant platform that you can use for demos and stuff. That's pretty neat. And the GitOps repository that we're going to be looking at today has a registry of tools that have been delivered to our Kubernetes cluster.

Here, let me, uh, bring that up real quick. So we have, uh, GitOps repository is powered by Argo CD. Argo CD is a GitOps engine. So the way that GitOps works is basically you have a registry that you bind, or registry being a, a directory in a Git repository. And when you bootstrap your cluster, you point your GitOps engine at this directory and it hydrates the cluster with whatever your instruction is in that tree in that GitOps repository. So let me show you an example.

This is a registry Argo CD application that we create. And it is bound to, um, where's my registry? Here it is, it's bound to this directory in our GitOps Git repository. So when we see here, for example, that cert-manager is running, let's see, this is pointing at the GitOps repository, a component, cert-manager. So let's get a components, cert-manager. And here we can see we have this cert-manager application, and it's running on an older version. It's on version 1.9.1. And just earlier, I think it was the middle of the last month, they dropped a 1.11.0. So I'm going to show you, basically, like what it looks like to GitOps changes to your infrastructure.

Basically, all we need to do is take this content and commit it, stage it, commit.

And I'll push that up. And then when I push that up, now my GitOps engine is running in my cluster, on my local host right here. And it's continuously watching that GitHub repository. And if we had all the time in the world, which we don't, I would just let that run for about three minutes and it would, I would reconcile this change. I'm going to expedite that process by hitting this refresh button here, and then go to my cert-manager components, and repress there. And you'll see it jump from green to yellow. And that's because this new cert-manager version is being delivered.

It recognized that there was divergence between what I have in my desired state in my main branch in my GitOps repository from what is actually deployed in my local K3D cluster. And it says, if there's divergence, I've got to do something about it. So, Argo CD is going to start working and trying to reconcile that change and make it into the new version. And, you know, whatever the configuration changes, whatever the secrets changes are, it'll reconcile those changes and turn this into a new version of the application. And here you can see it's just about deployed, and now it's fully synced.

So, that process right there that you just watched me go through is basically all it takes to deliver changes. But because the driving force was to deliver it by setting the desired state in Git, it sets you up really nicely for a whole bunch of things like Disaster Recovery. If someone were to accidentally be in your Kubernetes cluster and they kube cuddle deleted some content that you need in order to have your production environment running, your GitOps engine is going to see that, recognize divergence, and immediately reconcile and say, "Hey, no way. There's supposed to be a config map. There's supposed to be a deployment. There's supposed to be all these things." And it'll put your application right back. One second after you delete it, GitOps is going to put it right back. So, from a disaster recovery standpoint, it sets you up really well to keep your nines high.

From an application delivery standpoint, we only talked about this for a quick second, but if I can go to, let's get a development real quick. So, if I go Applications, and I swear this is my last 60 seconds. So I'm going to get a development. I have this Metaphor Front End application here and just to sort of show what I'm describing, if I delete this metaphor front end development application manually, it's going to get rid of it. But then, voila, it's immediately right back. It's right back because of the power of GitOps, because it's driven from the desired state of my main branch in my GitOps repository. So that's pretty much the show. I ran out of minutes, um, but if you guys want to talk more about GitOps, about Monocle, the Kubernetes IDE, if you want to talk about Kubefirst etc., stop by afterward and I'll chat with each of you.

I wanted to give a little bit of credit real quick. Open GitOps Project, this is how you can find them. They're great at giving you that vendor-agnostic, view on the GitOps discipline. Helm package delivery, Kustomize package delivery being able to drive the configurations for multiple applications of your environment.

Kubefirst, we can install instant cloud-native platforms like the one you were looking at today. Monocle itself, Visual Studio or on Kubernetes IDE and we don't have time for questions. I will be giving a talk tomorrow. My buddy can't be with me but I'll be at 12:30, talking about the eight steps that it takes to build out a cloud-native platform from scratch. Super cool talk and I hope to see you guys there. Thank you so much.