Pitch the Password: Securing Access to Cloud Native Resources with Certificates

Good afternoon, everyone. I think we're right around that time, so I'm going to kick off. Thank you for attending. Like any conference, at the end of the conference you're tired, your brain hurts... Uh, yeah, I know that feeling. But I appreciate you all taking a few minutes to join. This is the session that, if you inadvertently stumbled in, or I appreciate you being here or pitching the password: Securing Access to Cloud Native Resources using Short-Lived Certificates. Not pitch as in, "Hey, passwords are good," but pitching as in, we're getting rid of them. So that's the idea.

A little bit about me, my name is Alan Valancourt. I'm a Senior Solutions Engineer with Teleport. I live in South Carolina, originally from Florida, it's my home state. So I've been an OG Bucks fan since the orange and white days of, you know, Vinnie Testaverdi and all that. So, a picture of my family when we watched the game at the Mercedes-Benz Stadium a couple years ago down in Atlanta when it first opened up. Back in the day when Jameis Winston was, you know, you never knew what you're gonna get.

I collect board games. That's probably my current collection on Board Game Geek. I also collect speed cubes. I think most of us that are in kind of sort of technology, we tend to be collectors of something or another. And collect passwords as well, but hopefully we'll get rid of it.

For those that have seen Speed, or haven't, there's a scene in there where Dennis Hopper is like, "Hey, Pop Quiz, Hot Shot!" He's talking to Keanu Reeves trying to tell him where the bomb was on the bus. And uh, so we're gonna take this real quick pop quiz and see if you all know what that answer might be. So, the greatest security threat to your organization comes from hackers, insider threats, or credentials like keys and passwords. Anyone care to throw an answer out?

Three, you are correct, right! So that is, you know, kind of a loaded question there as well. Right, so credentials like keys and passwords are a number of ways in fact. Thus those that were on the Defense.com keynote earlier today, he's been talking about that and how they, how they hack in and get in and escalate once they're within an environment. So, this comes from the Verizon Data Breach Investigation Report 2022. The use of stolen credentials is the number one cause for breaches.

So, hacks start the same. They, you know, human error and this is a marketing slide so it's got, you know, a little bit of market flair. I'm a tech guy, so definitely not as nice as this. You know, human error, a leaked password, secrets left in code because none of us ever do that, and then, you know, they gain a foothold then they pivot and then move from one system to the next. And that was the key point of the earlier talk today, talking about how, you know, breaches happen. They get in, they pivot, and so you know, when they, when that happens privileges get escalated, the blast radius grows, and then of course business aspects of slowing business to a crawl.

So, as organizations scale, secrets multiply. For those of you that work in organizations and even your own home websites, passwords, those secrets, just there's no ending of them. Probability of human error as well increases exponentially. So the question is, how do we go fast, how do we stay secure, and that's what we're going to spend the rest of our time talking about here today.

So, hopefully this is not you, right? You know, "I love passwords. Change my mind." because I'm in a business that we're trying to help customers solve this problem, we still encounter those that like their passwords.

Anyone remember this incident from 2018? That was, you know, big news, right? They were all, you know Hawaii Emergency Management and they're all talking about something happened. But then people zoomed in and realized that, hey, there's a post-it note with a password. And of course, they said that was an old password not being used. But the fact is, it's on a post-it note. We've all... We've never seen that. I probably shouldn't have to say a whole lot more about this, right, this is, this has been on the news. I migrated away from LastPass like eight, nine years ago to Bitwarden and, you know, for my personal stuff. But just seeing that just makes you shudder.

Here's another statistic: Stolen and misused SSH keys. How many here use SSH on a daily, weekly, monthly basis? How many of you are using keys to access those resources today? Okay, so now you take yourself, multiply by your co-workers that are also accessing these resources, multiply by the hundreds, dozens of servers and also you see that there could be a lot of keys that could be in any environment at any given time. And this came from SSH.com.

So if we're being honest, we all use passwords, right? Our apps use passwords, our databases use passwords, our Cloud-native resources use passwords. It's hard to change what has worked so well. It's why everyone raised their hands, like "Yeah, I still use SSH keys. I used one this morning in fact to access a broken EC2 instance." But a lot of times, people don't realize that things like API keys are also considered a password. SSH key can be considered a password, although some you might argue against that. But the key is, we don't necessarily call them passwords, we call them credentials, you know, we call them secrets. When you go to AWS or a big cloud and you're downloading, they're like generating that PEM key, they're like "Save it because there's no other record for it. Once it's downloaded, guess what? And you store that in your config file, you're storing a password, in essence." So, really the same thing is what it comes down to. And one of our folks at our company wrote a blog about talking a little about why API keys are passwords, along that line.

In the pre-Cloud days, for those of us that are old school, I spent 16, 17 years in the production world, you know, managing resources pre-Cloud, pre-AWS, on-prem data centers, the whole nine yards. It was easy, we had a corporate firewall or we had a VPN. Everybody logged in pre-work from home days and everyone was trusted, even though you had printers with SMB shares out there. Right? It was that perimeter idea of perimeter security. It was, it worked for the most part. But then as the cloud started growing, you know, that model stayed the same. It didn't evolve a whole lot with that.

So even internally, we shared passwords, sticky notes, shared password database. In fact, the last production place I was at, this is actually what we used. We used KeePass over Dropbox to share among 30 some developers and engineers. So, we all had it. And a year and a half, two years later after I left the company, I realized I still had that share in Dropbox and I'm like "Oh, I can still see all these passwords and none of them were changed."

So, you know, every server has a public key on it, whether or not they needed access. I used to write Terraform or Ansible playbooks and whenever a new engineer came on board, I'd say, "Hey, just give me your public key." I ran it and it just got deployed on all the servers, whether or not they needed it. On logging in, everyone used Ubuntu or root or Windows Administrator. Right? Databases, we just all had one or two users. It was just simple to manage that way. Just use that. We trusted one another. Of course, we probably also have stories of why that didn't work.

Like one of, I had a developer come to me one time said, "Hey, do we have a backup of this database?". I'm like, "Yeah, why?". He goes, "I just accidentally made a change to a production table. I changed 60,000 rows for Verizon Wireless on a production server middle of the day. Lost 15 minutes of data." But you know, we were just using generic username, password. No audit, no capabilities, best practices back then? Yes. You know, it's kind of what we all did. Today, definitely not so. Don't be surprised if you, or the companies you're with, or those that you know, still use the same types of authentication access management. Except today, we're getting away from that on-prem or using all the clouds, whether it's Civo, AWS, Google, Azure, you name it. The idea of perimeter security doesn't apply in today's world, so we have to evolve and change how we access it.

Cloud-native security is a lot different because you have this huge platform, you have APIs, you have microservices, you have all these little pieces and they're all segmented. If you just use something by default because the job of AWS or Civo or Google is not to secure your stuff, they're just like, "Here's your platform. Here's it's a pass, go have fun." But if you don't know what you're doing, you're going to mess something up, which we see.

The other thing is we get challenges with passwords and password management. So this article, just from January 18, talks about this company here that, "We have no choice, we have to use password managers until passwords are dead." Yikes! Or adult or whatever it is, you know, just thinking through that. Really cool blog from Palantir that came out recently. And this is part one and they have part two and part three of that. Definitely recommend that you all check it out. But they talk about almost a 24-month journey for them as an organization to go away from passwords to go toward a passwordless architecture that involved what they needed to do. Some of it was the lack of 502, lack of technology adoptions, but just the logistical, the planning, I mean just, it's really good, really good. I definitely recommend reading that for sure.

Then you got this MFA fatigue, right? We get it for everything. Pull up your phone, put in this code. In fact, there's been some recent, that link there talks about recently where from the Russian targeting government business by the threat actor taking advantage of this, issuing these multiple requests and then after a while you just ignore things. You assume it's legit and you enter it in, or something like that. So, you know, you've got all these pieces that trigger but still make it complex. Simplifies it or makes it more complex, you know, however we figure that out.

So anyways, enough of the hum. The point why you're here today, right, is to find out what's a better way and that's using identity native access. So you're probably thinking, "What is identity native?" It's the idea of secretless plus zero trust capabilities. Not leveraging passwords, not using legacy PAM tools, no key vaults even though they're there, no password vaults. Identity-based access, because that's a huge key component of that. Using your identity, any location, without VPNs or, you know, whitelisted IPs or things along that line.

So I was playing around with ChatGPT and threw it in there. I said, "Hey, what does it say about identity native?" Even with a typographical error that it let me know, you know what it is. It's that approach to access management and security that prioritizes the use of digital identity information. So your username, or password, biometric data, secure digital certificate, etc., and providing a secure, convenient way to identify and authenticate users and control access to those resources based on that identity and associated access policies. And I thought, those are pretty on point of what identity native access is.

So an example of how this works: identity native. We start off with our identity, then we segue into device security, which you're going to hear a lot more in the next couple years as device security improves. Then we go into proof of presence. So, YubiKeys, Touch ID, Windows Hello, and then mutual authentication. So not only are we identifying that user itself but even your host that you're connecting to, to make sure there's mutual authentication across the board there.

So, let's dive into it just real quickly here. So, identity, we all have one. It's who we are, right? That's the key component of identity native. So you take that identity, you tie it into an IDP like Okta, Active Directory, Ping, etc. That's your source of truth. You know, five-six years ago, you mentioned Okta, and people are like, "Oh yeah, nah, we all use local usernames and passwords. Okta's kind of like, oh, that would be kind of..." Nowadays, if you're not using an IDP, you kind of get that strange look of like, "Why are you not using something like that to manage your corporate identity?"

Then we layer on device security. And how that is, those are Windows Hello, TPM, Trusted Platform Modules, Windows 11 supports that. Touch ID for those of us that are on Macs, biometrics, HSM, hardware security modules. AWS has them, there's cloud HSMs, there's physical HSMs. So what you're doing now, you're storing these identity information on encrypted devices that, then, are local. But they're not going to be able to, they're not hackable, for the most part.

Then we use a proof of presence tool. So, YubiKeys. Is everyone here have a YubiKey? Anyone not have a YubiKey or something similar? So we have a few. So, YubiKey, Google Titan, which is kind of cool. I picked one up at re:Invent last year. Touch ID, right? Anything that's MFA, 2FA, that to validate that user physically is present with that proof of presence, validates with a touch or fingerprint. And nowadays we've got passkeys. That's like the new, you're starting to see really good traction with passkeys. Same idea with that proof of presence. I'm going to look on my phone with Face ID, validate, and then gain access.

And then mutual authentication, right? So, we're not just going to trust the user. We're leveraging mTLS to trust both the host and the clients. Does require work because you're going to have to set up mTLS on both hosts and clients to, and have a way to sign and validate these certificates.

So, the rest of the time, we're going to talk about how we would implement something that's identity native using short-lived certificates. So, short-lived certificates, which is the key of all this. But first, we're going to do a primer. Not a primeval, for those that are PC gamers, Diablo fans, which I am.

So, let's say, let's take back and listen. What is a certificate? Most of us know, but in case you weren't aware, those that do OpenSSH and you're using keys, OpenSSH has had support for certificate-based access since 2010. That's like 13 years almost in March, but yet we don't see it or use a lot because it's challenging to configure. If you've ever done it out of the box or read the docs, it's a little bit challenging, but it's been out there for a long, long time.

Certificates are used everywhere. They come in a variety of formats. OpenSSH, as I just mentioned, for SSH activity. Then pretty much X.509 for everything else that's out there. A couple of links here as well, to check out some, you know, talking about OpenSSH versus X.509 from both our company and Smallstep, another company doing some really cool work in this space as well.

So, why certificates? They're tied to a user or service identity. So any connection or action can be traced back to that specific user or service, right? So that's, you know, it's been signed. This is the one I probably love the most, right? They automatically expire. There's no need to revoke a certificate. When you issue a certificate for four hours, in four hours, it's gone. It's done, it's terminated, and hopefully any access that you've leveraged to gain to within that is also terminated as well. You don't have to manually go back like, "Ah, John Doe just abruptly left the company, didn't return the company laptop. He had access to all certs." Oh, I'm not going to worry about it because they're all going to expire anyways. They solve that TOFU problem. Not tofu for the food. They trust on first use problem.

And then a little note here about Teleport, since a lot of this is derived from how our product, our open-source product, handles identity and access. And then they enable mutually authenticated channels, mTLS, right? Which helps mitigate a wide range of attacks when you've got mutual TLS on both ends of your communication path.

And they work better for large-scale deployments. So, anyone here running north of 5,000 EC2 or VMs, 100-plus Kubernetes clusters, 50 Kubernetes clusters, anything of that scale we're tying into? None? Okay, but even if you're just your own personal stuff, even when we talk about large-scale deployments, even if it's 50 servers, trying to manage keys or access, even if it's five Kubernetes clusters, it can be a little bit challenging. So, leveraging certificates can definitely help for sure because you got a cert that's been signed by a valid CA, and you don't have to copy that user credential to every device that's out there.

A little geeky slide here that's in our doc, so just a little bit different. What's in a certificate? I mean, these tools are on your laptop if you're a Mac, Linux user. Just do the OpenSSL or commands, and you can actually dig into your certs that are all, you know, that begin here and here. Don't edit anything in the middle, and it actually decrypts all of that. You can actually read it, you can see what type it is, you can see your valid principals, you can see the expiration date, the signed date, who signed it, and everything. It's actually pretty cool, geeky stuff that you want to impress people with, whatever, say, "Hey, check out these certs. These are super cool." But there's the differences really between the X.509 and as well as OpenSSH, and there's a lot more principles and pieces among all those.

So, it brings us to our ultimate goal, right? How do I secure cloud-native resources with certificates? The good news is that most of the tech stacks we use today support options besides just username and password or an SSH API key, whether it's an IAM policy, cert-based or whatever, they're out there. They're not necessarily going to enforce it or say, "Hey, you have to do it this way." But it does require you as a user, or team, or an organization, to change how you access these resources, to think a little bit differently. Say, "Hey, we've been doing it this way the whole time. It doesn't mean it's the best way," and to go to a better way of accessing resources.

So, a couple practical getting started steps. Leverage cert-based authentication whenever you can for your applications, your databases, cloud providers, you name it. Apply the Strangler pattern for deprecating secrets. When Kubernetes and Docker and all those things started getting really popular a couple of years ago, everyone's like, "Ah, just apply the Strangler pattern," you know, because I got this huge monolithic. So, leveraging that same idea, right? We don't just throw everything in at once. We throw it up there, we change the process, but we leave it running. And eventually, hey, there's no dependencies, no one's using this method anymore. We've strangled it, then we can eventually terminate it.

Leverage your IDP (Identity Provider) and RBAC (Role-Based Access Control) to limit who can see and do within your infrastructure. Not just give everybody admin rights, or Ubuntu rights, or EC2-user rights with sudo privileges, right? You want to leverage that with well-defined groups and roles.

And then you can build it yourself with standardized tools, processes, or use open-source tools. Netflix BLESS is probably one of the big companies that did this a number of years ago when they kicked off. The repo is still out there, but it's not updated anymore. The two that I've highlighted: one is our core product, Teleport. It was designed by our engineering team to access resources based on short-lived certificates. Then there's Lyft BLESS, which is a fork of Netflix. If you want to play with a super lightweight demo, that's my GitHub repo. It spins up a couple of Docker containers, signs a certificate, and lets you access one container using short-lived certificates.

But just as even as an access of cloud-native resources leveraging this. So, this is a MongoDB database, MongoDB Atlas. And by default, when you log in, it's going to be like, username password. But they do support the ability to leverage certificate-based authentication, but it's under something like 'Advanced', which most of us are like, "Ah, just give me the basic wizard. Let's go," because it tends to be a little bit more advanced. You got to know like, "Hey, who's my CA? My cert? Myself? X.509?" But once you have all those processes in place, it'll dramatically speed up and increase your security posture.

So, if you example use cases, leveraging short-lived certificates for Kubernetes access, offload that kubeconfig management. Tie Kubernetes access to granular roles, right? So, right now everybody gets a kubeconfig. You have it forever. But if you could leverage somehow short-lived certificates and access to manage that, then you don't have to worry about that. Or granular roles so that John Doe only has access to this namespace, Jane Doe has access to this one.

Eliminate the homegrown Bastion host that has all the SSH keys on it. A lot of people still use that, a lot of Bastion hosts out there. It's your single point of failure in some ways because if somebody gets to it, oh wow, I now have all these SSH keys on here. I can access it.

Tie service accounts, Ansible, your GitHub pipelines, Terraform into short-lived certificates, versus pulling a static credential that might be injected in runtime. Helps mitigate some of that man-in-the-middle type of attacks.

Eliminate excess named users on databases. Not everybody needs 15 named users on a database, so maybe consolidating some of that. Reducing or eliminating usage of VPNs to access web-based applications. Like, "Hey, this is a Prometheus or Grafana dashboard or internal website and we're using VPN to access it." You can leverage short-lived certificates to manage that. Then, tying into your Cloud API SDKs using identity-native versus static credentials.

So, a few next steps if you want to try it out. Today wasn't a demo, but you can go check out our site on GitHub. We have a Slack community where we have a couple of thousand people on there. You can talk to me, other, even our founders are on there as well. Some articles on passwordless, and that's going to be a really big initiative over this next year. You're going to see a lot more on that passwordless piece moving forward.

And then we actually have a virtual conference that's a one-day event. It's free to attend. We had it in person this past fall in California, but they're redoing it with the virtual. If you want to hear use cases of actually some of our customers that leverage our non-open-source core product into how they're securing it, feel free to sign in there, register, or whatever it is. That's just a bit.ly URL back to that.

And that's all I've got. So, I know we got about one minute or so, but if you have any questions, feel free to grab me here afterwards. I appreciate your time today.