Bad Guys and How They Hack It

Good morning, how is everybody? Everybody good? Oh, the lights are bright.

I actually was doing a bit of research on Tampa before I came out, and I didn't actually realize how much of a history that Tampa actually has in cybersecurity. I think it has some sort of a name, was given in 2006, I think it was, of being the US cybersecurity city or something like that. So yeah, it was interesting to hear.

Let me start with a guess a bit of my background. So I've been in the cybersecurity industry for coming up for 20 years now. And I've done all sorts of things, breaking into buildings, try bribing people their passes, try and gain access to their systems in order to be able to help them to understand how to better improve cybersecurity. And then a few years ago, we started out a project to try and help businesses to develop their cybersecurity in a really material way to try and improve the security really quickly. Lots of small businesses really struggle with this, right? That they haven't got the right resources, they don't have the right expertise, they don't have enough time to really manage their cybersecurity threats. And that's where we came up with defense.com. But we do loads of stuff, right? We do pen testing, we do all sorts of consulting to help people improve their security.

But what I thought would be interesting is just to talk to you a bit about how we do what we do and why we do it. And so my talk is really just to really help you to understand a bit about how to think like a hacker and think a bit more like we would when we're doing a penetration test. So what we're going to talk about very briefly is container best practices, just cybersecurity stuff. I'm not going to go into the weeds, because I would imagine that you guys probably know a lot more than I do about containers. But I'm going to touch on it a bit and explain some of the concepts. And then, like I said, I'm going to bring some war stories, which I think normally helps to bring more of what goes on in the cybersecurity space to life, rather than me just kind of talking about best practices. And then we've got a hack at the very end, which I'm challenging you guys to try and see if you're going to try and identify how we did it. And I'll explain why it's kind of the impossible, what would seem like the impossible hack, and how we solve it. And then I also wanted to mention that we are doing a, we are going to give away some prizes. So if people do work out how we did the hack, then you can get one of these world renowned defense.com games, Game Boys. They've got 500 games on here. So if you do manage to work it out from the talk, then come and see me. I'm not going to tell you how we did it. And so you can kind of give me the concepts of how you think we managed to achieve it. And then we also have a workshop where we're going to be doing some real hacking. So we'll kind of teach you guys how to hack. So if you register for that, it's going to be great fun.

Key takeaways

So let me start with the key takeaways. So what we really want people to take away from this talk is, like I said, to discover why you're a target. A lot of the time when I talk to organizations, they tend to feel like they're not a target for a number of reasons. Normally, it comes down to they don't think what they're doing is going to be interesting to an attacker. Unfortunately, loads of organizations fall into that trap. But the reality is it doesn't really matter what you're doing. It really comes down to, can they monetize that attack? And that's why you see lots of ransomware attacks and lots of DDoS attacks happening these days is because it's just an easy way to make money.

I also hope that it will help you to build some confidence in your ability to identify the key areas where you need to monitor for systems and to identify where you might get targeted. Ultimately, the main thing that I always say to organizations is to limit your attack surface as much as possible. So we'll talk a bit about what we think of as an attack surface.

And then hopefully to learn to think like an attacker, you'll definitely get that from the workshop if you're coming to the workshop, because we're going to talk very much about how we think about cybersecurity, slightly different to most people. And certainly, our pen testers do exactly that. They don't think, how does this application operate normally? They normally think about it in a different way. They normally think about it from the perspective of, how can I make this thing break? And then learn from how it's broken to then try and exploit the system.

Red Hat survey:

All right, so Red Hat did a really interesting piece of research, actually. They managed to survey 300 DevOps engineers. And they asked them a few questions specific to container security. And one of the things they mentioned was about the fact that 55% of them in a 12-month period when they ran this research found misconfigurations in their Kubernetes. And I think that's the key thing as well, is that sometimes it's not about vulnerabilities. In actual fact, most of the time, it's not about actually vulnerabilities in systems. It's more the fact that you're misconfiguring something, because there's a lack of knowledge or understanding of how that thing should be configured. Also noticing that they'd had to delay projects, because cybersecurity wasn't something, or they had vulnerabilities and they had security concerns at the point in which they were going to release their project and realized that there was a problem. And then, of course, that slows down the project.

Now, in the industry, we talk about a couple of different concepts. One is privacy by design, and the other is security by design. So the concept is that if you develop it into your application, and that's how you think about building applications with security and privacy in mind, then it just goes so much smoother. You're not thinking about security retrospectively. You're not trying to fix something once it's about to go into production.

So if you're thinking about it from the perspective of an attacker, you can take that operation into every day when you're trying to develop new systems or build new infrastructure. And 57% of the respondents said, also, they were most worried about securing workloads at runtime.

Best practice for container security

So the way I look at this, or the way I like to think about it, is the four Cs of security. So it's cloud, cluster, container, and code. But when you really break that down, what it comes down to is a couple of different areas, so five different areas. The first being the images themselves. So if you get access to those images, you can do all sorts of things. So one of the ways in which to solve that problem is to sign those images. So if you keep a registry of the images that you have, you can then identify if somebody has made a change to that image by looking at the signed cryptographic key of that image. Obviously, keep them up to date. Images are really important to keep up to date. That's why we always say in cybersecurity, always run regular vulnerability scans, because vulnerabilities are coming out all the time. Almost every single day, there's a new vulnerability coming out. So it's important to keep those up to date.

The other area is your image registry. Now, a lot of people are moving to the cloud and deploying this on AWS or other providers, Civo. And so the responsibility can often land towards that provider. And sometimes people don't really think about this too clearly, in the sense of they just think that cybersecurity is not important anymore, because you've given the problem away to somebody else, the cloud provider.

There's a concept of a shared responsibility model, which is that they want to give you enough rope to configure it yourself with.

And so if you're not looking at that from that perspective, you can often even misconfigure it enough to be more insecure than doing it yourself. And that's clear with registries as well. Main thing is about access control, so making sure that people can't get access to your image registry. If you're going to host it yourself, then make sure it's secure. And ultimately, also looking at if you're dealing with third parties, making sure you do your dealing as long as third party.

The other area is orchestration. It's really powerful, right? But unfortunately, it has so much power that if you get access to that infrastructure, then you can have a lot of fun if you're a pen tester, because you can almost take over the whole environment. So also from a monitoring perspective, it's really great when we see organizations that are fully using orchestration tools, because we know that if somebody then starts doing something more manually and they're not using orchestration tools, it's a real clear indication that somebody's doing something malicious. So we always look out for that sort of stuff when we're monitoring people's environments.

So the main thing to address some of those challenges is around looking at least privileges. Again, a big thing in cybersecurity is about giving people as minimum permissions as you can. I'll talk about some concepts of how you can do things like privilege escalation. So that's one of the big things we do when we're pen testing, is we take a very limited level of access and we try to escalate it to as high access as possible. And then also just monitoring, right? Monitoring the orchestration tools, like I said, looking for obvious areas where the tools are not being used.

And then the last two areas is really around the host OS. If you can deploy an operating system that's really slim, so a slim OS, also if you're hardening it, then it massively reduces your risk. So using something like CIS benchmarks is a good way to make sure that your host OS is hardened well. And of course, again, monitoring those systems as much as possible.

And the last thing is, of course, the container runtime. Monitoring things like network connections between those systems, run regular checks, but also make sure you keep it up to date so that it doesn't get hacked.

Brief History of Hacking

All right, so I then thought, well, maybe we should look at the history of hacking, actually. Because hacking really started out as, I guess, something that was more done as fun. And it was something that was maybe there was something that we saw this week that was involved in the very early stages. So I found this clip from 2001. Hackers are the outlaws of the computer. Hopefully it gives a bit of a feel of how security's changed.

“Stealing, spreading viruses. No one has a good word for them today. But it wasn't always so. This hacker started a revolution in computers. By 1982, Apple sales had reached over half a billion dollars a year. But hacking and business were not compatible. Hackers could claim to have created the modern world. But it was a world in which they would no longer be welcome. Oh my god.”

So yeah, hacking's been around for quite a while. And like I said, it was fairly innocent in the early days, like when Wozniak was involved in it. He actually, and that clip is on YouTube. I think it's from 2001 or something. And he talks about phone phreaking. That was kind of the early stages of hacking, right? I love the story about the Captain Crunch serial. If you guys are familiar with this story, it's basically there's a whistle in the Captain Crunch serial which allowed you to whistle the perfect tone to be able to get free international phone calls. And so in the early days, they were using that to make these phone calls. So of course it was fairly innocent in those days. And unfortunately how things have changed, it's no longer like that. It's all about making money and monetizing your attacks. And these are the sorts of things that we're seeing right now.

Ransomware

The big ones being things like ransomware. You've probably seen tons of this in the news. I probably don't need to tell you too much about this. But there's all sorts of concepts now that are coming out of this, which is that when the first attempt to extort people out of their money doesn't work, they try other things, right? So when the encryption, when they're encrypting their disks and they want to get access to those systems and people are not paying, then of course they might do other nefarious things. But ultimately they also have scenarios where they'll pay that first ransom and then the second ransom is on the data they've stolen. And that's the concept of direct extortion. I've seen all sorts of things as well where once they've stolen the data, they then try auction on a site, like a real James Bond sort of villain style, where you go to an auction site and you're the one company that you want your data back and they're trying to get other people to bid against getting your data. I think that was something that was seen in the Garmin hacks that they tried to get other people to buy the Garmin data and Garmin were trying to get their data back by paying the ransom off.

Cloud attacks

Of course, cloud attacks. Like I said, it's just because these major cloud providers have a really good security posture and they've got lots of experts in this field, it doesn't mean that it's not a target. It doesn't mean that attackers are not gonna be looking to try and exploit those systems.

Social engineering

And then of course, social engineering. So to give you a couple of examples of this where we've seen it work really well is a recent example we had where we managed to break into a building that was linked to a bank. So it was really high security. When you go into the lobby, there was turnstiles that you couldn't tailgate. And our team identified that there was an opportunity where when you were in the building as a visitor, you'd get a visitor's badge. But what would happen is when you went to leave the building with your visitor's badge, there was a little clear box that had the visitor's badges in and you'd drop them in. So our guys thought it'd be a good idea to try and see if they could get a visitor's badge because that would be a good opportunity to try and gain access to the building. So they walked past the reception area on their phone, kind of pretending that they were having a phone call. They went past the security guards out of the view of security in order to be able to come back into the building as if they'd been in. And so they came walking back and had a dummy pass. And as they walked up to drop the pass in, the receptionist wasn't looking. So they grabbed a pass out of the box and then they left. The problem was when they went back to try and test that card, it wasn't working. And so they had to think on their toes really quickly. And so they decided that the idea would be that they'd try to go into the building and they'd say to the security guard, my pass is not working, can you let me in? And so they did that, but the security guard went, no, you've got to go to reception. So at this point, our team was thinking, oh, we've been blown, this is not a good time. So they walked over to the reception and said, I don't know why, but my pass is not working. I was here yesterday and it's not working. And the receptionist just literally went over to them and said, sure, I'll give you a new pass, gave them a new pass, and then they gained access to the building. Once you're in, of course you're trusted. I've seen this in so many scenarios myself where I've been able to get access to an underground car park. And as soon as you're in the car park, although it's not actually a secure area of the building, people will still let you up through the lifts even without a pass. So sometimes that kind of early leverage of getting that kind of eventual access is again, much like privilege escalation when you attack a system. It's about having that confidence that once you're in, you can go wherever you like.

And there was another really interesting one that one of the pen testers told me the other day, which is where I think it helps you to understand how the pen testers think very differently or hackers think differently. They were doing an internal penetration test and they noticed that there was a printer on the network that was making calls to an SMB share. And they said to the customer, this was only this year, hey, is there any way in which I could reconfigure that printer, is that okay? Because it's connecting to another SMB share somewhere. And they said, sure, we don't even use it, so it's fine. So they weren't even using this configuration. So what the pen tester did was they got the printer to talk to their laptop as if it was the server. And it basically gave them admin credentials because they'd used admin credentials to gain access to the SMB server. So just setting up a little listener, they could gain the admin credentials. As soon as they did that, they had domain admin basically on the box. So thinking about things like printers and other systems in your network, and that's why social engineering works so well, is once you kind of get that early access for a social engineering exercise, you can then pivot and leverage that access and go on to gain access to other areas.

Now, look, cybersecurity, I hate the fact that it's about FUD, which is fear, uncertainty, and doubt. Because we don't want to scare everybody into thinking that it's too hard and we shouldn't be doing it. I can't do anything about it, I'm just gonna get hacked one day, so I need to just be prepared for the worst. You absolutely do, unfortunately. So having some way in which to be able to deal with it if it happens. And if you think about it like any other scenario, the more you do it, the better you are at when it actually happens. Most people don't test their policies or their procedures, so when it actually happens, everyone freaks out and goes crazy. So you gotta be more prepared than that.

New and upcoming attacks:

So the last couple things here is just about some of the new and up-and-coming areas of attack. So AI-powered attacks, of course, everyone's heard of ChatGPT, right? So within about a week of that having been, really hit the mainstream, of course, immediately security researchers are looking at ways in which they could use it to write software, and some people actually managed to write their own malware, even though it has all those restrictions to say, when you type something in, hey, write me a phishing email, they all do it, but it will just say, you shouldn't be doing this, and it's against our policies, but of course it will still do it.

So the guys managed to, have seen writing actual malware through chatGPT for them. Now the reason why this is so problematic is that in the hacker scene, the concept is that normally you're not good at everything, there's specialisms, right? So there's somebody who's really good at writing malware, there's somebody who's really good at social engineering, there's somebody really good at writing tools, like hacker tools, but you've got to bring those people together, and so if you're a script kiddie who doesn't really know what to do with this stuff, things like AI and chatGPT make it really more accessible to those people, and which obviously increases the amount of attacks we're gonna start seeing, and we have already started to see.

And the last area is IOT. I think there's something crazy, like 1.5 billion IOT breaches were reported in the first six months of 22. That's insane, like how many breaches, that's just mad. I don't think we've seen anything like that in recent years, but the main reason for that is just IOT is not developed in a way that you can run really powerful security on it, because they're just very small microcontrollers, and so it's very difficult to develop security software for it. So it's all about how you contain those systems, and how you restrict access to them, and how you manage that part, which is probably why we just see so many breaches in that space.

Exposed password in AWS

Okay, so let me give you an example of a couple more. So this one, I wanna talk about some stuff around AWS, because the hack that I'm gonna show you that I wanna see if anybody knows how we did it, is on AWS.

In this example, it was a really interesting hack, actually, because what the team had worked out is that the way the test went is the customer asked, they said, look, we give contractors access to our AWS account, and what we do is we give them guest-level access, so they have least privileges, they don't have a lot of access to our systems, but we do give them access to the Lambda service, which is not a very good plan, because most of the time, they use the default credentials, templates for these, and unfortunately, in those credentials, what people don't realize is that you can actually escalate your privileges using that method.

So we were given these guest credentials in the system, and we managed to log into the system as a guest user. So once we were in as a guest user, we could then use the Lambda services to enable us to escalate our privileges up to the highest level. Once we had the highest level access, we then had access to, we were testing a cryptocurrency exchange, we had access to basically a whole ton of additional credentials of various different crypto accounts, and in theory, it could have been a really bad day for that customer had we not been the ones that tested it and found those issues. So a lot of the time, it's again, misconfiguration. You know, it's not thinking about hardening those systems, it's not looking at things like AWS and saying, well, actually, these default policies, do they really work for me? You just kind of naturally just say, okay, this is a good policy, AWS set it up, so it should be perfect for me.

Introducing the Hack:

Okay, so I wanted to also kind of intro a bit more about the hack itself, just so that it sets the right tone, I guess, for what I'm gonna show you next. This is something we see quite regularly, actually, and surprisingly, a lot of the time, people say to us, well, I would never set that up that way, but trust me, we've seen it in the wild so many times, these sorts of configurations.

The idea is that what we're doing in the attack is we are looking to escalate our privileges from a very low level of access, and then we want to escalate up to the highest position, as I said before. But I kind of started thinking about this talk, and I started thinking, you know what, actually, hacking or securing systems is very much like space invaders, because when you think about the kind of invaders that are constantly moving towards you, and you've got these defenses, but the defenses are constantly getting depleted, right? While you're not patching stuff, and we're not thinking about security, the bad guys are getting closer and closer to you. And also, it made me think that also, where you have gaps, you know, when you have those defenses, and you're kind of moving your spaceship down the bottom, there's those gaps that if you don't fill those gaps, then you're just gonna get hit by one of these stray bullets. So we thought it'd be fun to try and position the rest of the talk in a bit more of the way of like an arcade game.

So we came up with this idea. So we came up with this idea.

What I don't get is why I've managed to get the worst scores in the game possible. Like I'm not as strong as John, our sales director, so I was pretty unimpressed by that.

So yeah, let me set out the scene for the environment. So the environment is a single EC2 instance that's in AWS. It's in an AWS VPC. It's running a web application via Docker. It's an application that's, we've actually seen this being used by customers. And again, it's one of those scenarios we've seen regularly, that they have like apps and stuff where you can enter like URLs in. So we've seen it in this exact case, we've kind of replicated what we saw in a customer environment where they basically just had, you could set up a URL, and it would keep pinging an application to make sure that the application was up. So yeah, and the EC2 instance's front end is just has a standard like single application load balancer, so it's nothing special. It's just a very simple environment, nothing crazy.

But what we have done is we have managed to deploy WAF. So we have a WAF in place. So you'd think that the WAF would be pretty good at detecting the attack that we're gonna be doing. We're also filtering input, and you can see that the input is in the rules that we've created. We've got SQL injection cross-site scripting rules in place. So that shouldn't be possible. We are doing input validation. So this is the UI of the interface of the system we came up with for this kind of health check concept, like I said, which is a replication of what we've seen within customers. And if you look in the URL itself, or in the tool itself, you can see that it's coming back saying the URL cannot be local host. So that's kind of validating that you can't submit local host, because what we're trying to do is to get to the metadata service. Now we are using the AWS metadata service V2, which is also problematic when you're trying to test environments, or if you're trying to hack an environment, because it does actually require you to be able to provide additional header information. So in this case, if we were testing this application, we couldn't enter any additional information aside from just the URL. There's nothing else we can put in there.

The other thing that we did is we specifically made it so it was impossible to actually call directly the metadata service on the local IP address. As many of you probably already know, the metadata service is always on the same IP address. So it's very interesting when you're testing, or if you're a hacker, because you always know that's gonna be consistent. You know that's not gonna change. So that means that you can do repeatable attacks against these systems. But again, so we're blocking again the system. So we've got WAF in place, we've got input validation. You can't connect to a local host, you can't connect to the IP.

And then, like I said, we've got a single load balancer, no other vulnerabilities are accounted for. It's just purely that. And like I said, we've got metadata version two, and that would also stop things like SSRF attacks. So that wouldn't be possible.

And then the last thing that's worth mentioning is that we do actually have policies already in place for the EC2 instances. So these policies are managed in such a way so we've got permissions, IAM permissions, and they're operating in a Lambda function, as I said before, like a Lambda service.

So, however though, we were still able to get full admin access. So what we wanted to try and get people to try and think about is how do you think it would be possible for us to be able to get admin access to this box? And what I mean by admin access is it's full admin access to AWS. So it's not like we would be only getting access to the application. So I guess just to recap everything, to go back over it. So we've got input validation, we've got a WAF, we've got a very small attack surface, we have IAM users in place, which is read-only, so we don't have a user which has full access, just read-only access. We have a single input field, and we have a very small attack surface. But it was very, actually fairly trivial for us to gain access to and escalate that through.

So like I said, I'm not gonna give you how we managed to do it, but I am gonna ask you to come and chat to us maybe and see if you can tell us how we did it. And of course, if you're gonna come to our workshop, then we will be showing you a bit more about how we did it.

All right, so just to, I guess, try and wrap stuff up a little bit. So from a overall attack surface perspective, like I said, the idea that we always work on is that you want to try and reduce your attack surface as much as possible, and that could be anything, right? It could be the application you're running, like how many different types of interfaces do you have? How many input fields do you have? If you're talking about large infrastructure, like an office location, like how much access do people have? When you think about even privileges, the more privilege we have, the higher the attack surface is, the bigger the attack surface. So what we do is we say to customers, think about it from an attacker's perspective. They start with a recon, and what they do is they try to recon your business. They look into people in the business, they look into the Excel infrastructure that you have, they look at trying to profile people on social media. All of this information comes together at the end to try and build up an attack, which would then try and target organizations. So as much as you can restrict that information is the most crucial thing, really.

All right, so to, I guess, finish up, the way I see it is that as an organization, we want to try and get you to think like a hacker. So we want you to try and tell us how we've done it, how we've gained access to the system. If you want to come and chat to me and ask me more questions, I can definitely explain more of it to you if you want to hear more about what we did, so you can try and get a bit more idea of how it might happen. And yeah, so we are going to do our talk, it's at three o'clock, I think, later today. So if you come along and speak to us about it, and we'll sort of explain the hack, then you can also try and get access to one of the Game Boys, as I mentioned before.

All right, and that's it for me. It's game over, or is it, I think, is the term we made. So yeah, I think we've probably got time for questions, if anybody's got any questions.

Q&A Session:

We do have some time for questions, so thank you. We have time for some general questions as well, not just the specific hacking questions. You can always visit Oliver at the booth in our expo hall to ask specific questions, but who has a first question? And just state your name and your question.

Question: Hey, I'm Adam, thank you for this awesome talk, by the way, it was really entertaining. You mentioned AI as being a threat as a malicious attack, but how are you thinking of using AI in defense against attacks?

Yeah, look, machine learning and things like that have been used in security for quite a while, and I think the problem has been that in the early stages, everyone just thought it was snake oil, because they didn't really have any kind of idea of how it would really work and how it would be used. And to be honest, again, we've done so much innovation in the cybersecurity space, and people are still getting hacked. A lot of that is just down to organizations not doing what they need to do in the basics. Like, you'd be so surprised how often we see the basic types of stuff happening all the time. And it's kind of frustrating to me, because I've spent the last 15 years talking about it, and you still talk about SQL injection, you're still talking about cross-site scripting, all the boring stuff that we should all be very good at remediating. So for me, it's always been about, well, if you just get the basics right, why do we even need machine learning, because you can solve all these issues. But of course, as we evolve our security capabilities, then the attackers are looking at new ways in which they can circumvent those controls. And of course, there is a lot of innovation in the cybersecurity space. Look, there's so many exciting use cases for AI, right? Because, you know, not least of which just to be that kind of assistance when you're doing an investigation in so many different areas. We have security analysts. They spend their time monitoring log data. And some of the time, you don't have a lot of time to think. And you could really benefit from the AI having some knowledge of what's going on in the environment for you. So you can just ask it simple questions, like, hey, how would this attack be possible? Can you explain to me how this specific piece of log data or this particular threat could help me, could have led to this type of activity? So those sorts of things kind of speeding up the process of doing an investigation would be really beneficial. Also, of course, just automating just general attack type stuff, right? It's kind of analyzing other attacks and seeing how the attackers would operate. There's some really cool concepts where if you take a certain type of tool and you gather all the data about attacks and you use that information to say, well, if they start with this tool, what are the most likely things they might go on to target? You can then be proactive in trying to contain a breach. Because you know, well, normally when they use these tools, they're looking to exploit SQL databases, the machine learning algorithms. The systems would understand that. The AI would know that. And it would say, OK, you need to block this port. You need to gain, you need to restrict any access to any database servers you might have. The other areas which we've always found really tricky is most customers don't actually know what they have in their environments. And I've had scenarios where I'm talking to big banks. They've made acquisitions of other banks. And then they'll ask you a question. You'll ask them a question and say, hey, do you know what this server is, what it's doing? And they said, no idea. We don't dare turn it off because the guy who runs the system is no longer in the business. And if we turn the thing off, we don't know what it's going to break. And so we don't want to risk it. So I think there's areas like that where it could take information, like gather large volumes of data and try to help us to understand, like what are these targets, what are the systems, and those sorts of things. So yeah, of course, there's always ways in which to benefit from the technology as well as attack us to use it against you. But I think the downside, I guess, is when you're building innovative technologies to deliver to customers, you have to make sure that the technology works really well. Whereas the attackers can use it today because they don't really care about how well it works in production. They're just going to use it to their benefit, not worrying about that sort of stuff. So we have to be able to innovate faster, I think, especially in the security space, which is what we're working on.

Very good. Thank you. Oliver, over here. We have a question in the back. Go ahead. Tell us your name, where you're from, and your question.

Good morning. I'm David Nessen from Evercast. And I have a question on UDP ports. We're doing some communication via WebRTC, and we've got a bunch of UDP ports we have to have open for the communication to go through. So what is the security concern if we have 10 UDP ports open versus 10,000?

Yeah, again, it comes back down to the question was more around, look, what's the difference between a single port being open and then tens of thousands of ports being open? From an attacker's perspective, it doesn't really matter how vast the attack surface is. You're only looking for that one entry point. So the one failure in security is the one that gives you the access you need. So it doesn't really matter how much you expose, necessarily. But of course, you want to try and restrict as much as you can. But the risk is, of course, that if you're scaling at that size, how do you monitor that sort of stuff? How do you make sure that somebody hasn't made a mistake and you suddenly scale out thousands of these things and all of them are vulnerable or just one of them is misconfigured in a way that it's vulnerable? So sometimes it's hard to see the wood through the trees. And that's why it's always good to try and reduce your attack surface, because you can get it down to something that's a bit more manageable that you can monitor. Then the chances of you detecting it is much greater. There's some statistic like 186 days before detection in an environment, where an organization just doesn't know what they're monitoring. And it takes them so long to find anything. By then, it's too late. Like, everything's gone. The data's all stolen. Most of the time, it's much shorter these days, because they want you to pay their ransom. So they want you to know they're there. But many years ago, that wasn't the case. They'd stay in the environment for as long as possible. The early stage of my career, we were looking into credit card breaches. And that was more about siphoning off credit card data over really long periods of time. And so they'd want to be super stealthy. So yeah, to answer your question, it doesn't really matter as long as all those systems are secure. But then the risk is just that how do you manage that? How do you know that they're all secure? How do you make sure that when you deploy these things at scale, that they don't all end up being deployed at long period of vulnerabilities?

There's another bit of research we did, actually, which I thought was quite interesting to address some of that question as well. Some people think, well, we don't need to worry about our systems being exposed for a short period of time. So we did a test where we put a server on the internet. We didn't expose it with any DNS records or anything like that. So nobody would have known this system existed on the internet. We just spun it out on a cloud service. And within 32 milliseconds, we had our first scan. And that's because there's tons of bots out there on the internet. But there's also tons of hackers out there into just scanning the internet continually. So if you spin up those 1,000 UDP ports and suddenly somebody comes across them and they are vulnerable, they're going to be found within 32 milliseconds. And then they're all going to get popped really quickly. So it's more about timing and making sure you're thinking about security. And that's why we always talk about this design and security in from the very beginning. Because then you know that you've got that confidence that if you reach out to spin out thousands of things, that you know that those things are all going to be secure.

Great. And I think we have another question over here as well.

Hi. This is probably just more of a general question. And I know this is probably security in layers. But do you have a suggestion of a security posture that you should be implementing based on infrastructure as code or whatnot for a multi-account architecture that has internet access in the AWS environment?

Yeah. So from our perspective, it's always about defense in depth. You don't want to rely on just one source of information. We talk about two different types of data. There's high fidelity and low fidelity. High fidelity data is the stuff that's generally related to, like we've actually detected an attack. So it's normally intrusion detection systems. Log data can't tell you that sort of stuff. Like it doesn't, it's always the kind of byproduct of something that's happened that when you look at the logs, it's like, hey, some admin started doing something suspicious. High fidelity data is you actively see somebody trying to exploit you. So from my experience, the best way to do it in places like AWS is to use some of their existing technologies. They do have intrusion detection systems. If you ingest things like flow data, it's useful. But normally, only in forensics cases where you're trying to validate your assumptions about where did this malware come from? Hey, we saw this malware getting pulled down from a location. Can we look at the flow log data to confirm that that was exactly the data that was downloaded? Of course, you want to capture your log data. I do say to people as well, don't just think about it at the operating level. Some people are developing their own applications that have really interesting security information in them, but they'd never developed the logs for us to be able to consume them. So in our services, we basically deploy so that we have technology that collect from the actual cloud provider itself. So anything we can get out of the cloud provider, admin activity, anything from the control plane is really handy, really useful. You can build detection rules around those systems. Obviously, collecting the log data, looking at network traffic, turning on things like application firewalls, as if a web app. If you can get network data and you can do intrusion detection, using intrusion detection systems running in the AWS environment, again, it's a great way to get the data out that you need to obviously flag the alarms to say, hey, go and investigate what's going on. And then the idea would be that you try and consolidate that data together. So you take the high-fidelity data of the attack and the low-fidelity log data, and you jam those two things together, and that will give you a timeline of activity. So here's the actual event where we saw somebody trying to attack something. And around that same period, we saw this log data going on. So somebody was messing around with the control plane, and somebody was messing around with the instances. So it's kind of a defense in depth strategy. I wouldn't say there's any particular tools I can think of off the top of my head that you could implement. But I would definitely say turn on a lot of the AWS tools like GuardDuty, and they've got lots of other cool tools that they're using in AWS that you can consume data from. But the most important thing is somebody's got to look at them. Somebody's got to do something about it. There's no point in just pulling loads of log data and you just never use it. So it's more about the activities you take, the things you do with that information. Once you've got it, that is the most important thing.

We have time for more questions. Anyone? Well, Oliver, as we wait for somebody to come up with more questions, I have a question for you. What do you read in your day to keep up to date on the industry?

Oh, there's so much. I guess there's so many places of information. This is one of the things we've said for customers in the past as well, that the challenge is that there's so much information out there. How do you make it relevant to your business? So what we've done in defense.com is we've actually developed the product so you can define what you have. So you can say, I'm running on AWS, or I'm running on Azure, or anything like that. And we'll consume that information in, and then we'll make it relevant to them. So we'll feed that information back into their feed of information. So we have a threat list, which we can say, hey, go look at this stuff. So a lot of it's just general threat intelligence news. Like, hey, these new vulnerabilities exist. There's all manner of cool places to go and get that information from, just general threat feeds. Obviously, podcasts are great. There's some really cool ones out there. There are really interesting stories. There's one called Darknet Diaries, which people might have heard about. That's pretty cool. There's Hacked, which is another podcast, which is really good. So they're good places of sources of information. General news is pretty good. There's things like Reddit. But of course, if you really want to find out what's going on in the security space, Twitter is actually where most of our pen testers go. There's tons of really good Twitter feeds. And Discord, as well, seems to be coming up as well. So you can go and check out some areas, some security relevant topics there. Yeah.

I guess you're probably thinking, why are we at a Civo event?

And I'll do a quick shout out for Civo, actually, because we actually use Civo for our infrastructure. So pretty much all of our stuff runs on Civo. We've got a massive data lake, which is processing around 660 billion log messages annually. And that's all the log data that gets fed in. So that's all being run on CivoKit. And then we also run the defense.com applications on Civo, as well as all our core apps. So we've been a very innovative company. We've always developed our own things. And so we run all that in Civo, because we've found that it's just super rapid. It's very easy to develop on. It's really kind of an innovative company. And that's what we really care about, is simplicity and innovation in the cybersecurity space. We've really found it a useful stack to deploy onto. And we also do honeypots on Civo. And we play around with other hardware and stuff on Civo, or other kit on hardware and stuff on Civo, as well. So that's why we're supporting the cause around the community, as well.