Episode 18 - Live Fire Defense at Locked Shields

Welcome to Corelight Defenders. I'm Richard Bejtlich, strategist and author in residence at Corelight. In each episode, we explore insights from the front lines of NDR, network detection and response. Today I'm speaking with Adam Donadio, senior sales engineer, and Nico Roosenboom, senior sales engineer. Welcome to the podcast.

Thanks for having us. Uh, recently I heard that you both participated in the Locked Shields exercise in Europe. Uh, Nico, I think you have a little more, uh, history with this. Can you tell me what is

Locked Shields? It's, uh, world's largest and most complex international live fire defense exercise, so it's multiple blue teams, um, protecting their, um, I would say isowed, uh, isolated, uh, game net, uh, which is completely virtual of course, and then a very small red team is trying to at least, um, have multiple waves of attacks, shooting at them, trying to see if they can hold their ground. And the great thing is, um, we've been invited a couple of years ago by a very friendly customer who's, um, competing a, a couple of times, uh, in, uh, in one of those blue teams. So a blue team is considered to be primarily handled by the government and military organizations, and they, well, form alliances, right, with different, uh, other countries and nations. So it's not only NATO-related, but also the friendly, NATO-friendly country- countries are participating in this as well. Could be, let's say, between the 200 to 400 people, uh, in just one, uh, one blue team, but the exercise as a whole is, well, that's a, an awesome exercise to, uh, to compete in and, um, to, to at least learn from the different, uh, people in there as well, so that's, uh, that's great. Can you give me a sense of like how many people might be involved with the whole thing? Yeah, so it's, it's, it's over 4,000, uh, cybersecurity experts and approximately, uh, more than 40-plus nations involved in it as well. Wow, I had no idea.

Do you guys set up your own network? Yeah, so basically, uh, most of the blue teams, they, uh, agree to have one of the countries as a guest country, right, to host the, at least the exercise, um, for that particular blue team, uh, to be at and to, to play along.

Um, so every blue team gets his own game net, which is basically a virtualized environment, and that virtualized environment is the same for every other blue team, and that's being handled and managed by the different, uh, well, by, by the green team, which basically handling the infrastructure and making sure that everything is being set up properly. Um, and well, you basically have to work with the stuff that you have available, right?

So it's not the luxury that you can install your own virtual sensor into the network or, uh, put a hard... Well, you can even, even place any hardware in the, uh, the game net at all. Uh, so basically it's working with the stuff that is there already, so with the tapping infrastructure that's there already, with the component, the network components and the, the host components that, uh, are installed in there already. So w- what is it that Corelight brings to this exercise? What we've done is we've gone into the exercise with our software, uh, sensor, so basically you can install it as a piece of software int- on, uh, an existing, uh, Linux-based operating system.

Uh, so that's what we've done for this exercise as well, installed, uh, the Corelight, uh, sensor as a software sensor, linked it, uh, to the centralized management platform, which we call Fleet

Manager, and then started off with that, uh, sensor to, well, to, to at least collect all the network information that is flowing in, right, st- and start building from there. I wanna dig in more onto that, but Adam, if I understand right, this was your first

Locked Shields? My role is primarily, uh, a administrator of our platform, so I help to configure, uh, configure that, stand it up, um, all those components, and then also part analyst, uh, to interpret the data coming out of our platform as well.

So kind of a hybrid role. So what were your impressions of this? 'Cause I know if I had, had attended, I would have probably been blown away by what was going on. Yeah. There was, uh, it was a lot of, a lot of information to take in all at once.

You basically gotta learn a, a brand-new network very quickly, um, and then also learn the current capabilities very quickly, which we're, we're not the only, the only solution, uh, on, on site there. So working with a lot of different other teams and other vendors, uh, that have also been, um, uh, w- were also guests, uh, to come, to come, um, uh, do this, this exercise with them was very interesting, and it also kind of allowed me to see, uh, I guess, again, what I saw from a, a security engineering role that I've had in the past is you have to play nicely with everybody else around you so that you can, uh, uh, make your configurations work with their configurations.

Can you say a little bit more about that? Like, were we integrating with, uh, like endpoint products or things like that? Uh, what we struggled with initially in the first couple of days, uh, while, while we were... Uh, we, we had missed the first week, which is where, you know, they would typically have you come in and do the initial configuration of, of devices. Uh, we were not there for that, so we showed up basically game week, and on Tuesday we went in. Uh, game net opens. They allow you to, to walk in and do all, all, all of your configurations for, uh, sensors and that sort of thing.

What, what we struggled with was after we got the sensor running, it would not communicate with the fleet manager, and as it turns out after, after several hours of troubleshooting, uh, we ended up having a accidental push, um, of a EDR agent to the platform that we were, that we were using, um, and that caused a series of different problems.

So I guess recognizing that, um, sooner probably would've helped us out. A- and then also, you know, it comes back to the, uh, age-old adage of, uh, trust but verify.

Uh, after we'd been assured multiple times it wasn't the case, you go do your own research and figure out that, you know-These are things that you have to worry about, um, in an actual environment that's not, you know, just a lab per se.

Yeah. That's the key. When you're dealing with the real world, there's all kinds of friction that you would not anticipate and assumptions that are made, and it can take a lot of work to figure out what is actually going on. Nico, can you tell us why Corelight was invited by that friendly customer and was eventually, I guess, so important to this exercise?

Yes, of course. Uh, so basically they invited us to the exercise because they were already used working with the product in their production environment of course.

They, um, wanted to integrate with, uh, another alliance partner that we work with, uh, pretty often, at least in the area where I work with, um, the most, and that's in, uh, Northern Europe. So basically we ask like, "Hey, how can you guys integrate with, also with Elastic?" Because we're working with Elastic as a SIEM. And, uh, it worked out pretty well, right? Because that's one of the most interesting stuff, uh, that, uh, that Corelight can do, is it's transparent, um, in its foundation, so basically very open, uh, in collaborating with other technology stacks. So the people who participate, they just get whatever technology is present. Like you said, they don't bring their own stuff at all? You can, but, uh, you must be aware. So basically it's, uh, an environment with a couple of existing firewall brands already out there. But, uh, basically, uh, for EDR, you have the freedom of choice, right? Uh, same for the, uh, for your SIEM solution. So if you want to bring in, uh, source solutions as well.

So basically every blue team has to decide, "Okay, what is the, the stack of tools that we are going to use for this particular exercise?" Can either of you tell me how did the exercise unfold? Did anything cool occur?

Our log set was, was being utilized from our perspective within the Investigator platform. We could find things like DNS ex- exfiltration, um, and I found a lot of this within, within SCADA segments.

So while we were looking, uh, through, uh, threat hunting dashboards and trying to identify things that were maybe anomalous with respect to the rest of the data that we were getting, it became very easy for me to pick those things out and say, "Well, that doesn't look right."

And then go track down the person that was, that was responsible for that segment and, and start asking questions. Uh, in which case we would come back and provide them, uh, things like port data so they could try to track it back to a specific service. Once we identify that service, we kill the service.

Um, and then beyond that, um, trying to rectify the situation, uh, was a little more challenging. Can you tell us a little bit more about what enriching logs means? Oh, absolutely. Yeah. Uh, so we actually had support from another, uh, labs participant from the Corelight side. Uh, David Burkett was helping us out with, um, uh, actually it was a, it was a older pack I think that Nico utilized, uh, in previous years, uh, that we took and we modified that and expanded upon it, uh, from the last year's version of that to include things like, um, the extensions for VLAN segments and the inner VLAN tags, um, uh, cert ages.

Uh, we extend certain other log components to others like ex- e- extending the local originator and responder, um, fields to all logs. Um, and then we also did network and host enrichments. So as we were provided a list of things like, uh, what team did, did a, uh, specific device belong to, whether it's the green team or the blue team, uh, uh, what was the host name, what's the operating system, what network segment does it belong to, i.e. SCADA, host and role descriptions, capability groups, uh, DNS, uh, domain name, interface names, which sometimes they were, they were, uh, different, but mostly the same. Uh, fully qualified domain names, egresses, um, whether they were allowed, uh, true or false components, that sort of thing. So a pretty large swath of different enrichments to what we, we would consider, um, our, our starting log set. Yep. So was this stuff that you all customized on site, or did we have some of it ready to go, or how'd that work?

Uh, we, we did some of it as pre-work, um, before, before we came out, so that would be the week prior, and then we did some of it on site as we were getting fed documentation on some of these hosts, um, uh, such as the OS's and the, and the descriptions. So we, we did a little bit of both. Um, but we actually formalized that on Tuesday and plugged it into the configuration,

I believe Wednesday morning as we were getting started. So for exports, were we sending our data to several locations? We were, yep. We had three different locations. Uh, Investigator was one of those. The SIEM platform was, uh, what was a second one, and then the, uh, third one was more of a, um, a kind of a detection platform that would try to pull threads on, on whether we thought there was, uh, a, uh, there was command and control traffic or not.

Um, and that person specifically sought us out and said, "Hey, I've heard that we can do this with your platform. I would like to try it." So that's why we set that up. That's awesome. That's something I've always liked about our offering, is that I, I'm secure copying the raw log files to a store that I keep here in my lab, but then I also put logs up into Investigator, and just being able to do those two things is, uh, so nice from a, a retention and an investigation standpoint. Nico, I wanted to ask, 'cause you've been involved with this a little bit longer.

I'm guessing that at the beginning of our involvement, we didn't really have Investigator, or it certainly wasn't the Investigator we have now.

Um, can you talk a little bit maybe about how Investigator has been used or was used, I guess, in this recent exercise? We introduced them to the Investigator platform as well as, as something, um-That is used by us, so if we, let's say, like last year, we bring in a couple of, uh, our own people to help in the threat investigation and the incident response.

part, then yes, uh, we work primarily on the investigative part. But it also piqued the interest of the people, uh, sitting close to us. and see the platform as well. And the feedback that I've received is like, "Hey, it's, really easy, um, to use your platform to find the information as well, and it's very speedy." Uh, so especially if you want to get information out of it, the speed of getting the results after, um, querying for the data, that's, uh, considered to be, uh, an eye-opener for some of the people as well. Were you able to use threat intelligence in this exercise? Because I'm assuming the red teams, are all, you know, they're coming from custom IPs and all that sort of stuff, right?

Yeah. So basically, if you look at, let's say, some of the JA3 hashes, um, have been very valuable during the exercises.

I would say, uh, f- some of the file hashes as well. Uh, of course, because they use an emulated form of internet, there's no real internet involved, of course.

Uh, so I would say the indicators of compromise when it comes to the real bad IP addresses is not really useful. Um, but there's plenty of information that we can extract from, uh, their, threat intelligence databases that, uh, have been proven really valuable in this exercise as well. So they were using tools that may have already been seen in the wild so that we would have some way of identifying them, using the threat intelligence. So Adam, you mentioned SCADA networks, so that makes me think that there were both enterprise targets and there were ICS or non-enterprise type targets. C- can you say a little bit more about that? The way that GAMET is, broken down is there are three different organizations. Um, Nico may have to keep me honest here. Uh, one of them was armed forces, uh, one of them was critical infrastructure, um, and then

I think the other one was more public sector, if I recall correctly, Nico. Mm-hmm. Correct. Yeah. Uh, so the, the, uh, critical infrastructure one was, was the one that had the SCADA networks on them, and, uh, you know, doing a lot of that pre-work as we' show up, um, that week, going in Monday morning, uh, and asking questions to that team as in, like, what protocols do you expect to see, what can we add in, uh, possibly, uh, as value adds here, was one of the, one of the things that I, that I, I had, um, a follow-up, follow-up item listed. And it was interesting talking to the administrators of those platforms. There were three primary protocols, um, and one of which actually stood out to me more than others simply because we have, um, uh, um, folks on our side who' are, who are busy creating packages around SCADA prot- protocols currently.

Uh, so Dr. Smoot reached out and, and actually asked, uh, what we expect to see, and he had already created, uh, Modbus packages, so, uh, specific Modbus detection content that we also pulled into a, custom bundle and deployed within this environment, uh, of which I will say did actually trigger one time at the very end of day two.

Unfortunately, right, near the end of day two of the live fire, uh, live fire event, so, um, we did not get the, get the opportunity to- go and, uh, and qualify that as a true positive or a false positive. But, uh, it was, it was quite interesting.

Yep. And I would imagine that in an environment like that, a passive network sensor like Corelight is gonna be, a star, because you're not gonna put an endpoint agent on a lot of these pieces of equipment that' just don't support fiddling with or even exporting logs, I imagine. Thank you both for joining me on the podcast today. I had no idea that' it was such a large exercise or so chaotic or, uh, complicated, but I think that' is the sort of exercise that you need in, uh, in NATO and in today's world. So I appreciate your, your views and, uh, sharing what, what you experienced during the exercise.

Thank you. Happy to support. Yeah. No, no problem. Thanks for having us. Thank you for joining us on the Network Defenders podcast, sponsored by Corelight. We will see you on the network.

You've been listening to Corelight Defenders. To stay informed with expert intelligence on today's cybersecurity challenges, please subscribe to ensure you never miss an episode. We'll see you on the network.