Episode 21 - Building AI Harnesses to Unify Detection and Response

Welcome to Corelight Defenders. I'm Richard Bejtlich, strategist and author in residence at Corelight. In each episode, we explore insights from the front lines of NDR, network detection and response.

Today I'm speaking with Jordan Hair, security engineer at Corelight. Welcome, Jordan. Hey, Richard. How are you? I thought it would be interesting to talk to you after seeing your work featured by OpenAI in a post they made to LinkedIn, and so we'll link that in the show notes.

But, uh, OpenAI wrote that, uh, "Corelight security engineer Jordan Hare built a custom harness now powered by Tech GPT 5.5 to minimize the time to respond to alerts, perform threat hunts, and develop detections with some fantastic results." So I'm wondering if we could... you could sort of explain that to me. Uh, maybe just first starting with what was the problem you were trying to solve? We had a lot of alerts.

An alert that would take a, an analyst, say, like, 45-ish minutes to complete. Say, you know, we have to run several, several searches for whatever tool sets that you have. You actually have to return those results, review those results, analyze those, make those mental connections for essentially what that story is telling you, put that together into a report, and either escalate, close, or tune the alert. So something that would take, like that, again, that would take about 45 minutes on average, but for the triage agent, all happens automatically. So instead of the analyst having to go out and retrieve that information, all of that information is now presented in a very nice, traceable, formatted way for the analyst to easily scan and make a very fast decision on whether something has to be done or not. It's very rarely wrong after doing a lot of tuning, and there was a lot of work that went into the tuning, obviously.

But it's able to do that and give us a better, uh, well, good answers and in a lot of cases, better answers because it's much more capable and just much more willing to do those extra steps of searches to build that additional context to tell that better story. This is work that you've done for defending Corelight, and because you personally had this problem of trying to make sense of all these alerts, you thought you would try to leverage this AI capability to help you with that. Yes, absolutely. I've used AI in a couple different capacities for either reviewing or helping to write code, uh, to try to understand some security data that I've looked at. I've used it to r- write some simple scripts to help me process data. But I heard this term harness, and it's always used and I never know what it means. Can you tell me what is a harness in the context of, of what you're, you're, what you were developing? Yeah, absolutely. So the way that I think of a harness is essentially just deterministic code wrapped around your AI model. So as we know, like, the, the AI models are pretty non-deterministic.

Um, so putting a harness around that, essentially a code wrapper around that, can help it do more deterministic things and also give it access to things to be able to do its job better. Um, so a few things that are typically included in a harness are things like tools. You have your custom prompts. You can also break out your agents into multiple agents, which kind of gets into context engineering.

Say you have a very large problem that you want an LLM to tackle. Um, instead of giving that entire problem to one LLM, give that problem to an LLM that has the ability to spin off other LLMs.

When you're talking about context rot in particular, is this something that you have to periodically revisit? Like, I would assume that the models would get better over time with that, or at least, like, the windows would get larger.

But I guess it's als- also possible that they might not. Like, they might be making a different trade-off where the, the window had to get smaller for some reason. So is this something that you periodically revisit? And if, if so, like, how do you do that? When I was first making this harness for the triage agent, the main model that was used was Gemini 2.5

Pro, and at the time, there is a l- there is just a lot of capabilities, um, that were just not possible out of the box that are now just se- second nature to the

LLM. When you upgrade the model to the next generation, it typically, it typically works out of the box. It's not like you have to go off and restructure everything for that model for it to work. It typically does work because it's typically a more intelligent mo- uh, more intelligent model. It's just you end up with a lot more junk than you would have before if you were, um, really kind of pushing the models to its capabilities and instead of using a bunch of band-aids to, um, try to get it to do what you want it to do, if that makes sense.

Yeah. So it sounds like the release of a new model then prompts you, or it's, it's like the minimum prompting that you would do to take a look at how it's performing.

Right now we're trying to figure out just how to make agentic harnesses work and do what we want to do.

I think obviously that's gonna become less and less of an issue as time goes on, and I think what is com- going to become more of an issue is just making sure that they're safe. Have you experimented with any local models?

And would there come a time where you say it would make more sense to try to run the models locally if we had, say, the sufficient hardware for that? Yeah. So I, I'm, I'm definitely interested in open source models, and I love the idea, and

I think it's going to be- I think it's going to become very important in the future and become, in some cases, even more important than the pr- proprietary models. But at least for where it stands for right now, they are, at least from what I've seen in my experimentation, they're decent for very small, well-scoped tasks. Can you give an example of something that you might have tried and the, the open source model couldn't handle it? So if I had a task to where, say,

I wanted to, say, make a tool that depended on an API, um, that's well-documented and pretty straightforward,

Quinn should be able to take care of that. But if I'm asking it to do more so like, "Hey, um, take a look at all of this information and give me your... Like, make connections between if you see initial, initial access followed by, um, privilege S. followed by lateral movement or some combination of those things, raise a flag," um, it's going to have a much harder time at doing that.

Um, why? I' mean, I'm not there training the models themselves. I can guess that it's because the use case for the model, and obviously they're limited on resources compared to these other proprietary models, that they're go- that they're really focusing on it just being a coding agent. And coding is very generalistic, but it's often, like, using that very strict lens to make all of your decisions can be inefficient or just wrong. I can kind of imagine how the alert management or triage and all that would work. Can you talk to me a little bit about, uh, how you leverage these models for threat hunting? The threat hunting agent should be able to take threat intel, some form of threat intel, let's just say it's MITRE for right now, for an example. Um, it should be able to see what coverage we already have, so obviously it's not making things that we've already created. Um, it should see what's capable out there from the threat intelligence, typically from MITRE. Um, and then it should write and run queries to actually, um, review that activity to see if it's happening in our environment.

So typically, it's running those queries. It will review the results, and, um, this is where, uh, tuning typically comes in. So let's say that you're trying to make an alert for a abnormal user creation.

You know, obviously if there's certain patterns and certain activity that is known to be expected, you don't want to have a critical alert on that. You may want to have a signal on that, so of course, you know, you can look at it and take a deeper dive into it. But you may not want that to actually be, like, a critical, "Hey, look at this right now," um, signal. So the threat hunting agent has certain context in order to be able to make those de- uh, decisions to be able to see if, hey, if, you know, if results are, returning from this query, is this actually safe? Is this actually most likely a normal activity, or is this something that we should look into? There's no actual actions that are taken on this. Everything is read-only for now. Um, but essentially at the end of that, it will gener- uh, generate a report, um, with all of its findings for a human to ultimately decide what to do with. And what that looks like is typically the human or me, I'll look at it, I'll read it, I'll see if it makes sense. I'll see what, um, the tuning, like, what types, types of tuning it would, uh, it did. And it basically is just kind of like a conversation with it there. So are you? the person who, if you find something, whether, you know, through any of the methods that we- we've talked about, do you have to do something with it? Like, are you notifying an asset owner or that sort of thing?

There's a lot of things that are heavily automated in this pipeline, but, there... it always comes down to a human-in-the-loop verifier. So part of the architecture is actually trying to figure out where the best parts is to have the human in the loop actually go in and verify these things.

So, so it's never, at, least for right now, and we would not even get to this phase until we had very, uh, uh, very high quality proof that essentially there's, there's no reason to have a human verifier in that particular point of the loop anyway. But essentially, how it happens is every alert that's triaged by the agent is reviewed by a human. Every threat hunt that is, um, produced, uh, by the agent is reviewed by a human, and the same with, um, detections and basically every critical point, um, to where a change is going to be made in production or, you know, some big impactful thing could happen. It's always reviewed by a human. In the beginning, like a million years ago, y- you were doing the job yourself. All the things that you were talking about, you had to do. Right. There wasn't AI, but you know, you cr- you did the detection engineering, you did the detection, and then you had to do the response. You had to do something with it. Over time, those functions got more and more specialized, and now you had a detection engineering team, you had a detection team, you had a response team.

And so there was an incentive, not... no one was being malicious by this, but in some ways, you know, the detection engineering team just threw the detection over the wall, and the people who had to deal, you know, they got tho- those alerts, and then they sort of did the best they could and threw them to the responders. But it's all unified back in you now having to do this job. So you can tell what works, what doesn't work, and, and try to improve it. Yeah. That, that's a great, that's a great point because that is... That's exactly what happened. Um, you know, uh, that's just kind of like the nature of, you know, the

SOC. You know, you have your detection team, you have your threat hunting team, or however you decide to, um, divide it. And, you know, hopefully, you know, you... that relation is, is good, but there's always going to be some type of friction. You know, there's gonna be people throwing threat hunts, over that the te- detection teams can be like, "Hey, what, what...

You know, what's this?" And, you know, stuff like that. So, um, yeah. That, that, that's how it was. And each team kind of had their own, like, little agen- like their little things or action items, and they're all kind of, you know, together but separate.

Um, but after doing this, um, it really did become unified. Like, having that separation of IR detection and threat hunting really became... It didn't make sense anymore.

Because it m- it, it, it helps so much if you're able to, to have the full context of, hey, beginning a threat hunt all the way to the incident response phase. If you own that from beginning to the end, th- that's the best because you have all the context of why it was there, why'd you put it there. If there's an issue with it, it's much easier to fix it because you don't have to go through this back and forth of, you know, "Hey, this happened. Did you mean to do this?

Blah, blah, blah, blah, blah." You, you just go and fix it. So that's exactly what happened here. We were much more of, like, the traditional, um, SOC to where you have, you know, that IR function, you have the detection function, and the threat hunting, um, function.

But now after making these agents, it really doesn't make sense for everyone to be working in those separate functions anymore, and now we're doing everything.

So I own everything from, uh, the beginning to the end, and anyone else on the team also does that and is doing it with the agents, which before the agents just would not have been... It, it just wouldn't have been possible.

There, there's just too much work. Um, so it went from basically having t- like needing to separate into separate teams because of that to, um, just not having to do it anymore, and now we don't. So...

And it's much, it's much nicer. The, the, th- the ability to have the full context of the entire story, it just makes everything move much faster.

There's way... There's basically zero friction because you're just... You're working on things that you already made, so if you need to make an adjustment, it's ve- it's very quick and easy to do. Um, but yeah, overall, it's been a very good, useful experience,

I, I, I would say. Well, that's a great place to end. Uh, I hadn't anticipated that, 'cause when most people talk about this topic, they talk about automation and reducing the amount of grunt work. But the idea of unifying that whole pipeline, uh, within a person who can feel responsible and able to act upon what they're doing, I think that's just wonderful.

So, uh, Jordan, thank you for joining me today on the, uh, Corelight Podcast. Yeah. Thanks a lot. I really appreciate it. Thank you for joining us on the Network Defenders Podcast, sponsored by Corelight. We will see you on the network. You've been listening to Corelight Defenders. To stay informed with expert intelligence on today's cybersecurity challenges, please subscribe to ensure you never miss an episode. We'll see you on the network.