Episode 19 - The Cap on Inference: Proving How Network Data Quality Drives AI Security ROI
Episode 19 - The Cap on Inference: Proving How Network Data Quality Drives AI Security ROI
0:00 / 0:00
About the episode
In this episode, host Richard Bejtlich sits down with Corelight Co-founder and Chief Strategy Officer Greg Bell to unpack groundbreaking research that quantifies exactly how data quality impacts AI-driven security automation. Moving past qualitative industry hype, Greg shares hard evidence from an empirical experiment pitting leading AI agents against real-world Capture the Flag (CTF) challenges and incident response report writing. The findings reveal a dramatic truth: basic firewall and flow logs place a hard cap on inference, throttling an LLM's capacity for deep insight. By upgrading to higher-fidelity, densely linked network data, teams can boost automated threat hunting accuracy, eliminate costly LLM hallucinations, and dramatically reduce token consumption budgets by enabling security agents to solve problems twice as fast. Read the full research paper: https://corelight.com/blog/data-quality-limits-ai-soc-performance
Episode transcript
Download transcriptEpisode 19 - The Cap on Inference: Proving How Network Data Quality Drives AI Security ROI
Welcome to Corelight Defenders. I'm Richard Bejtlich, strategist and author in residence at Corelight. In each episode, we explore insights from the front lines of NDR, network detection and response.
Today, I'm speaking with Greg Bell, co-founder and chief strategy officer at Corelight. Welcome back to the podcast, Greg. Thank you, Richard.
Thanks for having me back. You bet. The topic today is provably better data, and this is a research project that Corelight conducted to try to quantify the effectiveness of different types of network evidence. Is that a, a fair assessment of that project?
Yes, and, um, might be worth telling our listeners that the project began, um, with an impulse of curiosity. Of course, we've all been eagerly tracking the progress of AI generally, and the application to security workflows specifically in our line of work. And when it became apparent, I think maybe last year at Black Hat, that agents were doing a really good job competing in Capture The Flags, and in fact, better than some humans, we began to get curious about whether we could run an experiment related to that new capability.
It's important for our listeners to know that if you find this interesting, we have a whole research paper that provides the background for this. And one of the items in that research paper was that we do 100 or more CTFs per year. And so this wasn't just something that Corelight decided, "Oh, well, this might be interesting." We, we do this a lot, and connecting the CTFs that we do with the agent angle, I think was a pretty, a pretty neat starting point. That is exactly how it started. And honestly,
Richard, we had no idea what the answer would be. We knew that we ran a lot of CTFs, and they're a fun way for our customers and prospects to get familiar with our data in a, um, mildly competitive gamified environment where everyone, even if they feel a little bit of resistance at first, they get engaged in it, and they're sort of first forced to learn about, uh, Corelight data, Zeek data, Suricata data, and the process of incident response. and threat hunting.
Um, as you, said, we run a bunch of those, and as a result, we have a lot of data, tracking how human beings do in those gamified competitions, what kinds of questions they typically ask, in what order, what mistakes they make. And all that data is proprietary. We've never released it to the world, and therefore, it's not in the training data set of any large language model. So we thought we would use that data set and our expertise in, in designing Capture
The Flags to put agents through the same paces that we put human beings through as well. When I think about it, I don't know if anyone has done any real research yet on quantifying how effective agents are.
You see, you see periodic reports that agents were able to complete these CTFs, uh, faster or, and even sometimes better than humans. But, uh, in this case, we, we did-- I guess it's kind of a twofold experiment. One was throwing agents at these problems, but also evaluating what types of network evidence. would help them answer the questions most effectively. My colleague
James Pope and, and, some of his colleagues in our technical marketing engineering team built a fully agentic platform that let us do just exactly as you described. And it's quite a thoughtful platform.
In the research paper you mentioned available on our website, we' go into a lot of detail. So if you are curious about how to build an agentic platform like this and what pitfalls there might be, um, and what best practices, um, we might have observed, then feel free to dig into the white paper. It has a lot of detail.
And so we built, or my colleagues built the platform in such a way that we could change two things. We could change the model being used to explore the Capture The Flag data. So we could switch easily between
Anthropic, uh, models, OpenAI models, Google models, and open source models too. The other thing we, we kept as a variable, we made it very easy to change the source data available to the agents. Everything else was fixed. The methodology, many, many steps that we described in the white paper to assure fairness and to remove bias.
All that was, I think, thoughtfully designed and fixed. And we were able to run the experiments by ch- and change out the agents, uh, and the models being used and the data sources being used.
And there were a number of surprises. I wasn't sure what we, what we would expect, but we got-- we were surprised in a couple of different ways. So before we get to those surprises, which I assume are the conclusions, I wanna mention that we didn't skew the questions towards ones that were best answered with our data. Uh, from my understanding, we looked at what are the types of questions that are answered in industry-standard reports from Mandiant or CrowdStrike or other incident response investigators, and came up with a question set not only for this experiment, but also sort of based on our experience running CTFs.
These are the questions that needs to be answered if you're trying to determine the scope of an intrusion, figure out what happened. Is that, is that true as well? That's exactly right. And the research paper describes in a lot more detail what steps we took to make sure that the test was fair. Um, but, but the fundamental, um, step we took was just to reuse the same Capture The Flag exercise that many, many humans had competed in, and that one happens to have been focused on a Volt Typhoon threat actor. There was a second experiment, um, un- only partly related to the first about incident response reports.
And rather than describe that in detail here, I think I'll direct, uh, listeners to the white paper because it's a little bit more complicated.
But in every respect, we tried to assure fairness, and there's a lot of detail, even technical detail, about how we configured the tools we were using and steps we took to make sure that the agentic system wouldn't be biased in favor or against Corelight or any other data source. Great.
Well, can you describe a few of those conclusions then? Well, the first conclusion that surprised me a bit, and I was really curious about, was how well the models are doing these days. I think six or nine months ago, a year ago, there was a lot of discussion, uh, and lamenting of the tendency of agents to make stuff up. So, uh, they would hallucinate, they would invent data if needed, they would sometimes claim to do things they didn't do, and the list goes on and on. I think we're all familiar with that.
What we discovered based on, uh, tests a little bit earlier this year, just a couple months ago, is that the hallucination tendency was way down. If agents were presented with a hard question and data that simply couldn't be used to answer it, they would, they would stop. They would report that fact rather than making up a fictional answer.
So that was, that was pretty interesting, and it's great for the prospect of automating some aspects of cybersecurity defense with AI. It's terrific. The other thing we noticed, again, on the agent side, is that there are some differences in the agents in terms of their performance and their speed, but overall, they're all working at a similarly advanced level for the task that we gave them. And, and that's terrific. It was a little bit unexpected. We thought maybe there'd be one agent, uh, at one moment in time that would be drastically better than the others. There were some differences, uh, but generally, the models from different companies performed about the same on the Capture the Flag. So that's one set of surprises related to the models. The really dramatic results, um, though c- came when we tested how well did the agents do when presented with different kinds of source data. And, and I know, Richard, you spent a lot of your career thinking about the necessary forms of data that, that a SOC needs to have. And I'm curious, before you heard the results of... Did you have any instincts about, um, what impact source data might have?
I, I had some, but I, but I had a lot of questions as well. Yeah, I mean, just from a basic standpoint, if you're taking a look at, say, flow data, you're limited to source IP, source port, dest IP, dest port, protocol, timestamps, amount of data transferred, and that's it. Uh, you're not gonna get usernames, you're not gonna get these other important issues that are, that are captured in other forms of transaction data like you might get from, from Corelight or, or Zeek. Uh, and then in, in terms of intrusion detection alerts, those are biased towards whoever wrote the signature for that. So if you're specifically looking for, for behavior and you come up with a signature for it, great. It might tell you,
"I believe this, this has happened." But in terms of anything that does not have a signature associated with it, well, you're just not gonna get any data whatsoever.
So- Mm-hmm ... uh, that's why I've never said you should only have one. Um, if I had to choose, I would have se- you know, the rich, uh, transaction data plus the other forms of data you get out of, out of Corelight. Um, and that's, I, that's probably an important point we should talk about Corelight as well is
Corelight is not simply Zeek data. Um, many people might think that it's just, you know, Zeek on a box, but that's, that's actually not true.
But yeah, that's, that's sort of my initial, uh, my initial bias going into this is, is to expect that you would get richer results from richer data. Yeah, I had this exa-- Well said. I had exactly the same expectation. I just didn't know how to quantify that expectation. And, and honestly, Corelight logs improved Capture the Flag scores by over 350% compared to NetFlow logs, and over 60% compared to firewall logs. On the other experiment, which we haven't talked about very much, but on the incident response report writing task that we also gave to the agents, Corelight logs provided evidence for about 50% more IR findings compared to firewall logs, and about 300% more compared to NetFlow.
So NetFlow and firewall logs are good and much better than nothing, but the difference between a 60% attainment on a Capture the Flag and 100 or
98 could be very, very significant in the context of a real SOC managing real risk. And it was so interesting for me, having made this argument qualitatively for so many years, it was so interesting to finally get hard quantitative proof of the impact. Uh, another way I'm explaining this to customers and prospects is that each data source sets a kind of ceiling for what can be known. In the AI vocabulary, you can think about that as a cap on inference. Inference is the work that an, a la- language model does to answer a question. And a, a given data type, like firewall logs, create a cap on inference, just a, a cap on the sophistication and insight that any model can have simply because of, uh, deficiencies in the source data.
So, uh, that's the, i- in a nutshell, those are the results that, that Corelight data produce much better outcomes compared to the other alternatives.
And a- again, for the skeptical or dubious or curious, um, we explain this in great detail, and we're also, of course, very happy to talk about it and invite dialogue around the findings as well. I do definitely appreciate their quantitative approach, but for me, the qualitative is, is also really important because that captures questions that you simply can't answer using the other data types. And
I think we were able to put some answers around that, right? Uh, upwards of 90, high 90% questions could be answered using Corelight data, whereas the other data types just couldn't answer vast numbers of questions. Yeah, that's right. There's just certain kinds of questions, and you could imagine a question like the name of, um, a w- a Windows file share or a DNS reply.
Uh, you can just imagine a whole series of questions that could be critical in an investigation but just aren't available, um, within the schemas of those other data sources.
I think we found, I'd have to go back to the report to confirm, but that in most of our runs, Corelight provided all of the data necessary to answer the questions. Sometimes the, the runs would time out, um, just because we set an artificial limit on token count available for each question.
But that raises another important dimension of performance, we found that if you take a question that all the data types are capable of answer, the Corelight data was enabling the models to answer about twice as fast because they were able to get to the answer quickly with fewer iterative loops. And so not only can richer data help agentic systems answer harder questions and more questions, but they get to the answer much faster, which makes a difference, I think, in the era of post-mythos, uh, automated AI attacks.
That is such a great point because we're seeing these stories now of companies that have burned through their AI budget in the first four months of the year, and now they're wondering what they're gonna do next. I- I've spoken with people who work at some companies who they're in their budget cycle right now, and they're trying to forecast their AI spend for the coming year, and, uh, this is a real constraint.
Uh, we, we just can't assume that you're gonna be able to spend as much as you want on this and get the results that you want. You have to be efficient with your, your allocations. Yeah, absolutely. We're hearing this. We heard it discussed at our last board meeting just a couple of weeks ago.
I know I've spoken to Russ, our CFO, and he's hearing the same discussion among his circle of CFOs. It's as if there's been a, a little bit of a hangover for token consumption, and, um, when we have to look at the question of cost benefit of token consumption, you know, in the cold, hard light of day. Uh, and I think every organization that went hard at AI solutions is discovering that in some cases, tokens can be more efficiently spent. So I think it's a good message to get out there that higher quality, high fidelity, well-structured, densely interlinked data helps AIs get to the answer, a-agentic systems get to the answer faster, more efficiently, and cheaper. Well, well, let's talk about that agentic triage angle for a minute then, 'cause what this tells me is that you can have agentic triage, but if you're not feeding your agent quality data, then it should not be coming up with actionable, um, insights. In other words, it'll say, "I suspect," or, "There could be something happening here, but I don't have the confidence because I don't have quality evidence." Whereas I know with our own agentic triage element for Corelight Investigator, it's got good data to work with, so it can be more confident and show its homework when you're querying it as to how it came to a conclusion. It's absolutely true. If you start with poor quality data, you could end up with a triaging system that wastes both money, but al- just as importantly, wastes viable human attention. It puts humans in the loop or on the loop, um, to resolve, uh, questions that really can't easily be resolved with the underlying data, and, and that, that is just unproductive spend of tokens and unproductive heat dissipation in the data center. So it's better to start with very high quality data that has a chance of answering the questions. That will produce much better outcomes, lower costs, and more efficient agentic SOC projects. Would you happen to know if we have participants in our ongoing CTFs who are using agents in order to complete the
CTFs? Y- you're wondering if they are using them somewhat surrep- surreptitiously on the side- ... to as to, um, to supercharge their results?
Yeah. Yeah. I'm wondering, like, are, are we getting to the point where we'll just y- if you're not u- like, is this gonna be like the steroid Olympics, where if you're not, uh- ... using performance enhancement, you're just never gonna have a chance against everyone else who's participating? I think there's t- two dimensions to that. One is whether anyone's doing that a little bit secretly on the sly. And to the extent that in some cases they're physical, some cases they're virtual CTFs, I guess we can't rule it out. Uh, I'll have to talk to our CTF team to see if they put in place measures to keep people, um, wholly human-centered in their CTF performance.
But the other is more about, um, a future workflow where agents become efficient, um, really fast, specialized coworkers. And, and, and I think everyone expects that to become the case, and coworkers and delegtees over time.
As soon as, as soon as we can assure that, um, we can trust their results and their actions. And of course, many-- The security industry is evolving very rapidly w- um, to, to meet the implications of that assumption.
Lots of different companies are thinking about dimensions, security dimensions of that future architecture, including Corelight. I think it would be neat if we, if we just assume people will use it and they come up with results, but then maybe we also have a special category where it is j- just the human. Uh, you know, there might be some technical, uh, constraints that you could put around it.
Even, you could even do a, a human constraint where the people are working, you know, maybe there's only a s- small number of people agree to this, and you have proctors who are walking behind them, making sure that they're not using an agent or whatever. And I think that would be a f- a fun kind of contest to win, you know, the, the only human, uh, CTF, uh, response winner. Right.
Um, u- unassisted human CTF, it's a great idea. I'll definitely, um, pose a couple questions on Slack after this discussion. But what a, um, amazing indicator of the cultural whiplash we've experienced,
Richard, that, you know, I don't think we anticipated a few years ago that we would need to set aside certain CTFs maybe for human-only, uh, uh, security athletes. But that's- Right ... I think the world we live in now.
Yeah. Do you know if, if we're doing any follow-up research or what might be next for this project? We-- I'm certain we will. Um, we've structured the framework or the, the agentic test harness so they can easily be extended. And the first thing we did after publishing the research paper was look to see if we could run some of the leading open source models, and especially those designed to run, um, quite efficiently and pr- produce the same results. And when we ran those experiments, and this isn't reported in the paper, but when we ran those experiments a couple of months ago, we had limitations, especially on context window size, but also some other limitations that we encountered.
But I'm very curious to know in three or four months if the open source models of, say, October,
November can, um, meet the performance or even best the performance of the proprietary models of March or April. I, I would expect that to be almost true, but we'll find out. I'm very curious about that. Well, Greg, this was a really interesting topic, and it's, it's one of these topics where I'm sure, like you said, in a few months or six months, we're gonna get some different results. So, uh, I'm sure our, our wizards will be continuing to work on this. So thank you again for joining me on the Corelight Podcast to talk about, uh, what agents can do for you for security and how Corelight data is the, is the fuel that will, that will give your engine the right outputs. It's always a pleasure. Thanks so much, Richard. Thank you for joining us on the Network Defenders Podcast, sponsored by Corelight. We will see you on the network.
You've been listening to Corelight Defenders. To stay informed with expert intelligence on today's cybersecurity challenges, please subscribe to ensure you never miss an episode. We'll see you on the network.
Today, I'm speaking with Greg Bell, co-founder and chief strategy officer at Corelight. Welcome back to the podcast, Greg. Thank you, Richard.
Thanks for having me back. You bet. The topic today is provably better data, and this is a research project that Corelight conducted to try to quantify the effectiveness of different types of network evidence. Is that a, a fair assessment of that project?
Yes, and, um, might be worth telling our listeners that the project began, um, with an impulse of curiosity. Of course, we've all been eagerly tracking the progress of AI generally, and the application to security workflows specifically in our line of work. And when it became apparent, I think maybe last year at Black Hat, that agents were doing a really good job competing in Capture The Flags, and in fact, better than some humans, we began to get curious about whether we could run an experiment related to that new capability.
It's important for our listeners to know that if you find this interesting, we have a whole research paper that provides the background for this. And one of the items in that research paper was that we do 100 or more CTFs per year. And so this wasn't just something that Corelight decided, "Oh, well, this might be interesting." We, we do this a lot, and connecting the CTFs that we do with the agent angle, I think was a pretty, a pretty neat starting point. That is exactly how it started. And honestly,
Richard, we had no idea what the answer would be. We knew that we ran a lot of CTFs, and they're a fun way for our customers and prospects to get familiar with our data in a, um, mildly competitive gamified environment where everyone, even if they feel a little bit of resistance at first, they get engaged in it, and they're sort of first forced to learn about, uh, Corelight data, Zeek data, Suricata data, and the process of incident response. and threat hunting.
Um, as you, said, we run a bunch of those, and as a result, we have a lot of data, tracking how human beings do in those gamified competitions, what kinds of questions they typically ask, in what order, what mistakes they make. And all that data is proprietary. We've never released it to the world, and therefore, it's not in the training data set of any large language model. So we thought we would use that data set and our expertise in, in designing Capture
The Flags to put agents through the same paces that we put human beings through as well. When I think about it, I don't know if anyone has done any real research yet on quantifying how effective agents are.
You see, you see periodic reports that agents were able to complete these CTFs, uh, faster or, and even sometimes better than humans. But, uh, in this case, we, we did-- I guess it's kind of a twofold experiment. One was throwing agents at these problems, but also evaluating what types of network evidence. would help them answer the questions most effectively. My colleague
James Pope and, and, some of his colleagues in our technical marketing engineering team built a fully agentic platform that let us do just exactly as you described. And it's quite a thoughtful platform.
In the research paper you mentioned available on our website, we' go into a lot of detail. So if you are curious about how to build an agentic platform like this and what pitfalls there might be, um, and what best practices, um, we might have observed, then feel free to dig into the white paper. It has a lot of detail.
And so we built, or my colleagues built the platform in such a way that we could change two things. We could change the model being used to explore the Capture The Flag data. So we could switch easily between
Anthropic, uh, models, OpenAI models, Google models, and open source models too. The other thing we, we kept as a variable, we made it very easy to change the source data available to the agents. Everything else was fixed. The methodology, many, many steps that we described in the white paper to assure fairness and to remove bias.
All that was, I think, thoughtfully designed and fixed. And we were able to run the experiments by ch- and change out the agents, uh, and the models being used and the data sources being used.
And there were a number of surprises. I wasn't sure what we, what we would expect, but we got-- we were surprised in a couple of different ways. So before we get to those surprises, which I assume are the conclusions, I wanna mention that we didn't skew the questions towards ones that were best answered with our data. Uh, from my understanding, we looked at what are the types of questions that are answered in industry-standard reports from Mandiant or CrowdStrike or other incident response investigators, and came up with a question set not only for this experiment, but also sort of based on our experience running CTFs.
These are the questions that needs to be answered if you're trying to determine the scope of an intrusion, figure out what happened. Is that, is that true as well? That's exactly right. And the research paper describes in a lot more detail what steps we took to make sure that the test was fair. Um, but, but the fundamental, um, step we took was just to reuse the same Capture The Flag exercise that many, many humans had competed in, and that one happens to have been focused on a Volt Typhoon threat actor. There was a second experiment, um, un- only partly related to the first about incident response reports.
And rather than describe that in detail here, I think I'll direct, uh, listeners to the white paper because it's a little bit more complicated.
But in every respect, we tried to assure fairness, and there's a lot of detail, even technical detail, about how we configured the tools we were using and steps we took to make sure that the agentic system wouldn't be biased in favor or against Corelight or any other data source. Great.
Well, can you describe a few of those conclusions then? Well, the first conclusion that surprised me a bit, and I was really curious about, was how well the models are doing these days. I think six or nine months ago, a year ago, there was a lot of discussion, uh, and lamenting of the tendency of agents to make stuff up. So, uh, they would hallucinate, they would invent data if needed, they would sometimes claim to do things they didn't do, and the list goes on and on. I think we're all familiar with that.
What we discovered based on, uh, tests a little bit earlier this year, just a couple months ago, is that the hallucination tendency was way down. If agents were presented with a hard question and data that simply couldn't be used to answer it, they would, they would stop. They would report that fact rather than making up a fictional answer.
So that was, that was pretty interesting, and it's great for the prospect of automating some aspects of cybersecurity defense with AI. It's terrific. The other thing we noticed, again, on the agent side, is that there are some differences in the agents in terms of their performance and their speed, but overall, they're all working at a similarly advanced level for the task that we gave them. And, and that's terrific. It was a little bit unexpected. We thought maybe there'd be one agent, uh, at one moment in time that would be drastically better than the others. There were some differences, uh, but generally, the models from different companies performed about the same on the Capture the Flag. So that's one set of surprises related to the models. The really dramatic results, um, though c- came when we tested how well did the agents do when presented with different kinds of source data. And, and I know, Richard, you spent a lot of your career thinking about the necessary forms of data that, that a SOC needs to have. And I'm curious, before you heard the results of... Did you have any instincts about, um, what impact source data might have?
I, I had some, but I, but I had a lot of questions as well. Yeah, I mean, just from a basic standpoint, if you're taking a look at, say, flow data, you're limited to source IP, source port, dest IP, dest port, protocol, timestamps, amount of data transferred, and that's it. Uh, you're not gonna get usernames, you're not gonna get these other important issues that are, that are captured in other forms of transaction data like you might get from, from Corelight or, or Zeek. Uh, and then in, in terms of intrusion detection alerts, those are biased towards whoever wrote the signature for that. So if you're specifically looking for, for behavior and you come up with a signature for it, great. It might tell you,
"I believe this, this has happened." But in terms of anything that does not have a signature associated with it, well, you're just not gonna get any data whatsoever.
So- Mm-hmm ... uh, that's why I've never said you should only have one. Um, if I had to choose, I would have se- you know, the rich, uh, transaction data plus the other forms of data you get out of, out of Corelight. Um, and that's, I, that's probably an important point we should talk about Corelight as well is
Corelight is not simply Zeek data. Um, many people might think that it's just, you know, Zeek on a box, but that's, that's actually not true.
But yeah, that's, that's sort of my initial, uh, my initial bias going into this is, is to expect that you would get richer results from richer data. Yeah, I had this exa-- Well said. I had exactly the same expectation. I just didn't know how to quantify that expectation. And, and honestly, Corelight logs improved Capture the Flag scores by over 350% compared to NetFlow logs, and over 60% compared to firewall logs. On the other experiment, which we haven't talked about very much, but on the incident response report writing task that we also gave to the agents, Corelight logs provided evidence for about 50% more IR findings compared to firewall logs, and about 300% more compared to NetFlow.
So NetFlow and firewall logs are good and much better than nothing, but the difference between a 60% attainment on a Capture the Flag and 100 or
98 could be very, very significant in the context of a real SOC managing real risk. And it was so interesting for me, having made this argument qualitatively for so many years, it was so interesting to finally get hard quantitative proof of the impact. Uh, another way I'm explaining this to customers and prospects is that each data source sets a kind of ceiling for what can be known. In the AI vocabulary, you can think about that as a cap on inference. Inference is the work that an, a la- language model does to answer a question. And a, a given data type, like firewall logs, create a cap on inference, just a, a cap on the sophistication and insight that any model can have simply because of, uh, deficiencies in the source data.
So, uh, that's the, i- in a nutshell, those are the results that, that Corelight data produce much better outcomes compared to the other alternatives.
And a- again, for the skeptical or dubious or curious, um, we explain this in great detail, and we're also, of course, very happy to talk about it and invite dialogue around the findings as well. I do definitely appreciate their quantitative approach, but for me, the qualitative is, is also really important because that captures questions that you simply can't answer using the other data types. And
I think we were able to put some answers around that, right? Uh, upwards of 90, high 90% questions could be answered using Corelight data, whereas the other data types just couldn't answer vast numbers of questions. Yeah, that's right. There's just certain kinds of questions, and you could imagine a question like the name of, um, a w- a Windows file share or a DNS reply.
Uh, you can just imagine a whole series of questions that could be critical in an investigation but just aren't available, um, within the schemas of those other data sources.
I think we found, I'd have to go back to the report to confirm, but that in most of our runs, Corelight provided all of the data necessary to answer the questions. Sometimes the, the runs would time out, um, just because we set an artificial limit on token count available for each question.
But that raises another important dimension of performance, we found that if you take a question that all the data types are capable of answer, the Corelight data was enabling the models to answer about twice as fast because they were able to get to the answer quickly with fewer iterative loops. And so not only can richer data help agentic systems answer harder questions and more questions, but they get to the answer much faster, which makes a difference, I think, in the era of post-mythos, uh, automated AI attacks.
That is such a great point because we're seeing these stories now of companies that have burned through their AI budget in the first four months of the year, and now they're wondering what they're gonna do next. I- I've spoken with people who work at some companies who they're in their budget cycle right now, and they're trying to forecast their AI spend for the coming year, and, uh, this is a real constraint.
Uh, we, we just can't assume that you're gonna be able to spend as much as you want on this and get the results that you want. You have to be efficient with your, your allocations. Yeah, absolutely. We're hearing this. We heard it discussed at our last board meeting just a couple of weeks ago.
I know I've spoken to Russ, our CFO, and he's hearing the same discussion among his circle of CFOs. It's as if there's been a, a little bit of a hangover for token consumption, and, um, when we have to look at the question of cost benefit of token consumption, you know, in the cold, hard light of day. Uh, and I think every organization that went hard at AI solutions is discovering that in some cases, tokens can be more efficiently spent. So I think it's a good message to get out there that higher quality, high fidelity, well-structured, densely interlinked data helps AIs get to the answer, a-agentic systems get to the answer faster, more efficiently, and cheaper. Well, well, let's talk about that agentic triage angle for a minute then, 'cause what this tells me is that you can have agentic triage, but if you're not feeding your agent quality data, then it should not be coming up with actionable, um, insights. In other words, it'll say, "I suspect," or, "There could be something happening here, but I don't have the confidence because I don't have quality evidence." Whereas I know with our own agentic triage element for Corelight Investigator, it's got good data to work with, so it can be more confident and show its homework when you're querying it as to how it came to a conclusion. It's absolutely true. If you start with poor quality data, you could end up with a triaging system that wastes both money, but al- just as importantly, wastes viable human attention. It puts humans in the loop or on the loop, um, to resolve, uh, questions that really can't easily be resolved with the underlying data, and, and that, that is just unproductive spend of tokens and unproductive heat dissipation in the data center. So it's better to start with very high quality data that has a chance of answering the questions. That will produce much better outcomes, lower costs, and more efficient agentic SOC projects. Would you happen to know if we have participants in our ongoing CTFs who are using agents in order to complete the
CTFs? Y- you're wondering if they are using them somewhat surrep- surreptitiously on the side- ... to as to, um, to supercharge their results?
Yeah. Yeah. I'm wondering, like, are, are we getting to the point where we'll just y- if you're not u- like, is this gonna be like the steroid Olympics, where if you're not, uh- ... using performance enhancement, you're just never gonna have a chance against everyone else who's participating? I think there's t- two dimensions to that. One is whether anyone's doing that a little bit secretly on the sly. And to the extent that in some cases they're physical, some cases they're virtual CTFs, I guess we can't rule it out. Uh, I'll have to talk to our CTF team to see if they put in place measures to keep people, um, wholly human-centered in their CTF performance.
But the other is more about, um, a future workflow where agents become efficient, um, really fast, specialized coworkers. And, and, and I think everyone expects that to become the case, and coworkers and delegtees over time.
As soon as, as soon as we can assure that, um, we can trust their results and their actions. And of course, many-- The security industry is evolving very rapidly w- um, to, to meet the implications of that assumption.
Lots of different companies are thinking about dimensions, security dimensions of that future architecture, including Corelight. I think it would be neat if we, if we just assume people will use it and they come up with results, but then maybe we also have a special category where it is j- just the human. Uh, you know, there might be some technical, uh, constraints that you could put around it.
Even, you could even do a, a human constraint where the people are working, you know, maybe there's only a s- small number of people agree to this, and you have proctors who are walking behind them, making sure that they're not using an agent or whatever. And I think that would be a f- a fun kind of contest to win, you know, the, the only human, uh, CTF, uh, response winner. Right.
Um, u- unassisted human CTF, it's a great idea. I'll definitely, um, pose a couple questions on Slack after this discussion. But what a, um, amazing indicator of the cultural whiplash we've experienced,
Richard, that, you know, I don't think we anticipated a few years ago that we would need to set aside certain CTFs maybe for human-only, uh, uh, security athletes. But that's- Right ... I think the world we live in now.
Yeah. Do you know if, if we're doing any follow-up research or what might be next for this project? We-- I'm certain we will. Um, we've structured the framework or the, the agentic test harness so they can easily be extended. And the first thing we did after publishing the research paper was look to see if we could run some of the leading open source models, and especially those designed to run, um, quite efficiently and pr- produce the same results. And when we ran those experiments, and this isn't reported in the paper, but when we ran those experiments a couple of months ago, we had limitations, especially on context window size, but also some other limitations that we encountered.
But I'm very curious to know in three or four months if the open source models of, say, October,
November can, um, meet the performance or even best the performance of the proprietary models of March or April. I, I would expect that to be almost true, but we'll find out. I'm very curious about that. Well, Greg, this was a really interesting topic, and it's, it's one of these topics where I'm sure, like you said, in a few months or six months, we're gonna get some different results. So, uh, I'm sure our, our wizards will be continuing to work on this. So thank you again for joining me on the Corelight Podcast to talk about, uh, what agents can do for you for security and how Corelight data is the, is the fuel that will, that will give your engine the right outputs. It's always a pleasure. Thanks so much, Richard. Thank you for joining us on the Network Defenders Podcast, sponsored by Corelight. We will see you on the network.
You've been listening to Corelight Defenders. To stay informed with expert intelligence on today's cybersecurity challenges, please subscribe to ensure you never miss an episode. We'll see you on the network.
Subscribe now
