Skip to content
  • There are no suggestions because the search field is empty.
Corelight is a Customers’ Choice for NDR according to Gartner Peer Insights   Get the report
Corelight is a Customers’ Choice for NDR according to Gartner Peer Insights   Get the report
Home/Podcasts/Episode 28 - Base Rates,...
Episode 28 - Base Rates, Likelihood Ratios, and Correlative Alerts for Defenders
Guest Speaker: David Burkett
November 19, 2026

Episode 28 - Base Rates, Likelihood Ratios, and Correlative Alerts for Defenders

Episode 28 - Base Rates, Likelihood Ratios, and Correlative Alerts for Defenders
0:00 / 0:00

About the episode

Richard Bejtlich sits down with David Burkett, Corelight's security researcher, to explore how base rate fallacies and signal detection theory can improve intrusion detection. They discuss false positives, true positives, and the posterior odds that help SOCs make sense of overlapping alerts from EDR, network sensors, and email gateways. The conversation covers test-driven detection, overlap in coverage, and practical guidance on tuning rules without drowning analysts in noise, with insights on how to measure and improve detection effectiveness in real-world networks.

Episode transcript

Download transcript

Episode 28 - Base Rates, Likelihood Ratios, and Correlative Alerts for Defenders

Welcome to Corelight Defenders. I'm Richard Bejtlich, strategist and author in residence at Corelight. In each episode, we explore insights from the front lines of NDR, network detection and response. Today, I'm speaking with David Burkett, security researcher at Corelight.

Welcome back, David. Hey, man. It's good to, uh, speak with you again. I am glad you're back. We have a couple great topics, and who knows what else we'll be able to add into the discussion. But the one we're gonna start out with was you had mentioned to me in Slack that you had noticed something from my first book, which is, I think it was written on, like, papyrus scrolls or something. It's so old at this point.

Uh, but it was mention of something called the base-rate fallacy, and this was a paper from 1999 by Stefan Axelsson.

And the, the basic idea with the base-rate fallacy, and, you know, you're more in tune with this now than I am 'cause it's been such a long time. But it seems that if you have a fairly infrequent event, uh, it's very difficult perhaps to create an intrusion detection system that will reliably suppress the number of false positives around that event.

So apparently you looked at that, and you've added in some more interesting ideas around that and ways to, to make that useful for security people. So can you just talk to me a little bit about that? Yeah. So I actually came across it. So I, I, I have a copy of your old book, but it's been so long since I've read it. You posted that paper on LinkedIn actually in a comment, and I decided to just go read through it, which is what caught my eye. And, uh, it was really great. So the basically the kinda core theme of that was making the case that the false positive rate is more important to measure than the true positive rate because the idea being, you know, your analysts are gonna start ignoring alerts if your false positive rate is too high, uh, which you don't wanna have.

And, um, basically, uh, it gives you a, a whole kind of way to kinda academically go through and determine, like, your false positive or true false, positive rate on your, uh, your rule set. Uh, but one of the things, uh, that I've been kind of researching into lately is something that

I didn't really even surprisingly know existed, uh, until, uh, about a year and a half or so ago, uh, was called signal detection theory. It kinda spawned out of, uh, radar o- operator research post-World War II, uh, with the idea being, hey, we need to be able to distinguish are these radar signals real, uh, you know, alarms? Uh, are they real bombers, for example? Or are they just birds, you know, some sort of weather phenomenon, and how can we, you know, more reliably deconflict those and detect that? Uh, and they came up with something called the likelihood ratio, and the two really go hand in hand.

And, uh, I really love the likelihood ratio, uh, for cybersecurity because really it's just taking the same base probability. You get the same numbers, but

I think it's easier to, uh, apply and understand in the cybersecurity context when you're thinking about intrusion detection alerts. This is one of the things I like about what you're, what you're doing here is that when we first started out, because cybersecurity was new, you know, you think about this paper written in 1999, there wasn't-- We hadn't been doing intrusion detection that long as an industry. Now, you could say, yes, it was invented in, like, 1970 by the Anderson paper, but as far as, like, actual work being done with systems and real data, this was still a, a very new thing.

So people were open to bringing in other areas because that's what was available. And then over, you know, the ensuing decades, cybersecurity's become more and more of its own thing, and in some ways it's kind of become a little bit too insular, I think.

So the fact that you're looking outside again to something that has a similar problem set but is using, you know, a different set of cases, I guess, I think that's really interesting. So can you tell me more about how that could help what we're doing now? The, the, uh, likelihood ratio is gonna be the way that we expand onto the base rate paper that we were talking about.

Uh, one of the ways that I really like, uh, applying it, and one of the reasons that I, I like using it, is because in the context of cybersecurity, uh, the whole idea of it is, you know, it's saying, um, as you take multiple signals, the likelihoods of that being a true alert or a true positive, uh, increase exponentially. They're multiplicative.

Uh, and basically what you're doing is you're taking that base rate. So from that, uh, um, you know, 1999 paper, if you have, you know, 20 real events in a million, about, uh, one in 50,000 are your odds. Uh, you take your true positive rate and divide it by your, uh, false positive rate, and then you multiply that times your odds, and, uh, that'll basically give you your, um, what's known as the, uh, the posterior rate, which is basically your-- the odds of something being real after examining the evidence.

And the idea is, uh, it's very unlikely for multiple, uh, unrelated signals and systems to have false positives at the same time, so each independent signal is increasing the likelihood. And where that kinda comes into play in cybersecurity is where you start doing those correlative alerts.

So, uh, as you're doing something like maybe, uh, a lone EDR alert or AV alert might be, you know, uh, fairly low fidelity on its own. But whenever you start seeing, uh, a lone, uh, EDR alert, but then it's followed maybe by an intrusion detection system alert, uh, now that goes up a little more if you see more alerts like the email security gateway. So let-- maybe let's take it from like a, uh, uh, a, a maldoc example. You know, you might get a email security gateway alert and network intrusion detection alert for the maldoc coming in, uh, EDR alert for the person opening it and executing it, another, you know, a- alert for it beaconing out. Each of those increases the likelihood of that being a true positive, uh, substantially. And if you go through and basically take those two concepts, that base rate, uh, and the likelihood ratio, uh, it really comes up with a great way for you to get actual, like, metrics for your SOC. So you can start saying, "Based on these rules that I have and based on my, you know, true and false positive rates, uh, how many..." Um, you can start estimating how many, like, man-hours you're gonna need based on the likelihood of, like, uh, how many of, like, alerts in your queue are actually gonna be true positives versus false positives. So I think what you're telling me is that it is good for the defender to have overlapping coverage. That seems obvious, but you'll often see conflicts where someone will say, "I've got detection in my email gateway, so I don't need it with my, my network product," or perhaps, "I don't need it in my EDR." But the fact that you do get that overlapping coverage means that the analyst is more likely to recognize that this is a problem.

And you don't always get detections from all these different, uh, instruments, whether they're, they're evaded or perhaps they're not working or they're c- not configured properly. So you do want as much overlapping coverage as possible. Maybe you don't want, you know, duplicate EDRs running on a system or, or whatever. Right. But, uh, to have as, uh, you know, multifaceted detection, I think i- is really important.

Yeah. Whenever I'm looking and doing detection, uh, I'll commonly see, like, let's say I detect something. Uh, despite me detecting that event, I might see other behavioral, like, opportunities that I can detect on that would be an entirely separate de- detector.

And, uh, even if I'm already detecting, you know, the activity that I caught, um, having that second one, you know, would raise the fidelity should the other one alert, you know, in the future.

So, like, um, you know, going back to that maldoc example, uh, when Microsoft introduced the mark of the web, uh, to prevent, you know, from being able to open up, uh, malicious macros from, uh, public emails, uh, the amount of maldocs, uh, exploitation dropped, I think something by, like, 15%, um, for some of that initial access. And, uh, whenever you go through and you start adding other kind of behavioral detections, so, like, if I were to go through and say, "Hey, you know, a maldoc never needs to," or a, a, you know, a Microsoft document never needs to spawn PowerShell, so now it's got the mark of the web, and it's spawning PowerShell, that just raised its, you know, malicious, uh, you know, severity a lot more.

And then maybe it's also making, you know, network connections somewhere out that's beaconing in C2, and again, much more severe. And that's where you're going into those correlative alerts, raising that likelihood ratio. You, you, you mentioned true positives and false positives.

Mm-hmm. Um, but I think the, the un, unappreciated cousins or even siblings of those are true negatives and false negatives. Right. And, you know, false negative is something I think not too many people think about, right? The -- what is a false negative? It's an event occurs, you want to detect it, but your system is silent. Um, people tend to get lulled into trying to minimize false positives because you're getting flooded with events.

But if something happens, and you have a system in place that you think is gonna catch it, and you don't see it, and you're just thinking, "Well, it did-- nothing happened, so I must be okay," that could be potentially more damaging, right? Because it means something bad has happened, and you have no indication of it. Yeah, exactly. So this comes into the, one of the, uh, w- pieces called out in that base rate paper that's actually difficult for organizations is, how do you know how many actual malicious events happened? And, you know, kind of figuring out some of those numbers. And something that really helps is if you start adopting a test-driven detection, uh, methodology.

Uh, so as you start getting tools like, uh, you know, the Atomic Red Team, MITRE's Caldera, Mandiant Security Validator, those kinds of things, um, you're never gonna have, you know, every single, um, attack out there, every s- you know, you know, known behavior, whatever. But this can give you an idea of what you're actually detecting versus where your gaps are and can give you an idea of, hey, these are all of the, quote-unquote, uh, malicious events I should have caught. These are the ones I missed. And you s- can start getting some of these numbers a lot easier where, uh, prior to some of these, uh, uh, attack-driven tools, uh, existing, um, that wasn't the case.

Yeah. Yeah, just having a heartbeat alert can be so valuable. Um, I remember Ban Visher, who invented Squeal, he added a heartbeat function. And so every, whatever, I don't know if we did it every five minutes, every ten minutes, but we would generate, uh, w- activity that if the sensor was active, and the rules were loaded, and all the parts of it were working properly, the whole back end, you know, uh, events are getting loaded, we would get an alert of heartbeat type whatever. And if we didn't get the heartbeat, we would get an alert that said, "Heartbeat was missed," and maybe it was transitory, but two in a row, okay, we have a real problem now. Something, you know, something's gone down. Right. Even though the sensor is reporting it's alive, we're not seeing the events that we should be. So, uh, just something as simple as that can really save you. Yeah. Yeah, exactly. You know, there's a, a, a great talk by one of my old coworkers, uh, Ryan Ojoro, uh, who used to manage the red team for, uh, Target.

Um, basically, uh, whenever we adopted a test-driven detection methodology, we found something like 25% of our detectors, and this is in a public talk for BSides Augusta. He just gave it last year. Uh, but 25% of our detectors weren't actually alerting on the things that we thought that they were. Uh, and the reasoning was, in most of the cases, was because the intel that was sourced was wrong. Maybe the, uh, detection engineer or analyst assumed an event would be a certain way, when in reality it wasn't. Maybe it was just a formatting error with, like, the escapes whenever it got transferred to, like, the website or something along those lines.

Uh, but that was a really surprising number, uh, to me, and, uh, I highly recommend checking out that talk. It's, uh, Ryan Ojoro's talk at BSides Augusta, uh, 2025.

And, um, one of the things that I'll add to that is, uh, I think that that's actually gonna get worse, uh, if you're not doing some sort of test-driven development, because with the advent of AI hallucinations, now you have the hallucination of intel, and if you're doing, uh,

AI to grab and process those intel reports, now you have hallucination potential on both ends, increasing the likelihood of, uh, you know, uh, hallucinations or inaccuracies in that intel. And that was, that 25% number I gave you was before, uh, AI was really writing the reports and, you know, taking off like it was. So, um, something to kinda consider. Yeah. How do you put Suricata rules to work? So it's definitely it depends. But, uh, I, I think it really does though. Like in a smaller, more mid-size organization, uh, I might deploy my rules, basically say,

"Hey, let me get the malware categories," the, the categories that are typically gonna be definitely malicious or more likely to be malicious in my environment, uh, see what's triggering, figure out which ones are the false positives, tune those out, and just, you know, as new rules get added, if a new false positive comes in, just kind of clear it out that way. Uh, when you start getting into a large enterprise organization where, like you might be a Fortune 50 organization, uh, you can't really do that because there's gonna be so much noise and so many different alerts, uh, that are gonna be false positives that you have to kind of take in a different approach.

And, um, I think that there's also two problems. Uh, one, I think people misunderstand that just because something is called an alert doesn't necessarily mean you have to detect on it, especially in isolation. Uh, and then two, um, f- the, the process or the approach that I've taken is, um, honestly not super great in the past. It's been more like, hey, these are the, the groups that's targeting us, basically going through and onboarding them at a time. So like let's say I might have adversary group or malware family that I'm, you know, doing detection research on, go through, see which, you know, Suricata rules are in the ETOpen and ETPro rule sets or the Corelight sets, and then onboard those manually, uh, to my detections from my SIM, like after I've gone through, verified that they've not, you know, alerted and so on.

Uh, but the rule corpus is huge, and, um, that's not necessarily a, a great way, and I know that AI could probably help that. You know, you said before we were getting started on this that you had some strong opinions. I'm really curious what those look like. Yeah.

So it really comes down to what your team looks like, I think, and what your goal is. Mm-hmm. So I'll tell you what I do at home.

Um, I recently set up a new Suricata implementation with EVE Box by Jason Ish, which is, which is just awesome. Mm. Great job, Jason. They keep developing new versions. I downloaded the whole ET

Open. I think it's like 66,000 rules. I mean, it's massive, right? I remember when it was under 10,000, whatever, but I turned the whole thing on Mm-hmm ... because this is my home network, and

I, you know, it's a single class C, probably not even a quarter of it is used. Right. And you could give me false positives all day long, and I'm not gonna get swamped by them because there just aren't that many systems.

And over the last several weeks, I've just been tuning them out. So when I see systems reaching out to Microsoft for an update and it's the, you know, the dotted quad alert, or if it's a packed executable alert, uh, I just either, you know, I, I,

I suppress, um, put in, uh, either by source IP or dest IP or whatever it is. I, I, I tune it a little bit. I would never recommend you do that in an environment where you've got any significant number of systems or users or whatever, uh, because like you said, you end up just completely overwhelmed by alerts. Mm-hmm. And you start to question your sanity, and you burn out your analysts, and that's where people say, "Well, we're just gonna throw

AI at it." The, the method that I've tried to advocate and actually formally put it into writing now with, with the, the new Corelight NDR, book, is to go completely the opposite, which I think is what you were getting at, where, you know, if you have the expertise, uh, and what's kind of nice is that you can get access to this expertise now with, with AI or whatever if, if you're careful.

Mm. Um, figure out what is it that you're trying to capture. Like, what is it you really care about? And write rules that, that discover that.

This is what we did at GE. This is where I learned this, learned this methodology. So we would, uh, come up with rule packs, basically, of 10 alerts at a time, and, uh, you know, people like David Bianco and Aaron Wade and people like that were writing the alerts by hand and doing the documentation to come with each alert. So it explained what it meant and what to do if you saw it and what was the group behind it, and this is all like sort of more high-end, uh, you know, then called APT-type activity.

Uh, and then as the analysts learned how to use them, we would come up with another batch of 10, and we just kept adding batches of 10, and we were constantly balancing it against their capability to process that, you know, given the number of people we had and the number of alerts it would fire and the workload and all that sort of thing. But at the end of the day, is that something you could expect every team to be able to do? I, I just don't know. Um, it doesn't seem like most teams have that, either that will or that, um, skill, uh, interest. I don't know. So it seems like most teams are probably gonna fall somewhere in the middle. Like, they do download a giant list of alerts, and maybe they turn off all the notices or the infos or whatever, and they focus on the malware a-alerts, and then they throw some automation and AI at reducing the false positives.

Yeah. Well, you know, I think what helps too is understanding, like, some of the concepts or the ideas behind some of those more informational alerts is, um, a... Or at least my understanding is, uh, at the time when a lot of these were written, not everybody had Zeek or was running both, and it gave people a way, you know, if they only had NetFlow, to have some kind of idea of what else was happening on their network.

Um, and where we can tie this back, to, the likelihood ratio, uh, if you start tying some of those informational level alerts, like maybe it's a, an IP info type lookup to get an external IP address, maybe it's, uh, a EXE that was downloaded shortly after, um, you can start basically building up, uh, an increased likelihood or start building a risk-based analysis, uh, scoring method, you know, compounding some of these alerts. But that's definitely gonna be more advanced, um, kind of stuff. Well, well, that does, that does lead to the idea that maybe you only surface certain types of alerts and others you, uh, send to a database, and then there's a third category of ones that you either disable or you threshold or suppress or, or whatever. So like you said, when you're trying to get more signal around an event, something, some part of the iceberg is above the waterline, and you key off of that. But then you run your query, and you see a bunch of other less interesting activity like, uh, something about the, the TLS certificate or something about the, you know, it did use a quaded dot IP address as opposed to a website.

Although you do have the issue now of you're storing all this stuff that you may never look at, but that's kind of the tyranny of doing this kind of work, right? Yeah. You know, it's, uh, it's interesting. Uh, this is pivoting just very slightly, um, but I was doing some, uh, analysis on.

the, uh, malware traffic, uh, uh, like, PCAP corpus, and just all of the write-ups that they've done over the years. And something. that I found really, uh, I guess, fun, uh, was how many of the, uh, um, certificates actually had problems.

Uh, like maybe they were either self-signed. It' was something crazy, like, uh, 80% of the ones on there or something like that. I think closer to 60%, closer towards 2026.

But, you know, uh, having a self-signed certificate on its own, not enough to alert on, but maybe you're s- doing self-signed certificate in conjunction with a .top domain. You know, what admin is gonna have a, you know, a login panel with a self-signed certificate legitimately, uh, on a .top domain or something like that. And now you've just got a, a really, uh, high likelihood malicious, uh, signal by comparing two more informational level alerts.

Yeah. And I think what you're describing too is what we're trying to do with the agentic triage, uh, capability in, in Corelight, right? Is that you've got these, these different signals that individually may not catch your attention, but once something happens...

I mean, I, I don't exactly know how the system works, whether it's, it's simply looking in aggregate, and then because it sees something in aggregate, it surfaces it, or there's a certain alert that catches its attention, and then it goes out and looks for corroborating evidence or associated evidence, and then based on what it finds, it's, it gives you a, some sort of indication that I think this is a problem because I found all this other stuff. But that sounds like it's, that's one way to put that capability into a person's hands without having to do it all manually.

Yeah. Yeah, exactly. From my understanding, that's how it works as well. We're still trying to figure out the best way to use alert-based systems when, I mean, at least theoretically, they've been around for, uh, over 50 years at this point, um, although in terms of systems that have been in the field and used on production networks, we're only talking since, uh, you know, the very, you know, mid to late '90s, I guess. But we're still trying to figure out the best way to, to leverage all this data. I have a love-hate relationship with, uh, LLMs right now. On the one hand, uh, I feel like they're super useful for enabling teams to, uh, you know, have some of these capabilities that we've talked about, like going through and, uh, onboarding some of these alerts, where maybe they might not have had that prior, uh, and what have you.

And, uh, w- it's been-- It's nice to work at a company where we have a tool that generates such good data to work with and that has so much data that's been open source, not in the open for, you know, decades at this point, so that all of these LLMs are familiar with it and, you know, know how to work with it, so. Well, David, thank you so much for joining me again on the Corelight Podcast. There's never a shortage of cool things to talk about with you, so hopefully we'll, we'll have you back again soon. Yeah, man. Uh, like, uh, you saw in our Slack message, I have a whole bunch of things I'm down to talk about anytime, so happy to be here. Thank you for joining us on the Network Defenders Podcast, sponsored by Corelight.

We will see you on the network. You've been listening to Corelight Defenders. To stay informed with expert intelligence on today's cybersecurity challenges, please subscribe to ensure you never miss an episode.

We'll see you on the network.