Data driven detection: Corelight’s approach to AI-powered NDR

The Gordian knot of any detection strategy is knowing that two conflicting ideas are both true. On one hand, every SOC needs as much accurate detection coverage as they can get to find and disrupt attacks. On the other, the attackers you REALLY care about will find a way to bypass those detections so you need the ground truth of the attacker behavior on your network. The only answer is to have both: the absolute best data and the broadest detection suite possible on top of it. Trying anything else means you are choosing between having an airplane cockpit or the NTSB flight recorder … but not both.

That’s not a plane any of us want to be flying. Corelight’s detection strategy builds across three related pillars, to give you the best solution for not just today’s threats but the ones you don’t (yet!) know you are looking for:

The best data doesn’t just help you detect threats—it rewrites the rules of defense. It is AI ready, fuels new detection techniques, speeds investigations, and gives you a time machine to uncover threats that didn’t even have names yesterday. We built our data using Zeek® - the gold standard for network monitoring - and then extended and enriched it to turn it into evidence that enables best-in-class detection and automation. Think of Zeek as the microscope—and Corelight as the lab that turned what it saw into clear, contextual evidence built for detection and response at machine speed.

Corelight’s multi-layered analytics engine leverages the data to deliver both detection coverage and accuracy. We use machine learning (ML) techniques ranging from random forests to neural collaborative filtering, the unparalleled behavioral detection capabilities of Zeek and many other engines to not only find modern attacks but also fuel your threat hunting and detection engineering programs. Beyond simply bringing the right tool for the right job, this lets us provide the right accuracy and evasion resistance across the full spectrum of attack sophistication.

Great data and accurate detections then fuel a GenAI-powered workflow that supports the analyst from initial alert prioritization and triage through full incident resolution while minimizing false positives. This automation extends to deep integration with your existing technology stack including EDRs, SIEMs, SOARs, and even your other security GenAI tools. Beyond that integration, our open source heritage means that with Corelight you are not buying a black box. You’re joining a community that trains, shares, and evolves together to accelerate the SOC.

Let’s delve deeper into these three strategic pillars.

Pillar 1: The best data drives the best Detection

In security, foresight starts with hindsight. High-fidelity data empowers you to discover the unknown, investigate with speed, and gives you a time machine to uncover threats that didn’t even have names yesterday.

We are the custodians of the Zeek project, which has evolved for over 25 years to become the definitive source of network data for incident response and threat detection. That iterative, ongoing evolution–by security people and for security people–has resulted in data that is:

• Deep: Zeek extracts rich, security-relevant metadata from every single network transaction. This granular detail provides unparalleled visibility into network activity.
• Broad: It analyzes an extensive number of protocols, ensuring comprehensive coverage across your network traffic.
• Interlinked: Zeek intelligently connects the network connections, protocol logs, PCAP and files into a single story of network activity.
• Extensible: Anyone can add new data types, environment enrichments and even protocol logs.

We then extend Zeek’s core capabilities with both rich content options, new insights, and enrichments. Rich content, like extracted files and PCAP, are key supporting artifacts for investigations. New insights like analysis of encrypted traffic, identification of network entities and deeper insights into application behavior both power custom detections but also accelerate incident response. Finally, we extend the default logs with unique information that adds insight custom to your environment and further accelerates IR, including CMDBs, vulnerability scans, host data from EDRs, and more. In the cloud, we bring this same rich level of insight and add information from the control plane, including app ID and service enumeration directly into the network logs to improve context and clarity for analysts and AI alike.

Finally, we are open. You own your data. You can export any or all of it to our platform or any tool (or multiple tools!) you prefer. For incident response or detection engineering or compliance and more. This commitment to seamless data access is a critical differentiator and a key lesson learned from the Zeek community: as long as you have the right data you can take advantage of each new analytics evolution as it happens, so we are committed to giving you that strategic optionality.

Pillar 2: A multi-layered detection engine: The right tool for the job

Effective network threat detection demands a comprehensive suite of detection engines as well as support for your threat hunting and detection engineering programs. Each engine brings options for speed, accuracy and sophistication that are best for different threat types:

• Signatures and threat intelligence are ideal for quick, accurate detection of threats like newly discovered vulnerabilities in network devices (using Suricata®) or malware in extracted files (using YARA).
• Zeek’s behavioral detection engine excels at high precision behavioral detection. This engine is often best for protocol level detections (finding custom encryption, DNS over HTTPS or ICMP tunnels), command-and-control (C2) toolsets like Sliver or Meterpreter, VPN abuse or cloud-based data exfiltration.
• Supervised machine learning is best for analyzing complex behaviors such as social engineering domains or malicious file downloads that require multi-variate analysis (using techniques ranging from random forests to convolutional and long short-term memory neural networks). We use our Polaris co-development program to develop and tune these models across a variety of networks and verticals to prevent the high false positive rates that ML is prone to generate.
• Anomaly detection uses unsupervised machine learning to flag unusual activity. We use a range of techniques, including clustering, neural collaborative filtering, and deep learning, to highlight behaviors like unexpected service offerings, deviations in SSH versions or servers or odd HTTP user agent strings. These findings should be used not as alerts themselves but rather as context for prioritizing other attack activity, accelerating incident response investigations or starting threat hunts (unless in highly standardized environments like ICS/OT).
• Threat hunting is increasingly standard practice (and certainly best practice) to search for targeted attacks, evidence of newly discovered vulnerabilities, supply chain attacks or emerging attack types. In addition, threat hunts often reveal policy violations, uncover infrastructure problems, and inevitably help analysts know their environment better which improves day-to-day incident response - a true force multiplier.

Most modern NDR providers claim a similar approach of using a range of analytics engines for threat detection. Looking deeper, our approach brings you four critical differentiators:

1. Zeek’s behavioral detection engine allows us to perform behavioral analysis with rule-level accuracy. This reduces our reliance on ML engines, reserving them when truly needed and lowering false positive rates as a result.
2. The best data powers the best threat hunting in either our platform or your SIEM. The results of these hunts can then be turned into custom detections directly within Corelight.
3. ML on your terms: Unlike many NDR vendors who offer ML support only in the cloud, Corelight provides the flexibility to deploy ML capabilities either on-premise or in the cloud as your needs dictate.
4. Work with a community, not just a vendor: The Zeek open-source community is fighting advanced threats every day, and contributes publicly or privately back to the project. As a result, you aren’t limited by our insights and roadmap but benefit from community driven innovation from organizations like MITRE, AWS, and CISA to name just a few.

Pillar 3: Detections feed a GenAI-powered workflow

Corelight’s detection strategy continues from alert generation through full incident resolution. The starting point of that workflow is intelligent alert aggregation and prioritization so analysts can focus on what matters most. Within that priority set of detections, we then apply GenAI-powered enrichment and alert explanations so analysts of all skill levels can understand the situation at hand faster.

From there, we power the incident response workflow with GenAI as well. This ranges from alert triage to investigation guidance and beyond, which dramatically enhances the capabilities of junior analysts or helps senior analysts to operate much, much faster.

Next, we have built deep integration into the SOC technology ecosystem to help accelerate your whole team - not just analysis of our detections. This includes native support for your current SIEM/SOAR platforms to drive downstream analysis, EDR platforms for context and automation, ticketing systems for case management and NGFW platforms for automated remediation as well.

Finally, our open source heritage helps accelerate the SOC both today and tomorrow. First, Zeek is standard in the DFIR and threat hunting training programs from SANS, Cisco and many other sources, giving analysts ready access to training and an industry-standard skillset. Second, all major LLMs are “fluent” in Corelight (due to their training on the public internet and our work with the flagship providers), which lets you drive GenAI based workflow automation out-of-the-box with straightforward prompt engineering. Third, we are here to help as well, by sharing our own prompts to jumpstart your in-house GenAI efforts. To illustrate the power this can bring: we had customers using ChatGPT to draft Splunk queries against Corelight data to accelerate DNS exfiltration investigations… in March of 2020, mere weeks after ChatGPT 3.5 was released! Overall, with Corelight you are not buying a black box. You’re joining a community that trains, shares, and evolves together to accelerate the SOC.

Conclusion

We have the privilege to support some of the largest organizations in the world and defenders of critical infrastructure over a wide range of sizes. With their feedback, we have spent years developing an integrated system. We started with Zeek—the industry’s most trusted lens into network activity—and then engineered every layer above it to deliver precision evidence for threat detection and response at speed. Corelight’s detection strategy:

• Uses the right tool for the job, delivering broad and accurate coverage that complements your existing detection technologies.
• Provides the best data to enable rapid triage, power GenAI workflows, and empower your in-house threat hunting and detection engineering teams.
• Continuously evolves and extends through both our own efforts and open-source community contributions.

Ultimately, the proof is in the impact we can make on your SOC, which is driven by both our technology and the right prioritization of monitoring points for your architecture. If you'd like to explore how Corelight’s Open NDR Platform can modernize your security operation and consolidate legacy technology, please reach out. We are happy to share our insights and look forward to learning more about your specific challenges.