AI system evaluation is the process of continuously assessing AI system capabilities, limitations, and performance through quantitative and qualitative measures. Across the system lifecycle, evals provide continuous assurance: Validating system behavior before deployment and detecting drift, bias, and reliability issues in production.
Evaluation is particularly challenging in AI systems because small changes to prompts, tools, or workflows can produce unexpected behavior throughout the system. In Network Detection Response (NDR) specifically, minor adjustments to tool calls or model parameters can result in a silent drift in signal, either failing to surface critical intrusions or overwhelming security operations with noise. Continuous evaluation then becomes a core requirement for developing and operating trustworthy AI systems for network detection. This article shares how Corelight approaches that problem: The custom harnesses we built, the metrics we chose, and the lessons we learned curating security data for evaluation.
In traditional software systems, outputs are largely predictable from inputs. Behavior can be unit tested, integration tested, and reasoned about deterministically. AI systems break these assumptions. They are non-deterministic, invoking tools, maintaining state, planning actions, making decisions, and interacting with external systems at runtime. In an AI system, behavior emerges from interactions among multiple components including the underlying evolving models and their reasoning, prompts, instructions, and system context, tool availability and API behavior, memory systems and state management, planning and reasoning loops, environmental and runtime context.
In an NDR investigation, this is a multi-step process. An alert triggers, the agent decides which evidence to pull (from logs, entity context, or historical entity activity), correlates that evidence, and produces a verdict with supporting rationale. A failure at any step, such as pulling the wrong log source, misreading a baseline, or failing to account for entity context appropriately, can silently produce a wrong verdict even when every other component behaves correctly.
Since behavior arises from these interactions rather than any single component, evaluation frameworks must cover the entire system to reveal whether the overall agent performs as intended and to catch regressions when any of these components change.
Meaningful evaluation goes beyond any single measure and should track:
Without multi-dimensional evaluation, a change that improves one metric can silently degrade another and go undetected until it reaches production. Furthermore, AI systems evolve quickly. Teams may be simultaneously adding or modifying tools, refining prompts and instructions, upgrading models, and changing memory architecture or routing logic. An evaluation suite functions as a regression test suite for agent behavior, providing a baseline against which every change can be measured before deployment.
Off-the-shelf eval frameworks are useful starting points, but they make assumptions that don't always hold for production AI systems or customer data, especially in the NDR domain. Most general-purpose frameworks assume single-turn Q&A with natural-language ground truth. Neither assumption holds for NDR. An investigation is inherently multi-step (alert → enrichment → correlation → verdict), and ground truth is structured in the form of labeled PCAPs and logs. For Corelight's custom data pipelines, internal tool calls, proprietary business logic, and product-specific scoring criteria, we found it necessary to build custom harnesses that provide a higher level of precision and control for network-based cybersecurity.
There are many approaches to measuring model performance, and choosing the right metrics for specific use cases matters significantly. Traditional NLP metrics like BLEU and ROUGE rely on lexical overlap and were designed in the pre-LLM era: BLEU for machine translation, ROUGE for summarization. They tend to underperform when evaluating the abstractive, paraphrased, or complex outputs that LLMs typically produce.
More semantically-aware approaches like BERTScore and Sentence-BERT improve on this by measuring meaning rather than surface similarity, but still fall short for evaluating abstract reasoning or domain-specific outputs. This gap is especially pronounced in NDR. Our outputs are technical judgments, such as malicious/benign classifications, confidence scores, entity attributions, that need to match analyst reasoning rather than phrasing similarity. At Corelight, we use a reference-based framework augmented with performance metrics that give a broader view of model behavior, including LLM-as-judge evaluation for qualitative dimensions.
LLM-as-judge approaches are powerful but require care. Key considerations include:
A combination of deterministic and LLM-as-judge methods provides the best coverage, maintaining both interpretability and flexibility.
With our custom Corelight harness built around data pipelines and product-specific quality criteria, we can systematically probe model behavior across multiple dimensions:
Quantitative metrics (such as tool invocation rates, task completion scores, and LLM-as-judge scores) show how changes to the system improve or hurt performance. But these metrics alone don't explain why the system behaved the way it did. Qualitative metrics can bridge the gap by providing interpretation and insight into the nature and root cause of failures. This can provide crucial information in security contexts, where different failure modes carry significantly different levels of risk. Together, the quantitative and qualitative metrics turn evaluation from a pass/fail gate into a diagnostic tool, providing a grounded, reproducible picture of model behavior across the conditions our system actually faces.
Furthermore, evaluation runs across iterations are meaningful and actionable, and they contribute to long-term improvement:
By leveraging our custom evaluation framework, we have systematically established behavioral baselines for various model families within the Inspector MCP product. This analytical process provides actionable signal, enabling us to identify which models perform best based on specific metrics and the underlying SIEM architecture.
Over time, the eval harness becomes an institutional record of how the system has matured, where it has improved, and where risk remains.
A golden dataset is a high-quality, hand-labeled collection of inputs and expected outputs used as ground truth for model evaluation. Cybersecurity data presents unique evaluation challenges that compound the difficulty of AI evaluation, including:
At Corelight, golden datasets are built through a manual curation process: Developing question-answer pairs and complex text summaries grounded in network traffic data, using PCAPs replayed from open-source resources such as malware-traffic-analysis.net and from carefully reconstructed networks resembling real-world systems and attacks. Human expert curation remains the gold standard in this environment. It guarantees ground truth accuracy, mitigates bias, and ensures that evaluation reflects real security logic rather than artifacts of the data pipeline.
The current evaluation work has focused on specific use cases and agent behaviors. The goal going forward is to expand coverage systematically across the MITRE ATT&CK framework, ensuring that evaluations represent a broader and more representative distribution of the threat landscape, and that new capabilities are matched by rigorous, grounded evaluation from day one. In network detection, evaluation is not a one-time benchmark; it is an operating discipline that must evolve alongside models, tools, data, and threats.