What makes evidence uniquely valuable?

Editor's note: This is the third in a series of Corelight blog posts focusing on evidence-based security strategy. Catch up on all of the posts here.

American novelist F. Scott Fitzgerald famously wrote that “the test of a first-rate intelligence is the ability to hold two opposing ideas in mind at the same time, and still retain the ability to function.” All experienced security practitioners learn to master this mental trick. On the one hand, they believe efforts to prevent and detect breaches will be effective. On the other hand, they diligently prepare for the day when their efforts will fail. And on that day - when prevention and detection reach their limits - what tools do defenders reach for? 

The answer, in almost every case, is high-quality evidence derived from networks, systems, and other sources - evidence that makes it possible to reconstruct the mechanism of a breach, and understand its consequences.  

This blog is part of a series devoted to exploring the role of evidence in information security. Previously, Brian Dye explained the motivation for an evidence-based security strategy, and Richard Bejtlich illuminated an important use case for evidence. 

Here I want to talk about the attributes of high-quality network evidence. What should evidence look like, in order to be useful to defenders in the heat of battle, when the next Internet-scale security event such as SolarWinds or Log4Shell unfolds?  

1. In the first place, evidence must be continuously and automatically collected, without much preconception about the mechanism of future attacks. (The metaphor of security cameras is apt, here. Banks locate cameras everywhere, because they can’t predict the technique or target of the next bank robber). 
2. Evidence must be neutral, uncolored by opinion or bias. It should report events as they occur (eg, a DNS reply) in real- or near-real time, even if such events appear benign at the time of collection.  
3. Evidence should be richly detailed. In the past, coarse-grained evidence may have been sufficient for tracing the mechanism of breaches. But in the face of today’s more sophisticated attacks, detailed evidence is critical for distinguishing normal user behavior from lateral movement.  
4. Good evidence must be well structured - to facilitate indexing, search, and novel analysis. As we know, data science is transforming information security, and powerful analysis requires quality data.  
5. Finally, the most powerful forms of security evidence acquire the status of de-facto standards, to facilitate information exchange and collective, global response. 

Few forms of security evidence have all these properties, but those that do often come from open source communities. That’s not surprising, because proprietary data formats are less likely to be accepted broadly, and less likely to evolve quickly in response to new threats and defensive practices.  

The phenomenon of open-source standards for security evidence is part of a broader trend. As Microsoft’s John Lambert has written in an influential essay, “there is an open approach that is currently rippling across the infosec industry.”  (Lambert goes on to explore the ecosystem of open SOC tools, highlighting the potential impact of Sigma, MITRE ATT&CK™, and Jupyter notebooks.) 

In the realm of network-derived evidence, Corelight’s focus, the gold standard is Zeek. Tens of thousands of organizations worldwide have deployed this powerful network security monitoring platform. In fact, if you register for SANS training on threat hunting, you may learn that skill through the lens of Zeek data.  

There are many reasons Zeek is having its moment today, but the most important is related to the quality of evidence the tool provides. Zeek has all the attributes explored above: designed from the beginning for automatic collection, it also generates neutral, detailed, and highly structured evidence that’s well understood by practitioners. 

So it’s not surprising that Zeek became the foundational technology in open-source network security deployments, especially in large organizations and incident response firms. Over time, the format and structure of Zeek data have evolved to meet the changing needs of security teams. That power of constant evolution is rare, and in some sense it’s a sixth desirable feature of high-quality evidence.  

At Corelight, our founders created Zeek, and our technology stack was inspired by open-source network monitoring patterns.  Within that tech stack, Zeek plays the role of ‘evidence collector’.  Why? When traditional techniques of detection and prevention have reached their limits, evidence comes to the rescue - if we plan for that outcome.  At Corelight, it’s our mission to provide the best network evidence available.  

By Greg Bell, Co-founder and Chief Strategy Officer – Corelight