Chocolate and peanut butter, Zeek and Suricata

Some things just go well together. A privilege of working with very sophisticated defenders in the open source community is seeing the design patterns they use to secure their organizations – both technology and workflows. One of the most common has been deploying Suricata for signature-based IDS along with Zeek for both rich security metadata and behavioral detection. Defenders choose this pattern because it allows them to both find potential threats and radically accelerate investigating them, in a way that gives them wide visibility and no disruption to operations.

At Corelight, our goal is to deliver this design pattern in a commercial offering, not simply as a pair of great, but separate open source technologies, but as a more deeply integrated system that works better together than the sum of the parts. We are proud to announce that in our v19 software release we have delivered a sensor that combines and integrates Zeek and Suricata with three key benefits.

Focusing first on investigation, we wanted to directly link the Suricata alerts to the Zeek logs used to investigate them. You may have learned about Zeek through training courses (SANS and beyond), and heard about the UID that connects all of the structured protocol logs to each other. That UID enables defenders to quickly pivot from a connection, to unusual aspects of the protocol in use, to the metadata or even file content being transferred. With this integration, we have added the Zeek UID to the Suricata alert logs so analysts have that same direct access and acceleration. 

For example, if a Suricata alert triggered on a potential SQL injection attack, the analyst needs to determine whether the vulnerable page was exploited. That is a hard question to answer in many environments, as the analyst would need to pull up PCAP … if they still have it! Thanks to Zeek’s rich protocol logs, the analyst has that information available in the HTTP log – and due to the direct UID integration they can pivot directly to it saving them valuable time for each investigation. This is just one example of course; the UID connection here allows that same “direct to data” access from the alert to a wide range of evidence that analysts need, such as URLs, user-agent strings, files, or JA3 fingerprints.

Accelerating investigation is the biggest driver for this joint design pattern. In order to get that value, defenders have had to essentially deploy two network monitoring tools in parallel. Each tool is generally oversized as architects account for variability in network loads, traffic types, signature loads and many other variables. That oversizing results in higher infrastructure costs such as rack space in data centers or machine instances in IaaS environments. Our integrated system approach runs both services not just on the same sensor, but on the same individual CPU. Doing so lets us gracefully manage joint performance and ensure that the data itself is both accurate and consistent across both Zeek and Suricata, even as traffic volume and mix changes. This isn’t the sexy part for sure, but it matters: the whole point of network monitoring is to have reliable visibility across your operating environment. We have talked to several self-engineered organizations that thought their deployed estate was covered, only to realize they were unknowingly dropping 30-50% of packets. This is the worst kind of internal operational vulnerability, as it won’t show up on your next scan and it is a bit like finding a black hole – you have to go looking for what is NOT there in order to find it. The integrated system approach we used here ensures consistent, optimized performance and clear visibility into the rare instances where operational issues do arise.

Last, and this is the fun part, is what an integrated system approach enables for threat detection. Today sophisticated defenders are already writing custom Suricata rules and Zeek packages, for signature and behavioral detection respectively. Those two worlds exist largely in isolation though: the resulting alerts are aggregated at the SIEM but aren’t aware of each other at the point of analysis, which limits the amount of dynamic detection logic that defenders and threat researchers can use. Our architecture brings Suricata events into Zeek’s script land, so behavioral and signature threat detection logic can work together on flows while still in memory. This is a new architectural door, but one that our threat research team is excited about, so be on the lookout for more here and let us know your ideas as well!

A final thought on our philosophy: as we developed this integrated system, it was important to not just work with these open source projects but contribute to their continued success. Corelight has always been an active and driving contributor to the Zeek community, but now that we are working with Suricata we have extended our financial and code contributions with the Open Information Security Foundation as well. Expect more of this to come from us, as we learn about successful design patterns from sophisticated defenders across the open source community, and democratize these approaches so the broader industry can benefit as well.