Zeek
Finding Very Damaging Needles in Very Large Haystacks
In previous projects, I’ve tackled some needle-in-haystack problems. From these efforts, several high-level themes have emerged.
There is a saying you will hear tossed around by Black Hat veterans, and even in the presentations: “threat hunting on the Black Hat network is like trying to find a needle in a needle stack”. Still, knowing this did little to prepare me for how unique of an opportunity it is to work in the Black Hat Network Operations Center (NOC). I've been working for Corelight for a little over two years, and previous to that I worked incident response for the US government.
I've seen countless iterations of exercises and strange SOC working environments, so I thought little could surprise me. However, being in the NOC at Black Hat Europe was unlike anything I had experienced before: sitting in a square room with random “hacker” movies projected on the wall while staring at a new batch of 200 Suricata® alerts every hour, knowing all of them were true positives but representing very few incidents—that really set the tone for one of the most unique experiences in my career. It gave me a new appreciation for “trying to find a needle in a needle stack.”
Over the course of my time in the NOC, two things became apparent to me. First, I needed to adjust my working methods to be able to separate the wheat from the chaff. I added some context-based dashboards to help divide the traffic by the many classrooms at Black Hat, and tried to better utilize the enrichments that were added by a fellow Corelight alum. This helped paint a better picture of what was happening on the network. Using the enriched Zeek® data, I was able to easily tie alerts to the classes that were going on at the same time, which made separating legitimate hacking from potentially malicious hacking a bit easier.
The second thing that became apparent to me was relying on just alerts wasn't going to cut it. Adjusting to that reality was a bit more difficult. I needed to change my way of thinking about traffic. If you talk to any seasoned leader in the network forensics space, this is something that they understand pretty intuitively. However, when working directly with SOC analysts and a variety of SOC programs, sometimes this gets lost. Alerts are only one part of the equation. It's the context surrounding that information that really separates the value of those alerts.
Using some of the aforementioned analysts' reports and taking a deeper look into the traffic, we discovered that the tool responsible for sending the HTTP Post data was a Data Loss Prevention tool; yet this very tool, which the user was using for security, was leaking their laptop’s data over the internet. The security product would make regular check-ins using HTTP and perform “syncs” with their servers. These syncs contained usernames, processes currently running, service accounts, and a multitude of other information just sitting in the clear, ripe for the taking. We were only able to discover this kind of information by taking a look at the deeper picture. If we weren’t monitoring the network, we never would have caught such a blatant security error.
Working at the Black Hat NOC for the first time really opened my eyes to how deep analysis can be done in a fast-paced and intense environment, and just how important taking that next step is to defense. All in all, Black Hat Europe was a success thanks to all of the partners involved in putting together the network and the NOC: Arista, Cisco, Corelight, Jamf, and Palo Alto Networks. It was amazing to see how each of these technologies can be used and integrated, and what we can achieve in network security. I can't wait to apply these lessons not just to my next stint as a threat hunter in the Black Hat NOC, but also to any other threat hunts I embark on in the future.
For more on the Black Hat NOC, I recommend checking out our blog. And for more on threat hunting using Zeek, Suricata, and more, check out our Corelight Threat Hunting Guide.