Black Hat NOC: Findings from Europe & thoughts for Asia 2024

How quickly a year passes. 2023 was Corelight’s first year participating in the Black Hat Network Operations Center (NOC). It was a tremendous opportunity and responsibility in which we collaborated with teams from Cisco, Palo Alto Networks, Arista, Lumen, and NetWitness to keep events in Asia, Europe, and the US safe and functional for all attendees. As we speak, our team is gearing up for a repeat for Black Hat Asia 2024 in Singapore.

In blog posts last year, we described the logistical challenges of deploying our team and gear thousands of miles from home and insights we gained from integrating with our best-in-breed partners in the NOC. We also undertook some high-level analysis of these collaborative experiences, and unpacked an actual incident from our Las Vegas deployment.

One of the most important parts of the experience for our team was approaching Corelight’s Open NDR from the perspective of our customers. At the same time, we could evaluate our platform’s performance and leverage a live, high-stakes environment to approach its data collection and analysis from new perspectives.

As end users, conference by conference we extracted new insights that will inform the ongoing development of Corelight tools and help security operations centers become more efficient and effective in many types of environments. 

Black Hat Europe 2023 was perhaps the quietest of our three Black Hat deployments. But it nonetheless provided us with several opportunities to gain insights into user behavior by carving artifacts out of network traffic using Zeek®, Smart PCAP, and other Corelight-native tools. The intel we uncovered along the way helped enrich alert context, shed light on user behavior (advisable and otherwise), and even divulged reconnaissance of Corelight staff (OK, maybe it was just a lucky coincidence).

Use Case 1: Living in Syn-ology (aka Classic Synology)

During a routine User-Agent check this morning while enjoying our first cup of coffee (which took us back to the days of working in security operations centers before joining Corelight), we discovered something interesting: standard HTTP connections made with the Synology DS File app (version 4.17.1) from a Samsung Galaxy S20 Plus 5G phone. Presumably, this was occurring in the clear to a public-facing Synology network attached storage (NAS) device:

The usual recommendation for Synology traffic is a configuration that leverages their specific HTTP/S port (5001/tcp). So, we found it alarming that this particular connection used the much less secure 5000/tcp, Synology’s default HTTP port. This information alone was sufficient incentive to perform a deeper investigation.

Synology has a very convenient built-in feature: Users can leverage Dynamic DNS (DDNS) to access their devices while they're away from the local network.

However, there's a VERY important configuration element the user SHOULD enable. Alas, it’s optional by default:

Digging further, we investigated Zeek's metadata collection, which showed additional HTTP header extensions: In all, we found seven HTTP connection records between this client and the destination, showing signs that it was being used as a direct backup connection of media files and other items.

Extracting the PCAP using Corelight's Smart PCAP storage platform enabled the next step down into the data. Given there were already signs of images and other media within the traffic payload, we decided to run Zeek locally, using its native file extraction capabilities to acquire much more information:

From this extraction, we were able to collect details which, we suspected, the Synology user in question did not know was being sent in this manner.

Here’s one glaring example of this HTTP/S omission below:

Another captured image reveals … Corelight’s own VP of Global Alliances, Todd Wingler! (Reconnaissance? Perhaps. All kidding aside, we suspect Todd’s presence in this snap was entirely coincidental.)

In the confines of Black Hat Europe, these findings had no implications more severe than a bit of insight into how one participant planned to spend an evening. SOC teams, however, can extract some key takeaways here. Consider user behavior in the environment you’re tasked with protecting: How many might deploy the Synology DS File app for their personal or company-issued devices? Are any of them aware of the Let’s Encrypt configuration option?

Use case 2: Finding Suricata outliers in Black Hat training traffic

This use case highlights the relevance of David Bianco’s PEAK approach to threat hunting: It’s not about sending (overworked) humans out on more and better snipe hunts. Rather, effective threat hunting depends on automating detections and tuning them over time. As network analysts for the day (or, in this case, for the conference), we were interested in how we could set and adjust automated and machine learning processes to take faster, deeper dives into the metadata.

Black Hat’s scheduled training sessions generate a high number of Suricata alerts for many of the IPs utilized in the sessions (192.168.120.0/24 to 192.168.132.0/24). Maintaining a productive environment for everyone enrolled in these courses was a paramount objective. Our responsibility (and challenge) was to identify alerts unrelated to course activities or, at the least, find an effective means for prioritizing alerts and digging deeper into a select number. Challenge presented: Challenge accepted!

We developed a tool to compare and visualize the Suricata hits for the IPs of a given training session. The goal was to quickly distinguish per-classroom outliers from uncommon Suricata alerts that could warrant deeper analysis.

The screenshot below shows an example plot for the training session Reverse Engineering Firmware with Ghidra (192.168.120.0/24), which occurred on December 4, 2023, the first day of Black Hat Europe:

The screenshot highlights that:

• Most IP addresses were clustered (right side of the plot), which means they had a similar set of Suricata rules firing.
• There were a couple of outliers with different hits: 192.168.120.27 and 192.168.120.34.

By looking at the Suricata hits per IP (refer to the table at the bottom of the screenshot), we could quickly identify that, in this context, it was "normal/expected" to flag passwords in cleartext HTTP (ET POLICY HTTP POST contains pass= in cleartext). Other Suricata hits were outliers, like downloading PE EXEs or DLLs via HTTP (ET POLICY PE EXE or DLL Windows file download HTTP). These occurrences were the hits we prioritized for hunting and analysis, as they were unlikely to be related to course activities.

Behind the scenes

To leverage these visualizations for hunting, we scheduled an hourly machine learning job that:

• Retrieved all the Suricata alerts for the analyzed day
• Collected the list of Suricata alerts for each IP address
• Learned a compressed representation of the combination of Suri rules per IP (reducing the dimension of the data from ~100 rules to 20).¹
• Performed T-distributed Stochastic Neighbor Embedding (TSNE) to visualize the 20 dimensions in 2D.²

The result was one 2D plot per class, which showed similar IPs close to each other, augmented with the full table of Suricata hits for each of the IPs in the class/training.

The takeaway

This analysis helped us to prioritize the Suricata hits for threat hunting and get a sense of what alerts were part of class activities vs. something to dig deeper into. Beyond that, the exercise highlights the multifaceted uses of metadata pulled from a handful of outliers. A standard use-case can act as a springboard from which personnel with specialized skill sets (e.g., data science and big data analytics), can make productive use of their tradecraft.

Note, too, that in our role as Black Hat analysts, we weren’t using anything that isn’t readily available on the Corelight platform our customers deploy. SOCs in all sorts of environments can incorporate the same capabilities and iterate on data and findings to improve detections within their own environments. You could say this was our way of saying, “This is one way to do it; extrapolate, go forth, and be innovative!”

Use case 3: Transparent translations

While investigating a medium-level Suricata signature, we came across some URI content that looked very—shall we say—human. There were clearly emojis in the uri that suggested conversations taking place.

Using the Corelight Open NDR Platform to pivot on the host and the uri fields, we quickly found that certain applications (think messaging apps, online games, browser extensions) use translate.google.com in the clear. The following data extraction resulted from our following this particular Suricata signature; but to keep things interesting, let’s use an advanced-emoji-human-detector to start the hunt (AKA just pasting common emojis into the search bar within the Open NDR interface!):

These emojis were clearly visible within the cleartext uri. They were being sent by the application to Google’s translation service in the clear.

To determine the breadth of the issue, we then widened the search scope. We used a series of CrowdStrike® Falcon LogScale queries within the Corelight Open NDR Platform to clean and refine the output so that we could easily investigate the clear text in isolation (Zoom Enhance, you might say):

Some of the translation requests in this example were from non-English to English, so the English translation was contained in the server response (not the browser request). While investigating these, we were able to extract the translated material from PCAPs delivered by Smart PCAP. Some examples:

The takeaway:

1. Zeek logs within the Corelight Open NDR Platform provide the necessary flexibility to investigate and refine understanding of policy breaches (for example, sensitive clear text content)
2. Smart PCAP allows for selective capture of traffic, allowing incident responders to delve further than the Zeek metadata if and when required.
3. Encrypt sensitive traffic, even when it requires some extra steps.

Conclusion: Time to gear up for 2024!

Black Hat 2024 will present its NOC with a new set of challenges and opportunities to find virtuous integrations between our products and solutions. You can be sure we will be reporting back from the wire here on the Corelight blog.

But whatever new adventures we undertake, there are a few core concepts that will inform our approach. Namely, analysts without data cannot make informed decisions, but data without people results in no decisions, and no defensive action (proactive or reactive). Our team will leverage our experience, an evidence-based security approach, and input from our NOC colleagues to constantly reevaluate network traffic and parse it in ways that help us get to the ground truth faster and more efficiently. See you in Singapore, Las Vegas, and London!