Black Hat NOC USA 2023: A tale of sharp needles in a stack of dull needles

During Black Hat 2023 in Las Vegas, our Corelight team worked effectively and speedily with our first-rate Black Hat NOC partners Arista, Cisco, Lumen, NetWitness and Palo Alto Networks. I was fortunate enough to be a member of the NOC team at the show, helping to defend the Black Hat network. In this blog, I’ll share my experience within the Network Operations Center (NOC) as well as an incident that we detected, investigated, triaged, and closed using Corelight’s Open NDR Platform.

Read on to learn how our NOC team was able to:

• Detect a threat using Suricata^® within the Corelight Open NDR Platform
• Easily understand a Suricata rule using Corelight’s AI GPT-3.5 integration
• Deep dive into traffic using Corelight Smart PCAP
• Identify the C2 IP behind the threat using Corelight’s Intel Framework
• Pinpoint the MAC and host name using Corelight’s Known Framework
• Build and close the case that there was malicious traffic by leveraging CrowdStrike LogScale-powered queries and Zeek^®’s network metadata within Open NDR

Setting the Scene

It was nearing shift changeover at the Black Hat Network Operations Center, so I headed down to the NOC to catch up with the rest of the team. We’d all already been online for a while and were following the various Slack threads so as to be up to speed when we sat down in the NOC for the second shift. The Corelight NOC team consisted of six hunters, being two shifts of three allotted seats in the NOC. The shift times were not overly regimented and allowed for flexibility. As we were enjoying our time in the NOC and keen to complete our hunts, we tended to start earlier and stay on later—we were in our element, as it were. There is something quite stimulating about being in the NOC environment; you might call it the thrill of the chase. All of us are seasoned hunters and engineers, and armed with our Corelight Open NDR Platform, the Black Hat organizers trusted us once again to provide detection/hunting and analysis during the six day Black Hat event at Mandalay Bay in Las Vegas.

The conference consisted of four days of Training sessions, prior to two days of “Briefings” and vendors demoing products at hundreds of booths within a vast vendor hall. A crowd of 20,000 people was expected to attend. By day two of the “Training sessions, the network was well and truly alive with activity, and the NOC was in full swing. We had a couple of days prior to the event to generally familiarize ourselves with the network, which was a process we were well prepared for as this was our second engagement in the Black Hat NOC—the first being at Black Hat Asia in Singapore a few months earlier.

To paint a picture of the scene in the NOC: Imagine a large, dimly lit room containing around 30-40 people, all on a mission to protect. Then imagine being surrounded by lots of black shirts, EDM/tech music beating away loudly, and glancing up to see some cool old “hacker” genre movies like Sneakers, Tron, Hackers, The Matrix, and more playing on a big screen in one corner. Along the long back wall there are six large screens displaying dashboards that highlight key network activity metrics. In the right corner, and next to our Corelight seats, there is another big screen which shows a real time visualization of traffic flow between network nodes. To me, this visualization at first seemed a little glitzy, but it was actually quite useful in practice and maybe the most viewed screen in the NOC. The faces of hunters and engineers in the NOC are lit by the laptop glow, some with a furrowed brow, some with a look of curiosity, some surprise, some with that “Eureka moment” look, and yet others with a frown suggesting they’ve just realized the rabbit hole they were going down was a dead end - it happens! Some are sitting at the tables that circle the perimeter, and some are chilling on the two large brown couches in the center of the room. In the adjacent break-out room, there’s normally someone grabbing a coffee, donut, “NOC Pastrami” (thanks Mo!), or quizzing over the jar of Vegemite labeled “NOC Crew,” which strangely appeared.

The NOC has a guarded entry, which is off a glass-walled walkway where visitors can come and see what goes on in an active NOC, a scene that is also being live streamed 24x7 on Twitch. There are inside jokes, like a stuffed “SOC monkey” (see if you can spot it in the photos below) that looks back through the glass at the visitors with various booty hanging from around its neck - items like fake NOC passes, USB keys which were confiscated from the Social engineering class’ attempts to get into the NOC. All in all, the NOC is a very cool place to be if you are in the industry.

The Black Hat training network environment is absolutely unique. Where else in the world do you find such a concentration of talent and a broad range of offensive security testing tools and techniques being taught in one place? Given this, we fully expected to see nefarious activity emanating from the training classes as the teachers and students unleash all types of offensive testing tools directed to their external systems. Of course, this traffic is allowed as it is the point of the training, and it traverses our vantage point where our Corelight sensor is installed and monitoring. While class traffic is expected, our goal at the conference is to help protect against the following elements:

1. Threats against the Black Hat network and conference infrastructure itself
2. Nefarious activity not directly related to classes
3. Anything illegal, of course
4. Generally help protect our conference participants where we can

Given the nature of the event, one might liken our task to finding razor sharp needles in a pile of needles. The rest of this blog demonstrates an interesting detection, which goes to the final point above: Protecting conference participants. Note, some of the data cannot be shown due to privacy reasons and has been redacted with green boxes.

Inside the incident

As I scrolled through the events surfaced by our Open NDR Platform, I noticed a detection for a trojan called Autoit.F, a trojan that I can’t say I’ve seen before. I then looked at the corresponding HTTP logs within our Platform, which we’d enriched with the name of the training session that had been assigned the wireless IP referenced in the alert. The topic of the training didn’t seem to relate at all to the alert and it did not look like class activity, so alarm bells were ringing. The hunt begins ...

The red boxes show how the alert looked like within our Open NDR Platform, please note that we have since redacted sensitive information with the green boxes:

The threat was detected by Suricata, which was running on the Corelight sensor. Suricata is guided by a vast set of signature rules. If you aren’t familiar with Suricata rules, imagine a fairly terse coding language that instructs the sensor which malicious elements to look for within the network traffic. Unless you are used to reviewing or writing Suricata rules, they can be quite challenging to understand. To this point, one of Corelight’s new and interesting functionalities is the use of generative AI (GPT-3.5) to describe these complex rules in a more expressive way that those unfamiliar with the rule syntax can understand. Although I did look at the Suricata rule itself, this “Generated from AI” explanation (pictured in the above screenshot) was helpful to have displayed in such a consumable fashion early on in the investigation.

Enabling the GPT 3.5 integration is as easy as flipping the “Enabled” switch below.

The AI-generated content comes with caveats (explained in the screenshot above) and is meant to augment and assist, rather than strictly define facts and actions to take. With this is mind, there are also AI-generated outputs that describe “What this alert might mean,” as shown below:

Furthermore, there is even a “What are some possible next steps” guide, as shown below. I was happy to find this guide surprisingly accurate, especially for such a complex task as hunting and IR. While the AI is definitely helpful to smaller organizations and those with minimal Incident Response expertise on hand, it also proves handy for seasoned hunters to review, providing reminders to chase down angles that might have been missed.

Once we were armed with an understanding of why the alert fired, we were able to hunt in the Zeek network logs (specifically the HTTP logs) for more context. Not familiar with Zeek? You can learn more about how Zeek works and how it is the foundation for Corelight’s network evidence here.

Since the alert focused on HTTP POST connections, we explored the key Zeek log elements using the search functionality within the Corelight Open NDR Platform powered by CrowdStrike LogScale . Here, we confirmed that the traffic was NOT a False Positive, in that it had all of the elements that the Suricata rule was intended to detect, plus it did not look like benign traffic that happened to match the signature. It’s important to note the POST content, which includes encoded/encrypted information that’s redacted here, was being sent from the training class victim host to the external C2.

At this stage, there was enough to post our initial observations as a very strong “heads up” to our partners in the Black Hat NOC threat hunter Slack channel, and noted that there would be more analysis to come. Providing this heads-up encouraged collaboration with our Black Hat NOC partners and allowed the whole NOC team to contribute to the hunt by confirming or questioning findings as early as possible.

With the detection confirmed as valid, and given that Black Hat classes were expected to have “malicious-but-legitimate” activity, we needed to explore the possibility that the traffic could be class-related, even though that seems unlikely. Put another way, we were trying to determine if this needle in the haystack of needles was sharp or dull. One quick way to get a sense for class-related traffic is to look for any other internal IP that is contacting the external C2, because in a class situation, usually there are multiple IP’s (each belonging to a student) making similar connections to the same external IP. A quick change in the search string in LogScale told us that instead there was a 1:1 IP mapping between the victim and C2. No other internal IP had connected to the C2, which firmed up our suspicion that this was not related to class activity.

Note that the victim host was also attempting to connect to the C2 over TCP 9999. Although all such port 9999 connections were not completed (as can be easily seen by REJ conn_state) this was still highly relevant, as the well respected intrusion analysis group “The DFIR Report” had previously associated BOTH this C2 and port with the “DarkGate” C2.

It’s often useful as a hunter to look at key data from different angles, and this is one area where the Corelight Open NDR Platform shines, as it allows rapid manipulation of “what-if” style search queries by using LogScale to visualize the various facets of the data. Here we saw the victim communications to the C2, per destination port over time.

In addition to seeing activity drop off around midday, presumably as attendees took a break for lunch, we also noted the volumes of ports 9999 and 2842 over time were fairly spiky. This could be by design, something called “jitter” in C2 parlance, or simply because the C2 was unstable or resource-constrained.

Using Corelight’s Smart PCAP technology, which is sometimes thought of as a component of a “network flight recorder,” we were selectively recording NON-encrypted traffic. This is often useful in triage and as part of deeper investigations. Smart PCAP allows you to store only what you want to store without the burden of storing more voluminous, less interesting traffic. Here we used the common configuration of omitting encrypted packets. Retrieving a sample PCAP from Smart PCAP was quick and easy, and the following Wireshark screenshot shows a sample of victim-to-C2 communication that we saw, where the client POST its data to the C2 and the C2 returning an “OK” in the HTTP content.

Yet another data point supporting the initial finding came from the Corelight Sensor’s implementation of Zeek’s Intelligence framework, which showed the C2 in both the HTTP headers and Server IP as flagged by three organizations, including Mandiant.

Meanwhile, we were making updates on the Black Hat threat-hunting Slack channel as we went, with each observation strengthening the case that the traffic was malicious. The incident was soon escalated within the Black Hat IR process, where actions including blocking IPs, redirecting internal IPs to a “time-out” page or direct communication with the class members were considered.

As part of the context-building phase, we made use of the Corelight Entity Collection’s “Known Framework” package to build out information regarding the Victim PC.

• The “known_devices” log showed us the MAC address of the Network Card as well as the vendor that manufactured it.
• The “known_names” log showed us the NETBIOS name of the laptop being “LAPTOP-4[REDACTED]”

We then continued to monitor the traffic and build context. This can be a delicate phase of the hunt, as in almost every case you do not want to “touch” the C2 infrastructure for various reasons that may later become important. So oftentimes we use a more passive Open Source Intelligence (OSINT) framework. For example, a search for the C2 IP on Shodan shows us a record:

1. The C2 was in Amsterdam.
2. RPC, RDP, WinRM and NBT were open. With these ports open, it’s possible that this IP is a compromised Windows box itself.
3. NetBIOS Domain Name: WIN-LIVFRVQFMKO

From as far back as 2019 through to recent times, the system name WIN-LIVFRVQFMKO has been implicated with various ransomware activity related to “Lockbit”, “Medusa” and “DarkGate” [1] [2] [3] [4].

Now in case anyone still needs convincing this was malicious traffic: A public Joe’s sandbox sample of malware included ALL of the detection elements that we had already found in our traffic, including the Same C2 IP that we had been tracking as part of our incident. The screenshot below highlights the key elements in red, including the Content-Length of 2 (being the “OK”) that we saw earlier in our Smart PCAP.

In parallel with the context-building phase, as part of the IR process one of the NOC leaders, Neil (AKA “Grifter”) personally went to the training room, introduced himself and made an announcement that someone in the class was compromised and currently beaconing data to a C2 and that they may want to clean that up. Reportedly there were some chuckles, but no one fessed up. Shortly after this, I noted that the traffic to the C2 had abruptly stopped, and it looked as though the victim host had rebooted, as there was a brief network silence followed by a DHCP request from the same MAC address.

And with that, the incident was marked as closed, and attention returned to one of the many other hunts that were going on in parallel. It was a good way to start my shift in the NOC on day two, knowing that we helped someone who had come to the training sessions with more installed on their laptop than what they’d realized.

Summarizing the findings

This incident nicely highlights several aspects of Corelight’s technology, demonstrating how they can feed into a real world detection and investigation context:

• Suricata on the Corelight sensor detected the threat.
• Corelight’s GPT-3.5 integration generated content provided a readable rule meaning.
• Smart PCAP enabled a deep dive on the traffic.
• The Corelight sensor’s Intel Framework flagged the C2 IP.
• The Corelight sensor’s Known Framework provided the MAC and hostname.
• We could then build out the full story leveraging effective use of simple LogScale queries within Investigator, coupled with the underlying network metadata that Zeek provides.

To summarize the incident itself:

• Corelight Open NDR raised the alert
• Initial triage completed and finding communicated
• Confirmed detection is valid, malicious, and not class-related
• Our NOC Partners concur on findings
• IR escalation
• BH NOC staff informed class members, so that they may clean their host.
• Traffic to the C2 ceased shortly after.

Our team really enjoyed our time in the NOC, and we'll use these learnings to fine-tune our process for defending the Black Hat network in the future.

To read more about the 2023 Black Hat events, here are some other articles:

• https://www.theregister.com/AMP/2023/08/12/black_hat_network/
• https://www.blackhat.com/us-23/noc.html
• https://blogs.cisco.com/security/black-hat-usa-2023-noc-network-assurance
• https://corelight.com/blog/black-hat-asia-2023-detections-findings
• https://corelight.com/blog/black-hat-asia-2023-deploying-corelight
• https://corelight.com/blog/corelight-ndr-blackhat-noc

Stay tuned for more blog posts from my fellow teammates about their experiences within the Black Hat NOC. For more information about Corelight and how we can help your team disrupt attacks, please contact us.