In the NOC at Black Hat Asia 2024: It came from behind the podium

Black Hat, as past attendees know, revolves around training and information sharing in predominantly offensive techniques. So threat hunters in the conference network operations center (NOC), including the Corelight team, expect to see plenty of “offensive” traffic on the conference Wi-Fi.

However, the conference setting introduces an extra layer of challenge. In a traditional corporate network, analysts typically classify any evidence of offensive techniques or indications of compromise as an ‘incident’ and act accordingly (i.e., with urgency). On the Black Hat network, immediately blocking ‘offensive’ traffic or endpoints demonstrating indications of compromise may not be reducing risk. The action is likely to disrupt someone’s classroom experience — and undercut the purpose of the conference.

As NOC threat hunters, the Corelight team has to take a step back from the the real-world mandate (“find evil”) and balance out risk of compromise with the conference goals:

• Ensure attendees don’t compromise each others’ privacy and security
• Ensure infrastructure provides a good conference experience for all attendees
• Ensure attendees don’t use the conference network and Internet access to do things that will cause legal exposure for the conference
• Look for ways to help if any attendee is at risk

I went to Black Hat Asia 2024 assuming we’d need to watch over the attendees, not the folks who were leading talks from behind the podium on stage. As you’ll see, I was quite wrong! Read on to experience the education of a Black Hat NOC threat hunter.

What’s behind podium #1?

Sometimes you have to search for the needle in the haystack; sometimes you take a seat and find the needle right on your chair. This event belongs in the latter category.

While reviewing alerts in our Corelight Investigator platform, one of our threat hunters confronted a page of detections —228 in all —that all originated with the same host. The nice thing about this is that Investigator automatically gathered these into the 15 shown detections, based on the characteristics that made them unique. This makes analysis much easier, as there are fewer things to investigate, detections can be closed out faster, and analysts and threat hunters can get through more alerts in less time.

Viewing all the alerts simultaneously, we saw they came from the same source IP address and roughly described the same behavior. That made it a cinch to validate that this represented a likely compromise of an asset.

The next step was to rule out potential classroom behaviors, such as participation in a capture-the-flag or a lab exercise. We tagged all of the logs from the monitored traffic with the name and location of the source and destination of the traffic, so it was easy to confirm the source machine was in the general Wi-Fi network, and not on a classroom-specific network.

Next, we looked to see whether the alerts started suddenly, and if there was any correlation between the alerts and when the source machine was active on the network. We found that the machine appeared on the network at the same time that the alerts began. This was unlikely to be classroom behavior; the most likely scenario was that the machine was compromised with malware before it arrived at the conference, and we saw the indication of compromise as soon as the attendee joined the conference network.

We tried to identify the attendee, not least because it was possible they had no idea their machine was compromised. Network traffic often includes identifying information, because many network protocols are not designed with privacy in mind. For example, Apple devices often name themselves automatically based on the first name that a user enters when setting up the device and logging in for the first time. This hostname is broadcast across the network when the machine is attempting to obtain an IP address dynamically via DHCP. In other protocols, such as NTLM, Kerberos, HTTP, or LDAP (not an exhaustive list), usernames are revealed in the clear even if passwords are obfuscated or encrypted.

The Corelight Open network detection and response (NDR) platform includes a Known Entities package that automatically aggregates all of these potentially identifiable markers from various protocol logs into a subset that can be searched easily to identify hostnames, usernames, and services associated with a given IP address. Our threat hunters turned to these logs for evidence of the attendee’s identity. And we struck gold: The user had a unique first name that was present in their hostname. With a bit more research, we came to the startling conclusion that the attendee was actually presenting a talk at Black Hat. What’s more, they were on stage at that very moment.

With the rich alerts and telemetry that the Corelight Open NDR platform provides, the Black Hat NOC detected the situation within a few minutes of the presenter connecting their laptop to the conference network. We were able to have a staff member waiting by the side of the stage before the presenter’s talk concluded to discuss the situation with them.

We hoped that the alerts were part of a controlled demonstration involving live malware or interacting with a command and control system on purpose.

However, the presenter said that they were not doing anything of the sort. Our representative then broached the possibility that the laptop in question was infected.

The presenter expressed disbelief. “The laptop has an advanced endpoint detection and response (EDR) agent installed,” they insisted.

Working in the NDR space, we often confront the rationale that EDR prevents compromise, so you don’t need to worry after you install the agent.

In reality, while EDR provides significantly better prevention and detection than antivirus did, it is not a perfect solution (indeed, perfection has yet to hit the market). At Corelight, we believe the belt-and-suspenders approach of EDR+NDR gives security teams the best of both worlds: reduced frequency and severity of incidents, plus comprehensive visibility for more complete coverage and better agility for the defenders.

After we provided the evidence of compromise to the presenter, they contacted their IT staff to continue the investigation. Whether we sold them on the NDR + EDR approach is unclear, but we did our job for the NOC and the conference.

What’s behind podium #2?

Interesting stories can originate with even the most innocuous-looking alerts, which is why, in that perfect world, we’d review every one of them.

Thankfully, as noted above, Corelight Investigator has a built-in automatic scoring system, which ranks the possible alerts across several criteria and automatically prioritizes a subset of higher-value alerts for analysts. All alerts are available in the Log Search function for threat hunting, but analysts have a smaller pool of detections to focus on and triage before moving to threat hunting.

Below is another alert we received in Singapore. At first glance it was completely unassuming because from the Black Hat network, credentials are regularly posted directly to IP addresses without a domain name, often as part of a capture-the-flag event or a classroom lab exercise. We expected to look into this alert, confirm that it was classroom behavior and close it out quickly.

Our first clue that this investigation might get tricky was the fact that the device in question was on the general conference Wi-Fi, not a classroom network. But it was only a clue, not enough to prove or disprove anything, since attendees often perform classroom activities on the general Wi-Fi.

Searching for more clues, we dived into the evidence to see how many other attendees interacted with the destination, what credentials were used, or other evidence that would help identify the server.

The password used to log in was simple, which caught our attention:

The server header that identifies the server type was also interesting, since t it did not identify any of the well-known web servers or proxies commonly used. We’ve obfuscated it below because it is so uncommon, and details in this post could be used to search out and compromise them. But it provided a useful clue: We were able to perform OSINT reconnaissance on the server name and find a public posting identifying it as a water treatment controller.

In order to get more evidence to confirm this was, in fact, a water treatment controller, we wanted to look at the full session, instead of the logs that transcribe portions of the session and actions.

Luckily, Corelight Smart PCAP  makes that easy. Before the show started, we configured our Corelight Sensor to capture sessions based on our criteria, so that when we needed to dive deeper, the packets would be available to us. Smart PCAP makes an analyst’s life much easier, because sessions can be downloaded individually, and analysts don’t need to mess with Wireshark filters to get to the packets they want to inspect.

Once we had the session in question downloaded, we could then read through the entirety of the request and response and look for evidence that would confirm whether it involved a water treatment controller.

In the PCAP, there were references to “Walchem,” “W900,” “pH,” (a measure of acidity), “CCond,” (a measure of salinity), and “Temp” (presumably temperature). Hypothesis confirmed: The device in question was, indeed, a water treatment controller.

We then looked at the approximate location of the device based on IP geolocation services. Once we knew that the device was in the United States, we considered the likelihood that the attendee had a legitimate reason to log in to what was, likely, a piece of U.S. critical infrastructure from Singapore. There were only two realistic outcomes:

• The attendee did not have authorization or a legitimate reason to log in.
• The attendee did have authorization, but was unaware that they were exposing credentials to critical infrastructure by authenticating outside of an encrypted session.

After reviewing this information with Black Hat NOC Management, we decided that in either case, it was worth attempting to make contact with the attendee to discuss the matter with them.

Similar to our approach in identifying the attendee in the Podium #1 threat hunt, we used the Corelight Known Entities package and the logs that it produces to identify a potential first name. When we found one, we coupled that name with other available information and determined we were dealing with yet another Black Hat speaker, who was giving a presentation about a tool.

We then located the source code of this tool on GitHub and found snippets in the code that matched observed network behaviors and confirmed we had identified the correct individual.

When Black Hat NOC management approached the speaker to discuss the situation, they admitted to logging in as part of a demonstration of an open source tool. NOC management reminded the speaker that if they want to demonstrate a tool in the future, they should use their own targets instead of public services they don’t own or have authorization to use.

Black Hat is a very permissive network by design, open and educational. This sometimes gives people a false sense of authorization. Logging in to a system you don’t have authorization to use is illegal. Doing it from the Black Hat network doesn’t magically make it legal.

Conclusion

Corelight Open NDR makes it easy to monitor any network, from the most straight-laced corporate network to a Wild West of a network like the one we help protect at Black Hat. With powerful functionality like Smart PCAP, the Known Entities package and Investigator, Corelight makes it easy for an analyst to get to the evidence they need to answer pressing questions quickly, so they can triage alerts and close tickets faster, and get more work done.

Corelight would like to thank our peers and partners—Arista, Cisco, MyRepublic, NetWitness, and Palo Alto Networks—for partnering with us in the mission of protecting the Black Hat conference. We’re also extending an enormous thank-you to the Black Hat organizers for the educational service they provide to the InfoSec community and for selecting Corelight as a partner to help keep the Black Hat network secure.

We’ll see you all in Las Vegas at Black Hat USA 2024!