Thrown in the deep end: My first time hunting in the Black Hat NOC

If you have even a passing interest in cybersecurity, you no doubt have heard of Black Hat, the eponymously named conference that launched in 1997 in Las Vegas. Nearly 30 years later Black Hat is a global phenomenon that bears little resemblance to its humble, hacker-led origins. And yet, the core idea of giving security teams insights into the minds, tools and techniques of hackers, is alive and well.

It was with some trepidation that I approached Corelight’s Black Hat network operations center (NOC) lead to ask if I could join the team for Black Hat EMEA in London. Unlike many of Corelight’s Black Hat veterans, I had limited exposure to security operations centers (SOC), NOCs, or other venues where the goal is to find attacks in real time. Luckily for me, the team accepted me — and this blog is a digest of what I experienced.

Whilst I do know the tools deployed in the NOC, this would be my first experience of working with real data in real time — and with meaningful consequences if I made a mistake that wasn’t picked up by one of my colleagues. Fortunately, I knew I would be serving with some seriously experienced individuals. Still a bit nervous about my capabilities, I flew to London confident that I was in good hands and hopeful I wouldn’t be a burden, at least after a day or two.

If you’re curious about Black Hat and how it works I strongly suggest you read this blog by my esteemed colleague Mark Overholser. Here, I will just mention that the conference runs on its own network, which you must access to connect and access any of the facilities and available resources. This proprietary network is what I helped monitor.

I also won’t go into detail about what we found and when (you can read up Corelight’s Black Hat adventures and findings here and here). Rather, I want to convey my experience of being a member of the team. What it was like to be a threat hunter for a week, on tools that I had only experienced when positioning them in front of customers, relying on their experiences and my knowledge to connect the dots?

Getting up and running — fast

The first thing that really jumped out at me was that the tools work! The approaches I have been running through with customers for several years really are a great starting point. I found they helped me quickly connect the dots between different traffic flows. I even identified an external attack only moments after one of my colleagues had processed the same alerts, but had yet to record the attack on our shared Slack channel. It was a great boost to my confidence to contribute to our mission and understand what was happening.

During the setup on Saturday and Sunday, I developed my skills using LogScale query language, which we use in Corelight Investigator. I significantly increased the complexity of my searches, building and joining tables to identify the connections between alerts, traffic patterns, and the devices on the network. I delved deeper into the various alerts we received, understanding how they related to the network configuration and what they were actually telling me. At this point, I was really feeling very proud of myself and starting to wonder what all the fuss was about. 

…And then the conference opened

Suddenly things that I talk to customers about on a daily basis became very real. 

First, the sheer volume of ‘stuff’ that happens on a network as people go through their daily life is staggering. We’re not all the same and everyone has a very different sense of what’s acceptable on public or free Wifi such as the Black Hat network. But as the newbie, it was easy to get overwhelmed with a multitude of alerts. 

The key challenge is understanding the context. Black Hat is not a corporate environment; many of the attendees use tools and applications that are best described as “outside the mainstream.” For me, it was a huge learning exercise to be able to establish my own baselines for determining how these alerts were connected and what was actually happening.

Gradually, I felt more confident in what I was seeing and why, and I undertook a deeper analysis of some of the alerts and why they were triggering. This meant not only using the searches I had been developing in LogScale, but also looking at the full PCAPs that Corelight had captured and were associated with each alert. I was now starting to understand not only the network and the behaviors, but also the things the alerts were looking for and why they were triggering. 

Ultimately though, I was still just responding to the huge volumes of alerts that were coming in. And I realized that I was in the midst of another real-time experience: alert fatigue. As mentioned, we weren’t doing much at the tool level to suppress or filter alerts; we preferred to capture as much detail as we could. That did mean, though, that I started to mentally ignore certain alerts, knowing that every one I had looked at so far had been a false positive. I could identify the high level information and know that it wasn’t a concern. 

It was at this stage that my more experienced colleagues suggested that I might want to start taking more of a threat hunting approach, rather than simply looking at alerts. 

There’s a number of ways that you can go about threat hunting — and in the NOC we have numerous dashboards colleagues have built specifically for Black Hat, based on past experiences. This of course is no different to what many of Corelight’s customers do to investigate the weird or anomalous in their particular environments.

For me, I decided I was going to go back to the Corelight data, especially the data that didn’t necessarily produce an alert. Unfortunately, there isn’t the room here to go through the entire process, but by starting with some data found in the Corelight ‘Weird’ log and the Corelight SSL log, and comparing my findings to the Corelight Encrypted Traffic insights dashboard, I found a genuinely interesting scenario across a couple of devices.

In a nutshell, these devices appeared to be visiting an extremely large number of what are best described as X-rated websites pretty much consistently throughout the day. However, there wasn’t really any matching DNS traffic or indeed HTTP or HTTPS traffic. Some further analysis demonstrated that each connection was using a unique user agent string; in other words, the user appeared to have hundreds of different browsers installed on their device. Additionally, I realized that the traffic was being routed through an unnamed VPN via IP addresses that were generating numerous alerts, as they were related to sites that most folks wouldn’t want to visit.

Ultimately, thanks to a pointer from my colleague Eldon, we discovered that everything was related to a tool that promised to earn money without you having to do anything. The application does indeed connect to a VPN and then, using a unique user agent, makes fake visits to numerous websites, the vast majority being adult, and presumably lays claim to some kind of affiliate advertising revenue that is then shared with the end user.

The issue here is that none of this is exposed to the user, but they are leaving a network trail that implies they are spending the entire day very differently from what they are presumably actually doing. This isn’t all that unusual behavior at Black Hat, but in a corporate environment it might raise more than a few alarms.

The peak of my week, however, came when I was investigating the situation above. I wrote a search for endpoints with more than five unique user agents. Another colleague, Ben Reardon, a seasoned threat hunter and veteran of numerous Black Hat conferences (and author of the aforementioned blog post about building a detection for SSHAMBLE), told me that he had never thought of that as a search, thought it was a great idea, and would add it to his search kit bag in future. This demonstrated that a: beginner’s luck is a thing, b: threat hunting doesn’t have to be super-complicated, and c: there’s always new areas we can search, even once we’re seasoned professionals.

When we finished on Thursday, I was completely exhausted, mentally and physically. I had a whole new level of respect for SOC analysts and their ability to perform consistently, day after day. 

However, the experience of using Corelight Investigator in a real-world environment really piqued my interest in threat hunting and thinking about new things to find in a network, as well as novel ways to search them out. Nothing is guaranteed, and I certainly don’t rate myself as a skilled analyst, but I am very much hoping I get the opportunity to be part of the Black Hat NOCs in 2025 and beyond. 

Thank you to the Black Hat team, and to our partners Arista, Cisco, and Palo Alto Networks for a great conference!