Skip to content
  • There are no suggestions because the search field is empty.
PROTECTING OVER $1B IN DAILY TRADES
DEFENDING ENERGY FOR 32+M U.S. USERS
SECURING NETWORKS FOR 52K+ TRANSPORT VEHICLES
PROTECTING OVER $10T IN MANAGED ASSETS
SECURING 16+M ANNUAL PATIENT VISITS
Home/Podcasts/Episode 2 - Inside the Black...
Episode 2 - Inside the Black Hat NOC: Defending a hostile conference network
Guest Speaker: Mark Overholser
November 20, 2025

Episode 2 - Inside the Black Hat NOC: Defending a hostile conference network

Episode 2 - Inside the Black Hat NOC: Defending a hostile conference network
0:00 / 0:00

About the episode

Richard Bejtlich talks with Corelight Principal Technical Marketing Engineer Mark Overholser about what it takes to run the Black Hat Network Operations Center and keep a "hostile" training network safe. They walk through how partners like Corelight, Cisco, Palo Alto Networks, Arista, and Lumen build and monitor the conference network, how the team tells lab traffic from real infections, and why misconfigured self hosted services still show up in surprising ways. Mark shares how the NOC works together in one room to investigate issues, when they decide to block or intervene, and practical advice for attendees on preparing their devices, monitoring their own traffic with tools like Zeek, and staying safe on conference Wi Fi without living out of a Faraday bag.

Episode transcript

Download transcript

Episode 2 - Inside the Black Hat NOC: Defending a hostile conference network

Gain clear, actionable intelligence from Corelight's network defense experts. Corelight' Defenders translates complex cybersecurity detection challenges into concise, practical episodes designed to support faster, smarter decision-making across modern security teams. Welcome to Corelight Defenders. I'm Richard

Bejtlich, strategist and author in residence at Corelight. In each episode, we explore insights from the front lines of NDR, Network Detection and Response. Today, I'm speaking with Mark Overholtzer, principal technical marketing Engineer at Corelight. Welcome, Mark.

Thank you for having me. The reason I asked you to be on the podcast was your involvement with Black Hat, and specifically, being part of the Black Hat NOC. Could you explain to people what that is and what your role is? I always explain when people wanna know more about the Black Hat Network Operations Center, the people who manage the network there don't rely, generally don't rely upon the conference center to provide the network or the internet or the bandwidth or any of those sorts of things. Part and parcel of that is having a network operations center to ensure that things stay up, stay reliable, stay performant, and, uh, above all else, stay safe, you know? It's a conference that centers around teaching people offensive security tools and offensive security tactics, um, you know, red team stuff.

Uh, and sometimes people wanna practice those things . We need to make sure that everybody stays safe in the process. Do you guys end up contracting for your own, uh, pipe to the internet? Or are you, at the end of the day, still coming out of some hotel? Uh, and that depends on the conference.

Uh, for two out of the three, yes, we contract, uh, one of the partners. So Black Hat brings in partners to supply the various things. So, like, Palo Alto Networks brings the firewalls, Corelight brings network detection and response equipment, Cisco brings DNS and mobile device management and, uh, some other things. Arista brings the wireless access points. And then, in addition to that, for a couple of those conferences, there's a bandwidth provider. It would be somebody like Lumen or MyRepublic. Um, but for at least one of the conferences, we do ultimately come out through the, uh, venue's internet pipe, even if we're supplying the majority of the other switching, routing, and wireless access infrastructure from the network operations center. What kind of data do you have on hand to try to make those decisions? Oh, yeah, that's a great question.

I, you know, people, I think, imagine that, for instance, we go in and we know, like, "Oh, these are all the training servers, and those are all the, um, whatever." We do have a lot of labels for, for things. So for example, the networks are all divided up, uh, based on usage. So e- each classroom gets their own Wi-Fi and so their own, uh, SID, their own VLAN and, uh, and network segment, all of that sort of stuff. So we can tell, like, when somebody is doing something on the network. We can tell what, uh, classroom they're in or if they're on the general Wi-Fi or... A- and there are various other segments for other internal uses as well, um, 'cause there's lots of different s- pieces of the puzzle for the conference that fit together. Um, but we don't... And, and we know about the infrastructure that everybody tells us they're going to bring, but for instance, trainers don't tell us about the infrastructure that they're going to bring. They could walk up one day, um, th- day one of the training and, uh, you know, have, uh, uh, four servers in a duffel bag and, and go set them up in the classroom and, uh, and, and we get, we get no advance warning of that. Uh, or they, they could show up and have 30 VMs all stood up in, uh, you know, Google Cloud, for example, and w- we would have no knowledge of that either. So sometimes we know very specific things, like we know this is, uh, the registration server that, that got, um, moved on site or something like that that's provided by the conference. But for things that trainers bring, we have e- essentially no advance warning, and f- for those, we just have to infer what something is based on, based on the traffic. We do, we, uh, do a lot of inference, like if we see somebody, uh, that's causing a lot of intrusion detection alerts because they look like they're attacking something, we'll zoom out and see, like, uh, are they in a classroom? What is the topic of the classroom?

Like, what is the name of the training? What is the s- the synopsis of the training? And, uh, are other people also attacking that thing? Are other people performing the same s- types of attacks maybe against different endpoints, but all of those endpoints are hosted in AWS or in GCP? Like, we can sort of piece together the puzzle by looking at all of those things to say, "Okay, n- I have a classroom. There are, uh, 15 active IPs in the classroom. 12 of them are all attacking things that are in GCP." I can say, "It's probably lab time," right? If all of those things started around the same time and are ongoing. Uh, but for instance, if, uh, we have a different classroom and the classroom is about, like, defensive things and somebody's attacking something and they're the only one that's doing it, that's probably extraneous behavior or somebody p- painting outside the lines. Right? You know, I have definitely gone to classrooms to get more information sometimes, right? Like, uh, we had a situation in, um, Black

Hat US where we saw what looked like a malware infection, but it was coming from a classroom that the title of the class was, uh, malware analysis. It was, like, advanced malware analysis. And so, you know, we c- we constructed a couple of different hypotheses, and one was somebody in the classroom is infected. Another was somebody intentionally infected something like a VM on their own machine to do some analysis. Another was, you know, they were replaying some traffic as a demonstration. And, right, so I just went to the classroom, and I found the instructor on break and said, "Hey..."... this is what we're seeing. Um, do you know, like, can you help me eliminate some of these hypotheses? Uh, and in the end, uh, we did. We eliminated, uh, all of them except for somebody was infected, and then we found that individual and let them know.

Um, right, and sometimes that's what we have to do because we don't have, I don't have a full syllabus, I don't have lab materials, I don't have any of those things from the instructors. We just get, get what we need on demand. So that in-person aspect, there's two parts of it that I'd like to talk about. So one of them is visit a physical location, potentially stand in front of a classroom and look everyone in the eye and say, "I notice something in here that should not be happening." And possibly you see somebody fidgeting and you figure out, okay, something is going on here. Basically every conference, we do have to send somebody to a classroom to find an individual and pull them aside to have a conversation with them, or to address the classroom if we don't know which specific individual it is within the classroom. And usually it's just to have, you know, say, "Hey, we saw this and we want to make sure that you're aware of it in case you need to do anything." Um, but sometimes it's, "Hey, cut it out." The second physical aspect, you're all,

I'm guessing, in one room doing this work? Correct. Yeah, we do. We have a room. It is the network operations center, um, and all of the individuals from all the partners that I talked about, plus Black Hat staff, we're all sitted, seated, we're all seated physically in that room, which is a, also a huge advantage. Because if, even though we're all communicating with each other on our private, um, uh, chat instance, uh, to, to also discuss things and to document things, I can also just get up and go find, you know, the wireless people and ask them a question if it's truly urgent, or go flag down one of the firewall people. Or just stand up and say, "Hey, everybody, I have this and I think it's a situation. Can we all take a look, uh, at this particular thing together?" And that, that is also a huge advantage because it, um, it, we can remain collaborative. It can sometimes be distracting. Uh, I mean, having 40 people in a room, all working individually and together, but also movies playing on, uh, the wall and loud music playing, like it can get distracting. But it is, um, having everybody at your fingertips really makes it easier to, to do this sort of deep investigation that we need to do sometimes.

Do you all have access to the same data or are you sort of siloed based on technology? We offer each other access to our tools. We all have access to basically everything, but, um, we often collaborate anyway just because it's, we're, we're often faster, by doing it that way. Okay, that's, that's really good to hear. That's one of the issues I often worry about in some operation centers is people are tied to a certain tool, like they're the antivirus person and someone else is the firewall person. And, uh, it's tough to grow when you, when you're siloed according to your technology type. It's kind of demoralizing, I think. So I think everyone should have access to everything, you know, within privacy grounds. And so you talked about having to visit people, but do you have any other sort of response options? Like you mentioned the Wi-Fi providers. I'm guessing they could block something, you know, they could, uh, deauthorize someone from the wireless network. Yeah, for sure. That is definitely something that we would need to do in an extreme, uh, situation. Usually things are, we catch them and before they get too out of hand, and we can just go have a conversation with somebody. But if need be, yeah, we can block people from the wireless, we can put in, uh, firewall rules and things like that. You know, we'll, we'll often block external IPs that are poking at things that they're not supposed to be. Um, it's pretty rare that we'll block an attendee without trying to make contact first. Yeah. But the option is always there. Have you gotten in, into any running battles where you block and then they change MAC addresses or something like that?

Oh, um, nobody's been that cheeky. I think, uh, we, we often get questions like this as well, right? Like, oh, everybody there must be really naughty and mischievous. And, uh, you know, just because it's a, it's a, the conference is called

Black Hat for like what, of course people are gonna, uh, try to do things. But in addition to it being Black Hat, I, people are, they're there to learn usually. I mean I, it's, it's, we call it a hostile network, but really it's an educational network that just happens to have a lot of hostile traffic because there's people teaching techniques that in other contexts would be considered hostile. And, uh, but everybody's paying money to be there, and oftentimes it's their employer that's paying them, or paying for it, for them to be there. And so, you know, nobody really wants to get kicked out because they don't want to have to go back to their boss with their tail between their legs and say, "Hey, I got kicked out of Black

Hat and the, you know, X amount of money that you spent for me to go take that training, I only got the first day and a half of it, and then they sent me packing," right? So- Yeah. ... I think we have the advantage there. Yeah, I 100% agree with that. Black Hat is not cheap, but it's worth it. And if you're there to fool around and get kicked out or to try to avoid getting kicked out by somehow thinking you're going to be clever, that's a, that's a waste of time, I think. Have you noticed any trends over the last few years? You've been doing this for many conferences now. Anything that is interesting?

I mean, one of the things that, uh, that sort of surprised me was just, uh, how, I don't want to say like how frequently we see infections, because it makes it sound like it's a very frequent thing. But I came in expecting to see like an infection maybe every three conferences. And it seems like we see maybe one or two or three every conference, which I found very surprising. So, but I think that's just a numbers game. You get enough thousands of people in one place, somebody's bound to have something, uh, that's wrong with one of their machines. But what we do see, I mean, consistently is...... uh, self-hosted things that are hosted, uh, in an unsafe way. Um, you know, we had, uh, an attendee connect to a chat application, and by all accounts, it seemed like it was being hosted by an enterprise. May not have been a huge enterprise, but it was, it was hosted, self-hosted by the company, and was used for company purposes, and, uh, but all of the chats, all of the channels, all of the usernames, like absolutely everything was in the clear, including file transmissions. And we saw somebody interact with somebody else on- that chat, and they transferred a file, and that file ended up containing a database. Like, it was a small database, but it was, uh, it was a database that had, like, a- all of the employees' information in it. And it's, it's things like that that we see over and over again, where it's, you know, somebody hosting a mail server, and maybe they're doing it 'cause they're trying to learn. But in the end, you know ... Or maybe they're trying to do it because they, they want better privacy, you know, get, keep their information a- away from big tech companies gobbling it up. But in the end, the, it's less safe than just having somebody else host it for them, because by not encrypting their network traffic, absolutely everything that they're doing is available to anyone listening. Yeah. Yeah, I could imagine an era where so many people work remotely that if you were just always on somebody else's network, and even if you're at home on your network, most people don't look at it or know how to look at it or have the infrastructure to be able to do that, then it could be pretty easy to either be compromised or to, like you said, host something that is not, uh, you know, that's not gonna survive out in the wild. So, the first time somebody actually notices could be in your, your environment where y- that's what you're there for, you're, you're there to pay attention.

Yep. And, and, and we do, and we pay attention, and when we see things like, like the, the mail server, we saw a couple of different mail servers, we just figure out who it is and go tell 'em, right? Yeah. Because it's, it's the kindest thing we can do, say, "Hey, we loo- it looks like ... You know, great job, first of all.

Setting up your own mail server is not trivial. Um, and g- great job pulling it off. But on the other hand, you probably wanna know that, you know, all your traffic is in the clear, and, uh, we really hope you enjoy your, your cruise that you're going on in two weeks." Do you have any advice for someone going to one of these conferences? What's the best way to prepare and to, uh, not get a visit from, from the NOC while you're, while you're studying? Well, I suppose one of the things, uh, I encourage people to do is, uh, to monitor your own network traffic, uh, at home, right? If you're, if you're gonna be bringing devices to the conference, if you download something like Zeek and use it to look at your own traffic on your own network and see if there's anything there that you don't expect to see, anything that's unencrypted or anything that's, uh, leaking i- information that you don't want it to be, then that gives you a leg up before you come to the conference. I think a lot of people are probably expecting me to s- you know, say the, the, um, to give the advice that, uh, you should buy several Faraday bags on Amazon and, uh, and put all your devices in those and never con- never connect to the conference WiFi. And obviously I'm biased 'cause

I have the privilege of looking at the, the conference traffic, but I don't think that's in everyone's best interest necessarily. I think the ... We do a lot to, uh, ensure that the conference network is safe, uh, and we do a lot to look and, um, find these things that might be wrong and alert people to them. So f- I think people should feel comfor- comfortable coming to a conference like Black Hat and connecting to the network, uh, and we'll try to tell you if something's wrong.

Yeah. But ... Well, I, I appreciate the work you do there, Mark. I would not be concerned at all going to a Black Hat conference when you and the team are there working on keeping everyone, uh, safe and productive, so appreciate that. Thank you for joining me today. Thank you for having me, Richard. Thank you for joining us on the Network Defenders. podcast, sponsored by Corelight. We will see you on the network.

You've been listening to Corelight. Defenders. To stay informed with expert intelligence on today's cybersecurity challenges, please subscribe to ensure you never miss an episode. We'll see you on the network.