The Art of Team Building: Blueprints from the Black Hat NOC

It has been a distinct honor to be a part of the Corelight team that helped defend this year’s Black Hat events. I started the event season in the Network Operations Center (NOC) at Black Hat Asia, and then capped it off at Black Hat in Las Vegas. In this blog I’ll share my experience and learnings from participating in both NOCs. For a more detailed recounting of my time at Black Hat Asia, I encourage you to check out this blog to find out what the setup process was like, and this blog to find out what our NOC team discovered during the event. Overall, here is what I learned.

Scope yourself

When given the opportunity to hunt for potential threats, I get a real thrill out of the chase. There's something exhilarating about trying to track down a threat actor or identify a vulnerability. It's like being an uber-technical detective.

There's nothing quite like the elation of finding a true positive (TP). Knowing that you've helped protect your organization from a threat provides a real sense of accomplishment. However, there are also times when you'll come across a dead end or a false positive (FP). That can be discouraging, but it's important to remember that it's all part of the job. And when you're feeling stuck, looking for inspiration in unusual places can be helpful. Sometimes, the weirdest or rarest things can be the most beneficial in solving a problem. Feel free to think outside the box and explore new ideas! This means approaching situations from different perspectives and developing creative solutions to problems while adapting to change, as the threat landscape is constantly evolving.

While the validity of a goldfish having a few seconds of memory retention is highly debatable, it was one of my favorite quips heard while watching Ted Lasso. I've used that perspective once or twice when coaching my children's various sports interests. Regardless of accuracy, the statement emphasizes the importance of learning from your mistakes and moving on quickly. Don't waste significant time on searches or queries that are not fruitful. Instead, cut your losses and move on to something else with a better chance of success. And don't waste time on things that are not real or achievable by getting caught up in unrealistic expectations or goals. Instead, focus on what is feasible and make sure your efforts are aligned with your goals.

Of course, experiencing too much thrill chasing can also lead to fatigue. When you're constantly bombarded with alerts, it can take a lot of effort to stay focused and identify the important ones. That is why it's essential to have a system for triaging alerts and prioritizing your work. It's also crucial to avoid burnout. Information security is a demanding field, and it's easy to get burned out without adequate self-care. Make sure to take breaks, exercise, and get enough sleep whenever possible. Clear your head, recharge your batteries, and get back at it refreshed and ready.

Scope your team

Some of the most essential aspects of building a supportive team environment are positive communication and timely coordination. A healthy dialogue sprinkled with humor can help to break the ice, build rapport, and create an air of camaraderie, a sense of shared identity and purpose that can help to motivate and inspire team members. When working on a team, it is important to know what others are working on, share information and resources, and work together to achieve common goals.

It is essential to seek out the views of others, even if they are different from your own. Other team members may have insights that can help you see the problem from a different perspective. Each team member brings their unique skills, experience, and interests to the table. This diversity can be valuable, allowing the team to draw on a wide range of knowledge and expertise through challenging hunts, breach identifications, and response actions.

And don't be afraid to grow your team. It doesn't take "additional headcount" to find others interested in providing value toward the greater good. In the past, I've found opportunities to expand teams well beyond the information security org chart, including personnel in other areas of the organization. Cultivating a positive culture increases goodwill and affords a more comprehensive group of analysts, hunters, and doers.

Scope your environment

What information is the most important to your organization? This could include customer data, financial information, intellectual property, or other sensitive data. Identifying and addressing weaknesses related to the organization's crown jewels that a threat actor could exploit is a high priority.

As a quick refresher, the crown jewels are the most valuable assets of an organization. From the aspect of information security, the worst that could happen is that an attacker could compromise or leak these assets. This could lead to financial loss, identity theft, or damage to the organization's reputation. It's imperative to understand what these critical assets are and develop a comprehensive strategy to defend against unauthorized access to them, even when hunting on the Black Hat network. We take pride in affording Black Hat attendees a (hopefully) stress-free experience of using the event-provided networks without fear of their personal devices being compromised, unless it's their choice, and we also help protect Black Hat's critical assets in tandem.

The organization's environment may prove to be reasonably static or wildly dynamic. A static environment does not change often, but a dynamic environment varies frequently, like the Black Hat network! Baselines can be created for static and dynamic environments, offering a way of measuring the environment's regular activity. By comparing current activity to the baseline, analysts can then identify unusual activity that may indicate an attack, and not all threats are created equal. Some are more likely to occur than others, and some will have a more significant impact on your organization if they are successful. Prioritize your threats so that you can direct your resources to the most serious ones.

Additionally, bad guys can and do make mistakes. Information security analysts and threat hunters can use easy and low-cost searches and hunts to leverage these mistakes. For example, analysts can look for common errors that attackers make, such as using repetitive information in X509 certificates or reusing their attack infrastructure. Analysts can also look for unusual activity on the network, such as large data transfers or unusual login attempts, with these indicators to quickly identify malicious traffic. I've also uncovered a few findings by referencing Zeek's weird.log, looking for edge cases where attackers get caught trying to abuse protocols in nontraditional ways.

Scope your findings

Now that you have a clear understanding of your team's dynamics, the environment you're entrusted with protecting, and how to prepare yourself to hunt threats within that environment, let's finish with discussing how to adequately assess and explain any potential findings your team may encounter during daily operations.

This may seem like a challenging approach, but attempting to take a broad view of a situation and then magnifying it to highlight specific details can prove rewarding. Making this effort can be a beneficial way to understand complex problems and to develop solutions that are both effective and feasible, which is best accomplished as a group effort. Don't be afraid to play "devil's advocate" to explore these outcomes more thoroughly, but remember to keep the overall goals in mind when making decisions or taking action. It can be easy to get bogged down in the details. As a team, step back and look at the big picture occasionally to ensure you are on track.

When explaining your findings, it is important to consider the target audience and your desired outcome. For example, suppose you are pitching to decision-makers. In that case, you may need to provide more evidence and technical detail than if you are raising these concerns to others unfamiliar with the event or incident. The amount of evidence and technical detail you must provide will depend on this. Generally, providing enough evidence to support your claims is essential, but not so much that it overwhelms the audience. It would help to tailor the level of technical detail to the audience's level of understanding.

Summary

A high-functioning NOC is composed of people and processes first; the technology is only there to support them and make them faster and more accurate. This is core to our experience at Black Hat, and I think it is something that our community should talk about more regularly. Without highly productive technical teams of healthy hunters defending our organizations by watching the wire, where would we be? Iterating, improving, evaluating, and repeating through well-defined objectives is the cornerstone of producing and maintaining these watchers.

I encourage you to read these blogs from my fellow teammates about their experience within the Black Hat NOC:

• Blackhat NOC USA 2023 - A tale of sharp needles in a stack of dull needles
• Black Hat NOC USA 2023: Five takeaways for SOC teams

These posts are full of technical details, high-value findings, and next-level outcomes detected by leveraging our amazing platforms at Black Hat since that is what Corelight has a history of providing: the best evidence, the right telemetry, and combined analytics into a single source of truth within our OpenNDR Platform.

Want to learn more? Check out how Corelight Open NDR can empower your security team to disrupt attacks.