NDR
What the Black Hat NOC taught me about MCP & agentic SOCs (Chapter 2 of 4)
Discover what defending the Black Hat NOC taught me about using Model Context Protocol (MCP) to build an agentic SOC and accelerate threat hunting.
Whether they use custom implants for persistence, zero days for initial access, or live off the land (LOTL) to avoid detection, finding a state-sponsored adversary group can be a challenging proposition for defenders. This can be particularly true for adversaries that are sponsored by the People’s Republic of China (PRC). Historically, their focus has been on espionage and intellectual property theft. But more recent reports suggest the PRC is also prepositioning near operational technology (OT) systems so that they can disrupt military operations and civilian life in the event of open conflict with another nation. Threat intelligence feeds use a variety of names to describe the adversaries and their activity, but one of the most common designations is “Typhoon.”
Microsoft uses “Typhoon” to describe adversary groups that have ties to the PRC and then group them based on their tools, tactics, techniques, and procedures (TTPs). This gives rise to names like Volt Typhoon, Salt Typhoon, and others. Media outlets have focused on these adversaries while covering a series of high profile breaches of large telecommunications companies and critical infrastructure.
While these groups are not new, there is growing concern about their impact (or potential impact) on critical services like power, water, and waste treatment. Their intrusions have also prompted the US government to advise senior officials and politicians to avoid using normal text messaging and voice communications in favor of end-to-end encrypted channels.
Volt Typhoon (also known by the names BRONZE SILHOUETTE, Vanguard Panda, DEV-0391, UNC3236, Voltzite, or Insidious Taurus), one of the more prominently reported adversaries over the last year, has been involved in the attacks around critical infrastructure and OT systems. A 2024 CISA bulletin gives a detailed review of their activities and how they carry out their intrusions. A serious concern is their ability to exploit devices not managed by endpoint detection and response (EDR) and use them as initial footholds and persistence mechanisms into organizations. Network detection and response (NDR) complements EDR and captures the adversary’s activity in EDR blindspots.
These points of initial access are frequently devices such as firewalls, VPN appliances, and other publicly accessible network systems. Volt Typhoon has used known vulnerabilities and zero-day exploits to access these devices. Once the system has been compromised they are able to perform internal reconnaissance, move deeper into the network, and remain largely undetected.
Fortunately, all is not lost if they come calling to a network. Instead of giving up this ground to the adversary, let’s look at some methods using NDR to detect adversaries’ presence, determine where they are, and inform incident responders on how to eradicate them from the environment.
Analyzing threat intelligence
Let’s work through the process of threat hunting for Volt Typhoon with open-source threat intelligence using Corelight’s Open NDR Platform. We will use the previously mentioned bulletin by CISA as our source of information for the hunt. There is a lot of information in this report to use in hunts leveraging NDR and EDR. For our purposes, we will focus on those activities that are more difficult for EDR to detect.
The bulletin indicates Volt Typhoon has some key objectives and habits in how they work towards their objectives. Their main objective appears to be to establish access in a target organization and position themselves near critical assets like operational technology. They also want to harden their access in the environment, so that they are difficult to remove even when detected.
They are known to extract credentials from domain controllers and exfiltrate data from hosts like file servers. To do this, they will need to:
- Discover as much as they can about the network architecture
- Find critical assets like domain controllers, file servers, and OT systems
- Move laterally to those critical assets
- Elevate privileges and collect user credentials
- Establish persistent access to the network
Every adversary has to orient themselves to the environment where they’ve gained their initial access. While their initial compromise may be difficult to detect, their follow-on activity is usually much more visible. For example, ping sweeps, simple port scans, and failed authentication attempts from inside a network should only occur from specific, authorized hosts. These are all signs of an adversary attempting to find their way around a new network.
Hunting near the objective
Let’s focus on domain controllers to begin our threat hunt. We know they’re a likely target, but we don’t know from where in the network they could be targeted. The hypothesis here is that the adversary has gained access to some host in the network and will target the domain controllers to get credentials.
There are a number of location methods an attacker could use, but many adversaries, including Volt Typhoon, are known to use a network scan to find hosts.
While Corelight’s NDR metadata is enriched with Suricata signature-based detections, what if the adversary’s scan isn’t matched by a signature? We can still detect a scan because very few devices should be attempting to access a range of network ports on our domain controllers.
Here’s an example of a CrowdStrike LogScale query to find network scanning.
This query starts by looking at the Zeek® conn log for activity related to our domain controllers. In this case, these are 10.110.163.50 and 10.110.163.51. You can add more IP addresses as needed. Then it filters the data related to these IP addresses based on their connection state. We are interested in connection states where no reply was received (S0) or when a connection attempt was rejected (REJ).
Our query gives the following results:
We find that 10.110.154.113 has made 80 connection attempts that were either rejected or no reply was received.
We don’t know for certain if this is an adversary’s scan, so we’ll take a closer look at the connection attempts between our domain controllers and this IP address. To do that we will run another query to tell us what ports connection attempts were made to, what host it was on, and the state of that connection.
The results of this query make it look a lot like a port scan:
You can research each of the connection states in this result by using the Zeek Cheat Sheet. However, the requested ports and their varying state indicates that 10.110.154.113 tried to connect to 10.110.163.50 across a range of ports and are indicative of a port scan. We see ports for RPC (135), FTP (21), RDP (3389), LDAP (389), SMB (445), Kerberos (88), and various HTTP related ports (80, 443, 8080).
When we research this IP address we find that 10.110.154.113 is an employee workstation and is not authorized to perform network scanning. It may be that this host is not covered by EDR or has a policy that isn’t configured properly. Our investigation now pivots to this workstation to discover what has happened here.
We use the conn log again to triage what hosts are talking to the suspect workstation. This time our query is:
This gives us a list of the connections and times that connections have been made to our suspect workstation. Most of the connection activity looks normal, but there are two lines that stand out in the response to the query:
We find that 10.110.201.24 has made two connections to our suspect workstation: one on port 3389 and the other on port 445.
This is suspicious because 3389 is the default port for RDP and 445 is the port for Microsoft Server Message Block (SMB). It is abnormal for a workstation to act as a server, so now we need to figure out what 10.110.201.24 is.
We are able to use the MAC address to determine that 10.110.201.24 is a web camera that is on the network. While there might have been a reason for the activity we’ve discovered so far, there’s no reason for a web camera to log into another host. It’s now highly likely that we have discovered an intrusion in progress.
The use of an IoT device like this camera is something that Volt Typhoon and other adversaries will use to evade EDR detection. These devices usually come with default credentials and a poorly secured embedded Linux operating system. This makes them a great place to hide on the network, especially since they are frequently left with default configurations and not closely monitored by anyone.
Our next step is to determine if this host has been connecting to any other hosts in our environment. The query below looks for connections where the web camera is the source IP address (id.orig_h) and where the hosts responding are local IP addresses (local_resp = true).
There are limited results to this query. We find the connections to the workstation, but also find a connection back to a firewall at the IP address 10.110.100.5. That is concerning because we don’t know of any service that is supposed to be on port 7000 on the firewall. We also don’t know why the web camera connected to this port.
Let’s reverse this query and see what hosts have connected to the web camera. We change id.orig_h to id.resp_h, which makes our camera the destination IP address for any network connections. In the second and third lines we change id.resp_h to id.orig_h, since we want to see who initiated a connection with the camera.
The results of this query are more alarming. The firewall has made an SSH connection to the web camera, which includes the firewall into the intrusion.
Edge devices like firewalls and VPN appliances are not able to be monitored by EDR, so they represent a potential blindspot if not closely monitored by tools such as NDR. We could attempt to do a similar analysis of connections to and from the firewall, but it will be accepting many connections from hosts both inside and outside the network. However, we already have enough information to declare that we likely have an intrusion and start the incident response process.
NDR provides visibility across EDR blindspots
While this was a simulated intrusion, the TTPs used here follow those used by Volt Typhoon and many other adversaries. In this case, they compromised a firewall and found their way into an insecurely configured web camera. Fast Reverse Proxy (FRP) was set up on the firewall and the camera as a method to get deeper into the environment. Eventually, the adversary found their way to our workstation, where they found Advanced IP Scanner on the host already.
Perhaps the workstation had EDR, but it was configured to allow this tool. Or EDR was not deployed to the host because it was deemed not important enough to monitor as closely.
Regardless, the adversary used this tool to perform a port scan of possible domain controllers and found one. From there they could try to steal credentials to try to access the domain controller to access credentials for the entire environment.
We used Corelight Open NDR to perform a threat hunt based on the hypothesis that an adversary would attempt to access the domain controllers. While we started off looking for a port scan, we could have also searched for RDP connections from unusual hosts, suspicious SMB activity, or other TTPs. The path that the adversary took mostly occurred on systems that are unable to have EDR deployed on them. Volt Typhoon and other adversaries have learned to avoid EDR when possible, and there are many systems that they can operate from in pursuit of their objectives.
Corelight’s NDR platform provides strong visibility into these areas of the network to help organizations discover EDR evasive intrusions before they become an OT security breach. You can learn more about Open NDR here.