Hunting GTPDOOR: The case of the "Black Hat Positive"

Ben Reardon, Lead Researcher
Corelight Labs / NOC crew 

I'm a researcher on the Labs team at Corelight and, for me, working in the Black Hat Network Operations Center (NOC) at the USA show in Las Vegas is up there as one of the most interesting and intense activities on the calendar. As Black Hat’s trusted NDR provider, our team of hunters gets to use our own NDR technology and have a front-row seat to some of the most interesting network traffic on the planet, right beside the other NOC partners Arista, Cisco, Lumen, and Palo Alto Networks. It’s a high-stakes environment where we hunt for real threats amidst a sea of security training and demonstrations. 

This year, a particularly unusual detection led me down a fascinating rabbit hole, and in this blog I’ll walk through this epic hunt. In the blog I’ll cover:

• Detection
• Security posture and policy violations 
• Good old fashioned hunting
• Packet forensics 
• Binary decoding 
• Refining a theory in small scale
• Building out the theory to scale 
• Collaboration between researchers

Apart from the technical aspects, which I think are quite compelling, that last bullet point is a great example of the security community at work, and in particular seems to always play out at conferences like Black Hat. 

The hunt begins: a GTPDOOR alert

The story starts while I was hunting through our detections in Investigator, Corelight's Open NDR SaaS Platform, when two Suricata® alerts caught my eye: 

ET EXPLOIT GTPDoor Trigger Packet Request

ET EXPLOIT GTPDoor Trigger Packet Response

The fact that we had separate request and response alerts firing also indicated that something interesting was underway.

I wasn’t very familiar with the GTPDOOR malware, though very soon I would be. All I knew is that it was Telco related and that it was similar to another piece of malware called BPFDOOR. Among other things, BPFDOOR listens on the network for a “magic packet sequence” before taking action. These magic packets can be thought of as a secret handshake or secret knock. In fact, the precursor to magic packet sequence malware is a technique I had learned about early in my career called “port knocking,” where the secret lay not in the packets sequence themselves, but the secret sequence of TCP ports that were “knocked” prior to communications being opened. So, you would only gain access if you knew the right “knock.”

This style of C2 has always intrigued me due to the level of sophistication and covert nature, so I couldn’t help but to dig deeper:

My first step was to scrutinize the alert logic itself. In summary, the Suricata alert used the following logic: 

Protocol/Port: It specifically monitors UDP traffic destined for UDP port 2123. This port is used by GTP (GPRS Tunneling Protocol).

Payload Content: The rules look for specific hexadecimal values at precise locations within the packet's data. This fingerprint represents the default GTPDoor magic byte sequence, which the malware itself looks for in the network traffic!

Throttling: The alert triggers only once every 10 minutes per source IP address. This is to reduce redundant alerts.

After a couple of minutes review, I assessed that the alert logic was robust and had very little chance of false positives. It seemed that we were dealing with a genuine detection, something was definitely happening on the network, and so the hunt continued.

The alert itself also had a handy reference to a blog that explained the malware in incredible detail. As I describe later in the blog, I have since made contact with the author of that blog and compared notes. 

GTPDOOR is not your ordinary malware

GTPDOOR is sophisticated, targeting the Telecommunications sector; it has been attributed by Crowdstrike to APT group LIMINAL PANDA. It is specifically designed to use the GPRS Tunneling Protocol (GTP), which is used in the core of telecommunications networks. What is interesting is that it operates on the GTP control plane (GTP-C) to create a persistent, stealthy backdoor. As described above, the communication is authenticated by the malware by using a very specific "magic byte sequence," which is precisely what the Suricata rule was designed to find.

Connecting the dots

Using our Investigator SIEM (which is powered by CrowdStrike Falcon® LogScale), I quickly found the source and destination IPs from the alert. A quick pivot on the Zeek DHCP logs revealed the hostnames associated with the two IPs.

The attacker's machine name looked to contain a person's name

altaf-ThinkPad-T14s-Gen-1

While the target host (the one with the backdoor installed on it) was conspicuously named:

fast5g

The plot thickens.

All of this specificity gave rise to a hunch that it might be a researcher, and prompted me to check if there was any content being delivered at Black Hat that might inspire such specific interest. A quick search on the Black Hat site for "5G" revealed there was indeed a training and briefing on the subject.

These were being delivered by a Black Hat speaker/trainer named (you guessed it) Altaf who likely owns altaf-ThinkPad-T14s-Gen-1

This is clearly more than a coincidence–the attacker's machine name matched that of the instructor, as did the context. This led me to suspect a true positive detection but that was not malicious, and likely the result of a demo, training, or preparation for a talk. This scenario is what we in the NOC call a "Black Hat Positive" - where the true positive is based on network traffic that the rule is intended to detect, but is not malicious in intent. This is a fairly common scenario at Black Hat given the amount of cutting-edge security training that’s happening simultaneously.

So we were able to stand down on that alert, but still had two more points of interest to cover.

A deeper look and a policy violation

Even though we knew the intent wasn't malicious, I still wanted to see the packets. This was easy enough with Corelight's Smart PCAP, as a link to the PCAP contained in the spcap.url field within Zeek’s conn log as shown below:

I downloaded that full PCAP and opened it within Wireshark:

When analyzing the PCAP shown in the following Wireshark screenshot above, two things were immediately clear:

1. The payload was encrypted (or WAS it? Read on!).
2. When I bounced this detection around the NOC crew, our lead Mark Overholser rightly queried why traffic was flowing between two adjacent hosts on the general Wi-Fi network. In past Black Hat shows we had host isolation on gen Wi-Fi, but seemingly not here this time! This was a good example of the value of having a team of seasoned experts, as everyone brings their own lens and experiences to help solve complex problems early. 

That second point is a real problem. Peer-to-peer traffic on the general Wi-Fi network should not be allowed, as it can be used for lateral movement. 

Based on this policy violation, Mark worked with the network architect and our NOC partners to confirm that host isolation indeed was not enabled on that SSID. The direction was quickly given to enable it immediately. This was a great outcome for network hygiene, and a great example of the serendipity that often comes from threat hunting, as this policy violation was only highlighted because we were investigating these GTPDOOR alerts.

With that initial policy issue resolved, my hunt continued…

A chance encounter and a Zeek-powered breakthrough

That evening, I was lucky enough to score an invite to the Black Hat speaker's dinner at the Border Grill at the Mandalay Bay Resort. Toward the end of dinner, a large catering cart was being ushered through the large crowd of speakers. I and the fellow next to me both jumped back, standing side-by-side looking down at our toes to make sure the cart didn't run us over. I looked up at him and said, "mind your toes everyone." And then I paused. I thought, wait a second, this guy looks familiar. I asked, "By any chance, are you giving a talk on 5G?" He smiled and said yes. I found Altaf!

I told Altaf about the GTPDOOR alert we saw up in the NOC and the subsequent hunting I did earlier that day, and his eyes lit up. We spent time discussing the intricacies of GTPDOOR, its magic packet, the control plane C2 channel, and its payload encryption. As he was talking about some of the nuances of the GTP protocol and the backdoor, I formulated a way to decode the encrypted C2 payload on the fly. I thought the whole decoding process could be scripted using Zeek® to hook into the stream, extract the key, and decrypt the payload, lifting the veil on the malware. That chance meeting with Altaf had given me the final piece of the puzzle I needed to build the script.

The next day I attended my new pal Altaf's “5G Titanic” briefing, where GTPDOOR was mentioned. I was also glad to see him in the NOC summary briefing by Bart, Grifter, Pope, and Spicer at the end of the conference. 

Decoding the payload (AKA the FUN part)

The next day I set about testing the decoding theory that I had formulated while chatting with Altaf at the speaker’s dinner. 

Here is the manual process I went through to test the theory by hand before finally automating it with the Zeek script.

The hardcoded key

At this point I had read HaxRob’s blog on GTPDOOR about 4 times—something I touch on later in this blog when I connect with HaxRob. His blog really is a master class in reverse engineering.To be honest, I don’t fully understand a lot of the awesome content highlighted in reverse engineering blogs, so I tend to skim through them and pick out the network related parts that I know I can work with.

In this case, there was one “key” piece in the HaxRob blog that stood out to me–the default XOR key was baked into the malware sample. Below is a screenshot of the portion of malware code that checks for that key.

So at this point I knew the default key used by this malware was 135798642. As HaxRob pointed out in his blog, this number is someone typing odd numbers from 1 up to 9 and then evens back to 2–this is very human. The blog also includes a lot of information on the structure of the malware, but for this exercise let’s assume that all I knew was this XOR key.

Now 135798642 is decimal base10, and for bit-wise XOR we need to have it expressed in binary. Network data is usually represented as a Hexadecimal (Base 16) and since not everyone is overly familiar with the Hexadecimal, let’s make that conversion from base 10 to base 16 manually. You’ll probably only do/see this once in your career as there are tools that can do this for you and we’ll be using one soon enough, but some may find it helpful to have seen it done manually at least once.

Here’s how to convert a decimal to a hexadecimal:

1. Divide the number by 16 and record the remainder.
2. Use the quotient for the next division by 16.
3. Repeat until the quotient is zero.
4. The remainders, read in reverse order, give you the hexadecimal representation.

135798642 ÷ 16 = 8487415, remainder 2

8487415     ÷ 16 = 530463,    remainder 7

530463       ÷ 16 = 33153,      remainder 15 (F in hex)

33153         ÷ 16 = 2072,        remainder 1

2072           ÷ 16 = 129,          remainder 8

129             ÷ 16 = 8,               remainder 1

8                 ÷ 16 = 0,               remainder 8

Now that our quotient is 0, we stop and collect all the remainders, starting from the last one and moving to the first.

8181F72

Since hex requires an even number of characters, we add a leading zero to that first 8

08 18 1F 72

Now we need to reverse the endianness because that is what we will see traverse the network. We just reverse each hex from last to first.

72 1F 18 08

So, assuming this conversion is correct, I expected to see this byte sequence traversing the network in the GTPDOOR traffic, as this is the “magic byte sequence” that the malware uses to authenticate commands.

And I did! Note that this byte sequence was exactly what the Suricata rule fired on in the first place, so it was no surprise to find it.

Remember that what I  was trying to do in this stage was manually verify and refine a theory to decode the payload. Once proven, I aimed to automate this process so it could occur at scale, using Zeek, which was the perfect tool for this task.

I then used CyberChef to try some quick manual decoding. If you haven’t used CyberChef yet, it has a very impressive list of decodes/operations and is very handy for situations like this where you want to quickly try different operations. 

I selected two operations from the left pane and added to our recipe “From Hex” and “XOR.”

Then I gathered the ingredients for our recipe. 

• XOR key—I tried our computed hex key 72 1F 18 08 
• The encoded payload. For this, I tried the whole UDP stream, by using Wireshark’s “copy as a Hex Stream.”

Pasting these inputs into CyberChef, I saw our now familiar key mid stream (highlighted in orange below), and that even after using the XOR key, the output was still encoded. This was expected as the key was not normally XOR’d onto itself.

There seemed to be a lot of data after the key, so I assumed that the encoded data existed directly after the key. I simply deleted everything prior to and including the key itself.

As the output shows, it was still not decoding to anything meaningful. With my domain knowledge, I knew that padding and headers are often used in computing, so I figured that maybe these elements were still present in our input and messing the XOR decode up. These elements are normally added just prior to the data, so I tried removing some leading characters one by one and to see what shook out. 

Deleting the first hex 02 does nothing, there was still nothing meaningful in the output… Next I deleted the second hex 39–still nothing. Then I deleted the third character 01–and EUREKA I was onto something.

I saw a directory listing, the style of which would be produced by running ‘ls -l’ on any unix based system. I could also see in the directory listing that the content was highly likely related to some form of GTPDOOR testing, so I knew I was on the right track.

Since I had been through a manual decode, I was able to automate the decode using the parameters including those 3 deleted hex values using Zeek. 

The utility of the Zeek script is that you can run it on a wider PCAP or a live network tap and get it to carve out and decode all of the payloads on the fly. This Zeek script can also be generalized so that it will work when the default key is NOT used (GTPDOOR does have a rekeying functionality)  and when the placement of the key and padding and headers vary.

Here is a sample of what the Zeek script found. You’ll note it decoded the ls -l command that I suspected created the directory listing from the manual decode I had already done in CyberChef - this is good validation. There was also “YOU FOUND ME” payloads that were decoded, which is not an unusual style for Black Hat trainers when running training and CTFs. 

This was all a great outcome, and brought the hunt to a satisfying close, proving Corelight’s NDRtech works to detect and hunt modern threats– - even in an encrypted world. 

Even more collaboration!

Once back home in Australia, I was chatting to an old friend from a telecommunications company who put me in contact with the researcher “HaxRob,” the author of the GTPDOOR blog. Chatting to HaxRob helped me to cement my generalizer decode strategy, and also highlighted other similar malware families on which I could use this strategy. As with Altaf, working with a like-minded researcher on these highly technical issues can be a very rewarding experience, and this was no exception.

As always, we look forward to serving at the Black Hat NOC again as they always throw up interesting tales like this one, which we can all learn from.  

I hope you enjoyed another “Blueteamer’s Tale”!

Key takeaways

This investigation was a fantastic reminder of some core principles of network security operations:

• Context is king: An alert by itself is just a data point. In a unique environment like Black Hat, understanding the context is crucial to separate real threats from sanctioned research and training activities.
• Enriched data accelerates investigation: The ability to instantly pivot from an IP address to a hostname via DHCP logs was the key to connecting the activity to a conference talk. Without this enrichment, the investigation would have been much slower.
• Complete visibility finds more than threats: Even though the malware alert was benign, full network visibility allowed us to spot a network policy violation and work with the NOC to maintain the security posture of the conference network.
• An integrated platform is essential: Moving seamlessly from a Suricata alert in our SIEM to the full packet capture using Corelight’s Smart PCAP is what makes rapid, deep investigation possible.
• Security is a human endeavor: Collaboration, community, and even chance encounters can lead to breakthroughs.