Tales from the Black Hat NOC (USA 2024) - speed to detect SSHAMBLE

Another big day in the NOC

Welcome to the Mandalay Bay Convention Center in Las Vegas. I've finished another morning shift in the Black Hat NOC, where the Corelight team has been working with the other NOC partners as part of our ongoing commitment to protect the Black Hat network. With over 20,000 conference participants, there is always something interesting going on in the NOC, whether it's "Black Hat legitimate" activity from training rooms, attendees who join the wifi with pre-compromised machines, poor configurations or actual nefarious activity. The NOC must monitor and react to it all, to ensure a stable and safe network for every attendee and exhibitor.

The NOC is a very immersive environment. Everyone in it is fully engaged and it's not easy to switch off. Normally, if there is enough space on the couches or spare chairs, team members can stay around in the NOC after the formal shifts end.

At 3 p.m. on the day in question, the NOC is already getting quite crowded, so I change speed and decide to check out a Black Hat briefing session or two.

HD's talk on SSHAMBLE

Checking the schedule, I notice that the renowned researcher "HD Moore" (of Metasploit fame) is giving a briefing. HD is the founder and CEO of a company called "RunZero", which actively and passively scans networks to identify various devices and configurations. His subject is a new SSH scanner called SSHAMBLE, which he's going to release during the briefing. This piques my interest since I released an SSH fingerprinting tool called HASSH years ago while I was with the Salesforce Detection team. The fact that HD is dropping the tool in his talk makes it even more interesting: a brand new detection! Challenge accepted I think, and I make it to the briefing just in time.

At 4 p.m., HD and his RunZero colleague Rob King are halfway through their briefing, and I'm pretty sure that my HASSH fingerprinting methodology will be able to detect this tool on the network.

I've also envisioned a series of simple steps I'd take to test my theory. While having one ear on HD's presentation, I open my laptop to begin the process … and see the battery is close to empty, having been working hard in the NOC all day. Anndddd, what do you know, the battery dies right there as I'm staring at the indicator.

I decide to go to my room to get my power brick. And so I make my way back, past the casino floor and onto the Mandalay tower lifts — a well-trodden path during a busy week at the conference. When the lift doors open on my floor, strangely enough, a couple of RunZero salespeople (and a polar bear-like mascot?) are standing in the foyer, seemingly acting as floor guides for visiting guests.

Being on the blue team, I'm very much on the lookout for red team deceptions and social engineering, particularly at security conferences. However, I quickly decide these sales folk are who I think they are, and as we chat I tell them I've just come from HD's talk. They're currently hosting a mixer right here, right on my floor, and say I'm welcome to drop in, as HD will be there soon. They mentioned the mixer will wind up around 6:30 p.m. and as it's almost 5, I set myself the challenge of completing my detection and rolling out coverage at Black Hat. With any luck, I can finish before I meet up with HD so we can discuss it right there.

Scanning time!

I walk briskly to my room down one of the three arms of the Mandalay Towers and get straight to work on the detection idea, sharing the logic with the other Corelighter NOC crew in case I've missed something. My laptop is powered and back in business, and because I've already thought through all the steps on the walk to my room it doesn't take long to check out HD's just-released repo and spin up a lab environment where I can run the scanning tool to collect the PCAPs.

The following screenshot shows the tool being run from Kali, against a secured SSH server:

How HASSH works

Here we need to step back a little and discuss how HASSH works. In the briefest terms, HASSH is an SSH fingerprinting tool. It creates an MD5 hash of a set of four crucial elements that relate to encryption and authentication protocols, which are exchanged as part of initiation/negotiation between clients and servers. These elements necessarily occur in-the-clear, before the session becomes encrypted. The premise of the fingerprinting technique is that different SSH clients and servers often select a unique set of these elements. This should then make clients and servers identifiable at a lower level than the well-known "client" and "server" strings, which can easily be faked (spoiler alert - as we are about to see!).

HASSH is available on our Open Source GitHub repo, and is also available for use on all Corelight sensors. You can read more about the details of HASSH here: https://github.com/corelight/hassh.

SSHAMBLE's fingerprint

Now that I have the PCAP, I simply run the HASSH Zeek script over the PCAP and then go straight to the ssh.log file, which - if my theory is correct - should contain the HASSH fingerprint that would make for a very simple detection element.

Sure enough, a quick check of ssh.log shows a unique HASSH fingerprint of c6c8b23b1c966dad1173df11c4e4f431

In Logscale SIEM, our detection logic for SSHAMBLE is simply:

The reason this simple detection logic works is because the SSHAMBLE scanner (written in Go) organizes the SSH initiation packets in a unique way, which in turn results in a unique HASSH fingerprint of c6c8b23b1c966dad1173df11c4e4f431.

Notably, the tool also masquerades as OpenSSH (No Golang) using a faux client string of SSH-2.0-OpenSSH_9.8p1

Here is how the logs look in our Investigator platform, which is what we use in the NOC:

Creating a detection for Black Hat network and for our Investigator customers

A search of Black Hat networks later reveals that no one has attempted to run the tool from our network yet, which is good news. This simple search is also added to our hunting techniques, so we will know if it is used in the final day of Briefings tomorrow. We also add it to our Investigator "Saved searches" to give our customers the benefit of the detection.

Wait, is this IP address HD's Internet scanner ???

I complete a search of Corelight's "Polaris" systems, which contain extremely large volumes of real-world data from participating partner sites. In this context, this data can be thought of as a series of internet telescopes. I check Polaris to find out if there are any obvious false positives that would make this detection method less accurate.

Using the simple detection logic described above, I find a single IP that has been scanning three totally separate Polaris sites. The IP is the only one that has this HASSH fingerprint, which bodes well for the detection method. This smoking gun, I suspect, is HD's internet scanner, the results of which he shared in his talk. But we need to check…

HD's mixer

It's now 6 pm, and with my detection challenge complete, I head to the RunZero mixer, where there's a good crowd in attendance.

I seek out HD, introduce myself, and we have a great chat about his new scanner tool. HD is very open and friendly. I tell him I have a detection method for his scanner and that I'd seen one single IP scanning at least three distinct Polaris sites only days before, and that it was the only SSH client that had this fingerprint. I give him the IP and he later confirms that the IP was indeed the one that they were scanning with: Now that is ground truth!

The next day, my new friend HD visits us in the NOC:

They say it's sometimes the small things that make for a great time. As a blue-teamer, it doesn't get much better than detecting a tool that someone like HD Moore just released at Black Hat USA — with a fingerprinting technique that you released years earlier.

We are looking forward to repeating this scenario in future Black Hat NOC engagements, including the upcoming Black Hat Europe conference in London. Be sure to reach out to any of our NOC crew if you attend!

Key Takeaways

• Detection can be built out very speedily in response to evolving threats. It's not always complicated, particularly when you have quality metadata available, which is what Zeek provides.
• The power in using existing logs to find new threats. Looking back in history we can discover any prior activity. The metadata required for the detection is simply the HASSH field in Corelight's Zeek logs, which we were logging all along in the NOC. We'd already enabled the "HASSH" package on the sensor via the customer GUI, as we do at every show. This is very powerful as it allows us to look back in time in existing logs and find new threats with old logs.
• The utility of network metadata, even in an encrypted world. There is significant value to mine from real world, large-scale data (e.g., from our Polaris systems) when building detections.