Precarious exposure of cookies when QUIC rage quits

Threat Hunting in the Black Hat NOC is like “finding needles in a needle stack,” said Mark Overholser as we all stared at this massive ice cream that James Pope ordered.

It was Day 2 of setting up for Black Hat Asia 2026, two days before the first day of the event. Things were going very smoothly with the NOC setup, almost too smoothly. But I have that effect on the machines, they sense my presence and act on their best behavior. This was my first go-round with threat hunting at Black Hat Asia. That is sort of like getting a Formula 1 car and race team for your first car to get around town. As we were all sitting there digesting our lunch, I asked the group if they had any tips for threat hunting at Black Hat. Pope chimed in with a comment that stuck with me, “Look for the most common things, and the least common things.” It was my first conversation with Pope for any extended period of time, so this stuck with me.

Ok who is this new Ben, not to be confused with the other Ben (Reardon)? Why are the Corelight Sensors, Dell arrays, and other odds and ends afraid of him? Thought you’d never ask! I am a Principal Engineer in Corelight Engineering. I am a glorified plumber and hoarder of bits. It turns out that these bits actually have some semantic meaning outside of green test cases that enable merging of new software. I really enjoy debugging and diving deep into issues that impact sensor performance and stability. I participated in a number of Corelight CTF events and scored well. So, to close the loop, that’s why Mark thought that I had the potential to be at least a mediocre threat hunter. That, and nobody wanted to travel all the way to Singapore…

After lunch, packets were flowing thanks to our partner Arista. I picked the checklist that Mark had for me and worked through it. It went very well, with the exception of a cabling error involving the Dell array for Smart PCAP storage, which is a common mistake. Mistake corrected. The last step was to import Ben Reardon’s Zeek Bundle, which applied perfectly on the first pass. Pleased with myself, I made a comment, like, “This Black Hat thing does not seem that hard,” to which Mark responded with a grin and just said, “The sensors don’t want to upset you.”

Monday was another setup day. We had a few loose ends to tie up. Eldon vibe-coded a new Zeek Plugin that exported extracted files for malware analysis. We left early because we were done and dusted.

Tuesday morning was my first NOC shift as a bona fide threat hunter, doing real threat hunting work. I kept asking myself, “How do I do the threat hunting?” Tuesday AM was a little slow; there were not many attendees yet. I took a long lunch and went back to the NOC to see what was going on with the afternoon shift. It was a fantastic learning opportunity that I didn’t want to miss. I found some space at our table and opened the laptop. Then I remembered what Pope said about looking at the “most common” and “least common” events.

Ok, nothing too interesting here. Everything I expected for the most common log types.

However, there could still be something to look at from the least common log types.

The SMTP was nothing interesting, just someone using plaintext SMTP to send an email in 2026. generic_icmp_tunnels was a dead end. The http2.log looked a bit more interesting. This was cleartext HTTP/2, also known as h2c, and seeing this was odd because, while the HTTP/2 spec permits cleartext, h2c, browsers only implement HTTP/2 over TLS in practice, so you almost never see it in the wild.

This is interesting: Someone on the general Wi-Fi was hitting an authentication endpoint in plain text. The destination service appeared to be load balanced behind at least two IP addresses. This was pretty “normal”. However, it was definitely not normal to make HTTP/2 requests in plain text to an authentication endpoint over the public internet. Also note that this plain text request happened over port 443, which typically contains encrypted traffic.

Next, I tried to pretty-print the URI’s query params:

This appeared to be an iPhone 16.

Next, I looked at the client_headers. This is a pretty-print of ‘client_headers’ and we have a number of IDs and an ‘sgcookie’ header. These definitely look like secrets.

Something else that stood out to me was that http2 thought the transport was encrypted, when it clearly was not. Note the scheme below is https, not http.

The next question I asked was: “Why were there only three of these logs?” I was sure that there were more users of this App/Service at the show. I had to do some digging in the DNS logs and see if we could see other clients reaching out to the same destination. Indeed, there were many—too many to list here. A total of 57 unique clients reached out to this domain. You can see the queries below.

A quick Google search showed that the top query was for a link shortener service (i.e., tracking domain), so we could ignore that one. But with this information I was able to answer my initial question. There were, in fact, other users at the conference. This opened the next investigative path: Why were there only the three HTTP/2 plaintext requests and two associated DNS queries in a sea of 3,000 other client logs?

Taking another look at the conn.log, I filtered for the source IP and id.resp_p=443 and decided to build a Sankey of organizations based on asn data. Each organization was assigned a random number for privacy. There were 1,500 QUIC and 7,500 SSL logs. The HTTP/2 logs were included as well, but they were so small that they didn’t render.

If we zoom in on the organization that answered the HTTP/2 requests, we can see the HTTP/2 logs showing as just a sliver.

Below is another Sankey diagram, filtered by the ASN hosting the target service. The random numbers on the right represent unique IP addresses. Highlighted are the HTTP/2 requests going to two different IP addresses. Two conns to one IP, and one to a different IP address. This is common with load-balanced services.

We still didn’t know why the three HTTP/2 requests were sent in the clear when everything else was not. My first thought was that this may have been a downgrade mechanism that failed open when it should have failed closed. After some digging, a pattern emerged. This confirmed our hypothesis that an error path in the client SDK for this service was failing open when it should have been failing closed.

At the end of the show, we checked the HTTP/2 logs again and found 71 instances of this issue on 16 clients. We are working on notifying the proper parties through the official channels.

Not bad for a first-time “at-least mediocre” threat hunter! Thanks, Mark (and the other hunters), for the invite. It was an honor and a privilege to go hunting with so many talented folks!