Episode 5 - Detecting DNS Covert Channels in the Wild (Part 1)

Welcome to Corelight Defenders. I'm Richard Bejtlich, strategist and author in residence at Corelight. In each episode, we explore insights from the front lines of NDR, network detection and response. And today, I am speaking with Vern Paxson, our co-founder and chief scientist at Corelight. Welcome, Vern. Happy to be here. This will be fun. The topic we decided to speak about involves covert channels. How do you think about covert channels? What are we talking about? Yeah. So, um, a covert channel is a way for, uh, an adversary, well, to communicate. So, there's a system that they have access to, and then, s- let's say the external world, and they want to communicate between them. And what makes it covert is, looking at the traffic, it's not apparent that the communication is happening. So, uh, there are a whole lot of ways of doing this. For example, in web requests, there are headers that the web browser sends to the web server and, and in the other direction that describe all sorts of stuff. What's the language settings and what's the time zone? And, you know, zillions of these. The attacker, if they control the browser and the server, could add headers that have some sort of benign-looking name and some sort of encoded information that they want to communicate. The server knows where to look for these and how to decipher whatever's put in the header.

Any network monitoring or analysis of server logs or whatever will just see some header and say, "Darn it if I know what that is." There's so many different ways of doing this. It... Most of them are sort of catnip to certain types of academics who just wanna bat out yet another incremental form of, uh, cleverness. So, they think, hey, there's a covert channel you can do with TCP options or with size of SSH, randomness, or whatever it is. There- there's many, many of these. Most of them kind of miss the point that if the attacker controls both the client and the server, there's more straightforward ways they can communicate.

Uh, like just put it all inside encrypted TLS session and, uh, be done with it. Often, it's viewed as a theoretical interest. However, there's some forms that actually really matter. These arise when you've got a locked-down environment, like an enterprise that, um, really tries to control communication between its internal systems and the internet and is able to sort of scrub most of the traffic that goes out. Uh, for example, they might use web proxies so that all the HTTP headers that I talked about earlier in, in the web connections, uh, they just strip

'em out because they're gonna regenerate that connection from scratch. Uh, so in, in that context, covert channels can matter a lot. And, uh, the one that I wanna talk about is, uh, used in practice and, and can really matter, which is using the DNS name lookups that at enterprises, there's millions of these a day. In a lookup, a internal system will ask an, an, a, a intermediary system that's also owned by the enterprise, "Please look up this name," like, you know, www.goodguys.com. And that resolver or the internal system will in turn generate a query for it, uh, get back the reply, which is often like an IP address, and forward that along to the internal system.

That's a, a, a very limited form of communication that the enterprise allows and motivates attackers to figure out how to bury inside those names the communication they want. These matter to the intruder when there's a belief that the intruder could be caught by someone. If, if no one is watching, if no one is collecting any evidence, then the, the need to use a covert channel reduces drastically. So, we're com- It completely goes away. ... we're talking about environments where someone actually is paying attention, right? Exactly. Exactly. So, the case where there's no monitoring is very easy for the attacker. Um, the case where this matters is the enterprise that's been, um, subverted by this attacker really runs a tight ship. And generally, that's because they've got lots of important stuff to secure.

I'm glad that you addressed the issue of breaking the direct connection from the client and the server if both sides are controlled by the intruder. There's a lot of work that's been done over the years of, like you said, using these weird TCP options, or flipping a bit, or using some par- unused part of a header. And if you don't have clear end-to-end communication between those two systems, if there's any type of middle box, it's gonna do any type of rewriting. Even if it's simple NAT, that's just gonna go out the window usually. Exactly. So, a lot of those just get solved by having, you know... In some ways, it's a pain that, well, there are so many middle boxes. That's not kind of what the internet was set up to be. But at the same time, it does break some of those, those covert channels. But let's talk about the DNS stuff now. So, I think lots of people could imagine how that could work.

You ask something very simple. You ask to resolve a certain domain. You could get something back, and that could mean something. But can you tell me more about how this might work and, and what research you did looked at? Yeah, sure. Um, and, and I like how you put it about, you know, the... Essentially, the, uh, the question is how many degrees of freedom does the attacker have?

And in a enterprise where there are all these middle boxes that greatly reduces the attacker's degrees of freedom, in DNS, what they have is, uh, the name they look up and when they look it up. And, and there's one other detail, which is there's a type. Like, you could say, "Hey, I want..."... IP address or

I want alias for it, whatever. And for our purposes, let's just consider that part of the name, th- because it doesn't change any of the overall thinking. First, the situation we went after, which is a common one, is the attacker controls a remote name server that is, let's say, kittens.com, and lookups to kittens.com will come to that server and the attacker can see them there. The internal system that wants to, uh, and generally we're gonna care about exfiltration in this context, wants to send a bunch of data out to the attacker, will do lookups to kittens.com.

And I chose that name because the point is the name itself of, that's being, of which the subdomains are being looked up seems benign. And so, the enterprise can't a priori go, "Uh, you know, I, I've got a problem internally.

Someone is looking up kittens.com," 'cause they, they don't know that it's, it's malicious. Uh, the attacker can do quite a few things. So, the most basic thing would be to look up, you know, the first, uh, uh, a name that is Base64 encoding, you know, so some sort of ASCII readable thing of the first 100 bytes, that would be this big, long line of goop.kittens.com, and then the next 100 bytes, big, long series of goop.kittens.com. And, and just keep looking these up, and, um, as the attacker observes these coming in, they're able to, uh, just line them all up, assemble them into the overall, uh, item that they want to exfiltrate. There's other ways that these lookups can be used which are a little more subtle. Basically, there, there are two additional, uh, degrees of freedom.

One of them is you can look up names repeatedly and actually convey some information. And, um, I'll give a simple example. You might pick eight colors, uh, orange, blue, green, et cetera, and which color you look up determines three bits, because there's, out of a, uh, out of eight possibilities, there's three bits that describes which of the eight you chose. And so, you can keep looking those up over and over and, uh, and leak out information that way. That's a lower capacity channel. You don't use all of the potential information that you're sending, but it's, uh, trickier to understand that something sneaky might be going on, because the repeated lookups just seem, like, harmless, you know? So, so the thing ought to cache, and it doesn't, and keeps looking up the same name. Finally, you can use the timing of the lookups to kittens.com. So, you could just look up over and over www.kittens.com, which seems so benign.

Um, but the, uh, communication can be resolved in terms of, well, what was the spacing between those, say, in tens of millis- seconds? And, um, if it was between one millisecond and a thousand, well, that's essentially 10 bits of information if, if the attacker's able to resolve the timing that precisely. So, the- those are three different ways of using the channel. How does having all your DNS requests go to a local name server that does caching and all that, how does that affect the situation? Yeah. So, for the, um, the, the first case where you're encoding information directly, typically caching won't matter 'cause you're not gonna look up the same thing multiple times. For the cases where you're repeating the sort of code book style, uh, uh, or the timing one, um, DNS allows, uh, replies to have a cacheability essentially, a, a cache lifetime. And so, the, um, attacker's, uh, name server could just say,

"Hey, this one's not cacheable," and then, uh, it won't be cached. And, uh, the requests will still go out even though they're redundant. That makes me wonder, could that be something you look for, is when every response. that comes back says, "Don't cache this." Uh, at the same time, I'm wondering- Yeah, exactly.

So, so So, what- ... does a CDN do that or... I don't know. L- let, let me, let me, then, um, move into how we went after this, 'cause this was sort of an epic undertaking between, uh, my students at UC Berkeley, uh, folks on my research team at the International Cons- Computer Science Institute in Berkeley, and, uh, folks at IBM Research. And, uh, we spent about two years on this problem. And, um, and came up with a nice solution that I'll get to in a bit. But, but so, so, the starting point is, hey, these all sound kinda weird. You know, you're gonna have really big names being looked up that kinda look like gobbledygook, or, you're gonna have names looked up that have, uh, no cacheability. So, just look for that. And, um, our approach was we had access to tons of data, which is, the style of research that I really, uh, thrive on and love. And what I love about having tons of data, and, and here I'm talking enterprise data, is the world is often way more complex and messier than you might expect, and I love being surprised even when the surprise is, darn it, the easy thing doesn't work. The first thing we did was analyze, hey, it, it... surely, you don't see lookups for, like, hundred character long names very often at all.

And it' turns out, no, you see, like, you know, 50,000 a day. So, you're like, hmm, that's gonna be tough to, to do that. Or surely, you don't see uncacheable things. No, that happens all the time. And, and, so, those natural signals when you understand how the communication might be happening and wanna go after its quirks, the quirks occur for benign reasons a great deal, and, uh, do not give you a solid basis to, to proceed. In addition, it really... a lot of this problem boils down to, you know, what's the attacker... how patient is the attacker?

Hmm. Because, um, they might not look up-... you know, 100 byte names over and over to convey 100 bytes at a time. They might look up five-byte names and convey five bytes at a time. And if they do that, now, you no longer have any signal for,

"Hey, the name is really long." You might have, "The names are diverse." There's quite a few different ones, and we looked at that, and that's also common. The... And, and that's really the, the key to the problem, is realizing you cannot solve it perfectly. And the thought experiment for that, that I like is, each day maybe they look up www.kittens.com, maybe they don't. If they do, it's a one bit. If they don't, it's a zero bit. Mm-hmm. Um, and there's no way you're ever going to figure out, "Hey, they're very slowly leaking out, you know, a, a bit a day in that fashion." You, you cannot win that. When you've got that realization, then you can actually start thinking about, "Okay, what interesting subset of this problem can we solve?" The one we went after was, okay, you've got a bound and you wanna confine the attacker, to the... Have an upper bound on how much communic- how much information they can exfiltrate, say, every day. And you'll just accept that if they're below the bound, they win, um, but if they're above it, you'll find them. And, uh, the bound we used, uh, for our evaluations was either four kilobytes a day or 10kilobytes a day. Those both seem really quite good, you know?

If you've got a serious exfil concern, often the thing being exfiltrated is megabytes and, and the attacker is not going to be able to persist for years to exfiltrate that. Um, so they're going to have to go above these, these limits. Is this, is this making sense how I'm sketching it? Yes. Can you think of something valuable that could be stolen in a very small amount? I'm thinking maybe credentials? Exactly. Yes. Aside from that,

I don't know. Crypto goop, um, of, of various sorts is, is certainly a biggie there. And, and you can imagine other things like, um, the, the malware figures out, "Hey, the company is gonna go public soon. You should..."

You know, some small amount of information that can be leveraged into a large external gain. Okay. Um, but... An- and, and we're not solving that problem. We wish we could. We, we don't solve it.

But we do solve it when it's, "Hey, it's the plans for the fighter plane or whatever," you know, something big that, uh- Yeah. ... when one thinks about exfil, that's often what's being exfiltrated, is, you know, a database of something. Or even just sustained command and control while you're... Where you're issuing lots of commands and getting a lot of responses, and you're trying to figure out how to move around in an enterprise. If that's happening over obfuscated channel like this-

Yes. ... that, that could trigger your, your threshold. Yeah. As soon as, like, your... What you need to exfiltrate is a, is, you know, screenshots or just a, "Here's a list of all the systems," or whatever it is, you'll, you'll pretty rapidly use up that budget. That is all for part one of my interview with Vern Paxson on covert channels. In part two, Vern will describe the system he and his fellow researchers used in production networks to find DNS-based covert channels. Thank you for joining us on the Network Defenders podcast sponsored by Corelight. We will see you on the network. You've been listening to Corelight Defenders. To stay informed with expert intelligence on today's cybersecurity challenges, please subscribe to ensure you never miss an episode. We'll see you on the network.