Episode 25 - Zeek at 30: Vern Paxson on Origins and Enduring Logs (Part 1)

Welcome to Corelight Defenders. I'm Richard Bejtlich, strategist and author in residence at Corelight. In each episode, we explore insights from the front lines of NDR, network detection and response. Today, I'm speaking with Vern Paxson, Corelight co-founder and chief scientist. Welcome back, Vern.

Happy to be back. I'm glad you're back. We have a, a really interesting topic today. The Zeek project is 30 years old this year, and well, I guess there's some dates you're gonna mention that, uh, we can talk, uh, about that involve slightly earlier dates, slightly later dates.

Yeah. So this is as good a, a year as any, I suppose, to call, call it for 30. But I guess just in general, can you take me back to the 1990s and explain where this all started? Yeah, sure thing. And, and I do feel this is the right 30th, and I'll get to that in a minute.

So, uh, in the '90s, um, I was at Lawrence Berkeley National Lab. I trained there mostly as a software engineer and then was pursuing-- wanted to pursue graduate studies.

The lab had, and I hope they still have, this fantastic program, work study for PhD, where you would actually be paid a real wage and be doing lab work, but they would cover your attendance in a PhD program, and some of your work could be focused on that. That's the only reason I ever got a PhD, 'cause I could not abide the financial hardship that would've-- that most PhD students have to endure.

Uh, I was in the, the, the program, and the question arose, "What am I going to actually do my research on?"

And, uh, I was part of the network research group at LBL, um, which was led by Van Jacobson, one of the luminaries of the internet at that time, and the principal designer of TCP's congestion control, among other things.

Um, and the group produced Libpcap, tcpdump, BPF, of all the stuff that has endured to-today. Um, and so Van's idea was, "Hey, you should be measuring the internet." And, um, this seemed actually kind of boring to me, uh, because this is around 1991, '92, and it wasn't clear the internet was all that interesting, and I didn't really like network protocols. I liked building software. And, uh, but I had to do it, 'cause I had no, no other, you know, no other way to do my

PhD. So I did it, and it turned out to be fascinating. And, um, you just felt like discovering a, a new land no one had ever been to, 'cause no one was really measuring the internet in earnest. As I was doing that,

I was developing software to look at the network and piece together TCP health and stuff like that. This was covering part of my time, but another part of my time, I mean, the lab doesn't just pay you only to do your PhD research, uh, needed to be funded.

There was a happy sort of conjunction in around 1994, where

I was doing my measurements and the NetOps people who, 'cause there was no s-security team at the time, would come to me now and then and say, "Hey, in your measurements, did you happen to see this activity that we're interested in?"

And often I would, and that led to me, um, sort of on being funded by them in part to set up ongoing tcpdump monitoring and nightly, uh, batch jobs that would crunch on everything and try to find things and so forth. And that started running in 1994.

As it was, uh, doing this, and we would find attacks and, and, and they started to interest me. I mean, they sort of offended me, you know. I was one of these wide-eyed, uh, wonder of the internet guys by that time, and, uh, it, it was irksome that people were misusing the network towards their own ends. Geez. And, uh, so that kind of fueled a bunch of energy on my part. And these scripts, while, while they were handy, it was frustrating that it was only in the morning when you got the nightly report in your email to you that you realized, oh, yesterday there was, you know, some badness.

And I started thinking about how can you find it as it happens. And because I was, uh, rooted in software engineering and because, um, in an earlier life, I'd built an event-based system for, uh, uh, actually particle accelerator control, that had exposed me to how do you architect things for, uh, flexibility,

'cause the previous-- the predecessor thing had a, a little domain-specific language, and for high performance, which, uh, Van was terrific at. He just knew all these angles, how do you really get things to move fast.

Um, and thus, uh, I got the idea of let's build a system that's dedicated to this network monitoring, and it has the flexibility of the domain-specific language, and it has a bunch of, you know, careful filtering and algorithms and so forth, so it could go fast.

'Cause we had to monitor a 10-megabit Ethernet and, and- ... that was sort of as good as it got at the time. And then that went to 100-megabit FTDI. Wow, that's just really pushing. And, um, so in 1995, I started working on it, and in June of '96, uh, it became operational. And, and so that's why I, I feel like this really is the 30th anniversary, because there wasn't really a Zeek until that time.

Wow. Um, and it has run there ever since. And the, uh, totally amazing thing is they have the logs still today, and I poke through them now and then all the way back to 1986. And not only that, they have them back to

1994. So the ones produced by my, uh, scripts are still in their archives, and I can behold them. That's really amazing, and I would venture to guess there's nothing l-else like that.

Yes. Yes. It, it is, it is a incredible, uh, resource for, um, the, the longevity of those logs, it just blows my mind. Yeah. I, I almost wonder if you could only do that in an academic setting. Like, there seems to be so much pressure...

Well, first of all, if you're a commercial entity, you may not have existed longer than five, 10 years- Yeah ... and you just went out of business. Yeah, yeah. Uh, or you could have sort of like a classic "Doctor Who" situation where in the '60s, they just rewrote over the old, uh, video-

Yes. ... you know, to save to, to, to just record the next episode. Yeah. Yeah. Uh, but yeah, there's nothing like that, I would say. Yeah, I, I think it's even more singular. I mean, I think because you had to have money to, to afford the disks and stuff, and so in academia, I don't, I don't think that would've survived that long. And at Lawrence Berkeley

Lab, because there was actual mission to protect, and there was-- they took, you know, security, uh, was, was a funded thing and so forth. So I think that's part of it. And I think another part is, uh, the lab, and I imagine most national labs have this very data-oriented...

You know, they, they just live and breathe data, so the, you know, the notion of, like, losing data is like, "Ew. Let's not do that."

Yeah. Well, that, that's a testament just to their ability to manage that for so many years. That's- Yeah. Yeah, yeah. And they have gone back in time quite a few years. So there, there was one incident came up, and, uh, I think it was the inspector general's like, "Hey, do you have this thing from five, six, seven years ago?" And, and they went and, and looked for it. Mm-hmm. I believe they found it. Can you talk a little bit about the value of that neutral collection as opposed to the other... I would, uh, see, I wouldn't even say your approach was dominant. I think it's taken a while for people to appreciate your approach because I think the dominant approach back then, uh, and dating back to, you know, the '70s when there was academic research around this, was around intrusion detection. It was-

A-absolutely ... uh, generate the perfect alert. You know, it reminds me of, uh, Will Ferrell and the perfect cheer, you know? That's what they're trying to get, is the perfect alert, 'cause if we can get the perfect alert, that's what we're trying to do.

So that's when you saw, you know, uh, not to throw any shade on, on projects like Snort or whatever, but that was it. Like, that was what people were really concentrating on. Uh, and yet- Oh, absolutely

... I think your approach has, uh, for especially dealing with sophisticated intruders who are testing against those IDs's, your approach is more likely to catch them or get some evidence. Can you just talk a little bit about that idea? Yeah, yeah. But I can't resist first riffing on, on Snort 'cause that was indeed the thing that was, quote, you know, the rival. It, it took a long time to realize they're just different tools, and you want them both. And but that nobody got that at the time, and, uh, and I think in part 'cause Zeek's model was not a, not the expected one. And I've reflected a bunch on this. I mean, Snort had this fantastic trajectory in terms of uptake and, uh, and commercial success, and then, you know, it's hugely influential all the way through today. I, I had plenty of time to reflect on why isn't Zeek doing that, you know? And, um, and, and, and eventually I settled on, you know, it, it's the Snort's underlying model is conceptually pretty simple. It, you know, it's just not that hard to know what a Snort rule does and, you know, and, and the like. And so that's one. It, it was the barrier to entry was low. But, but another that was huge is that, um, people were happy to share signatures, and, and there was a, a network effect of, you know, the more people sharing, the more protection you got. And, uh, and it wasn't like that with Zeek, where, you know, you had your, your scripts for your site, and a lot of what was built into that was specific to the site and, and not, um, easily abstractable with. Um, so I think that that's why the trajectories were so different, and it took a long time, of course, for Zeek's model to, uh, be understood and appreciated and liked.

Um, but going back to that model, um, it, it's interesting to me to reflect on the original paper. I, I didn't appreciate the benefits of that as much as I should have at the time. So the original paper emphasizes real-time detection.

Like, "Hey, you can run this, these algorithms that have a bunch of state about what recently happened." So a more powerful model than, than Snort. But the goal, the primary goal, was find the thing.

And, um, and I wish I had realized at the time, no, actually what's hugely valuable is the record of what happened. And sure, it's nice to find stuff in real time.

That's great. But for, for the more sophisticated stuff, you're not gonna find it in real time. You, you didn't think about it even, and then you're gonna want that record of what happened. It's interesting that, you know, Marcus Ranum, who went on to be very influential in firewalls, had his company Network

Flight Recorder right around that time, and, and that has more... And, and it was quite similar to Zeek in structure. It had a domain-specific language and everything. And, and just in the name, you can see that he was already thinking about the power of that, that data, that just sort of neutral, detailed, and as accurate as you can make it description of what happened. Yeah. It's a very data-heavy approach, and, and that's already,

I think, something that makes it harder for people to grapple with 'cause now you gotta squirrel away tons of data that, that Snort doesn't produce, you know. You know, in some ways, its success mirrors the rise of big data and the appreciation of the power of, of data and the like.

I'm very glad I came up with that model, and I think coming up with it was rooted in part in my PhD research, where, you know, you gotta record everything 'cause you're gonna analyze it later.

Um, and, and in that lab culture of, "Hey, we're running an, an experiment. We're capturing data and so forth," for sure is, you know, sort of the bedrock that made Zeek, um, what it is today. Do you recall how many protocols you were dealing with or, you know, parsing so people could have logs?

And was there file extraction in the beginning? So yeah, it's in the paper, and I think it's like five, something like that, FTP and- Uh, Telnet and SSH and not HTP, which seemed just too scary, complicated, and so forth.

Mm. Um, that was done by a brilliant intern who for the first month I was like, "Is this gonna get anything at all?" And then he just was like, "Oh, well by the way, here's my HTP parser."

Oh, file extraction. Yeah. No, that wasn't even, uh, a, a conceptual notion for me. It- I suspect Seth Hall, um, brought that notion, but it, it might have been earlier, but it, it wasn't me. It, it, it makes total sense in retrospect.

It's part of the record of what happened, but it wasn't on my radar. Are you aware of the, the intruder's view of early Zeek? 'Cause I do have a story if you're not aware of, of what people thought of it. Oh, I'd love to hear it. Okay.

So, uh, I got to the AFSERT in the fall of '98, and, you know, we were using our NSM tool and all that, and someone told me a story 'cause they were plugged into the underground, and they said, "There are three places that intruders are reluctant to attack. Uh, the first one is the NSA because- ... they have their own," I forget what it was called, Docker or... They had some kind of monitoring tool that they used to, to watch everything. The second one was- They also wouldn't mind striking back.

Yeah, right. Exactly. They were the only ones who could, right. Uh, the second one was the AFSERT because we were catching people all the time with NSM. Uh, and the third one was, uh, Lawrence Berkeley National Lab because of Bro.

And- Wow. Yeah. Isn't that funny? And, uh- That is great ... at the time it didn't really make an impact on me 'cause I was so new, I didn't really understand, and, and I didn't even really become aware of, of how to use... Well, I don't even know if I used Bro until about 2003, 2004-

Mm ... and that was mostly due to Seth, and I think I went to a workshop and, and it took a while before I was able to really understand what this thing could do for me. And even I think in my first book,

I had to get a guest author who used some third-party scripts that he had written to, to make it easier to install. And, uh, it took a little bit, a little bit of time before I was able to incorporate it into my own workflow.

And once I did, I, I, I was teaching at the time and I completely scrapped the old version of my course and flipped around and I wrote everything around, uh, parsing data with, with, you know, then Bro, 'cause it just made s- things so much more interesting. But yeah, that, that intruder view is something you don't hear too often, but I never forgot it 'cause those are the three that, uh, that came to mind. That is, uh, that tickles me no end. By the way, I just looked, and my earliest message from you in my archives is 2003,

December 25th. You were working on Christmas, dude. Uh- ... error compiling the release 0.8A58 release. Okay.

Yeah, that's when I was writing that first book, 'cause it came out in 2004, so I was probably trying to get it working. 'Cause everything in the book is, you know, things that I've tried and show people-

Yep ... how to use. Yep. Uh, yeah, okay. That's cool. Yeah. So, you know, it starts out as this, this project to support your research, to support your PhD.

When does it sort of take that next step and become a, a, a tool that maybe others are using or starting to contribute to and, and ch- make that first change?

P- people were playing with it shortly after the paper came out, because I would get requests for it. And I think in the paper,

I g- actually don't remember anymore, but I'm pretty sure it was like, "Hey, it's available, but you, you should ask." And so people would come and ask, and so I, I knew they were playing with it.

Um, when did it start getting seriously used? Um, that, that's a great question. I'd have to research that 'cause

I don't have a, like, event in my head of, oh wow, you know, it's being used by, you know, this big company that I never thought, you know, never had on my radar. So, so I don't know. Do you recall who the first people were who started helping you with it? You mentioned, uh, an intern working on the HTTP profiler, but... I was joint at Lawrence Berkeley Lab and then the International Computer Science Institute in Berkeley.

And we had a, at, at the latter, ICSI, we had a robust internship program. And so I would get interns from, uh, other places to work. And so, so the... One of them was Robin, a co-founder.

Uh, one of them was, uh, Christian Kreibich, who's also at, uh, Corelight. Uh, Christian I think was the first one working on, on Zeek around

2000. And then, uh, Rumeng Pang was the one who did HEP, was from Princeton. So, so a bunch of it was, was through that. Um, the, I,

I, I would also have to spelunk through my archives to figure out who, who was contributing a lot in the early days external to that. I'm, I'm quite curious. I'll, I'll go look that up. Just going back to the original development environment, were you building this on Solaris, I'm guessing, maybe?

No. Um, uh, LBL has long been a freeBSD shop when they could. Okay. And so I'm guessing that's what it was. It was definitely not Solaris, which I, I worked with in my software engineering, uh, life and, you know, it, that was not such a fun experience. So actually we're talking probably original BSD, not even free, right? 'Cause

I think free... Yeah. Does that sound right? Oh, I don't know. I can't keep the l- I mean, uh, the Network Research Group was so deep into BSD that, um, I, I, a lot of that would blur for me. Okay. It was probably actually

SunOS- Okay ... um, a fair amount for development. I don't know what the working machines were. Oh, boy. A lot of it's coming back now. We got a equipment grant. In fact, there might not have been a Zeek had we not gotten this grant from good old Digital Equipment Corporation. You'll remember them, I'm sure.

Yeah. Long gone. They had the Western Research Lab. Like, Jeff Mogul was there. Jeff arranged a grant of hardware, so it would've been whatever the, the hardware was that they were making at the time. That was the iron that stuff ran on. It was, quote, "high performance," could handle the scary FTDI and so forth. Yeah. And the development was probably on SunOS.

Do you recall how you guys got access to the traffic? 'Cause I'm assuming at 10 megabit, it could've been a 10 megabit hub And so it's reflecting traffic to all ports.

100 megabit, now you're talking, uh, you know, I think, uh, did they have SPAN ports back then? I'm guessing that they did. No, it was FTDI ring, and so, so, so for the

10, I think it was just, it was a broadcast ethernet. Okay. It was as simple as that. Yeah. And, and for the 100, it was a token ring, and so the monitor was part of the token ring, which mattered, 'cause it meant if its interface went down, you lost the whole thing.

Oh, okay. And since that wasn't so, uh, appealing, that, but that was, that was the way to get it. And that was this, uh, equipment that DEC provided. Okay, and so if you didn't request the token, the traffic would pass by, and you'd just copy it, and as if you weren't there. Is that how that worked? Uh, that sounds like how it must have worked. It's so long ago that... I mean, it, the, the, the nature of token ring, it, it's gotta go through your NIC. Yeah. That's the only way it gets around, and so yeah, I think you just grab it when it goes through the NIC.

Okay. Do you have any thoughts on sort of the lack of visibility that's built into enterprise gear? Like, I- visibility to me has never been a priority-one feature for any networking equipment. Like, it's not like you build extra capacity into a switch to give it a SPAN port so that it can copy traffic so people can do troubleshooting. First, it's undertaught, just the power of measurement and, and the like. And I- not sure why that is, but it's probably because it always takes extra effort.

It's kind of like, you know, why didn't you build it securely? Well, it would've taken more time and cost more, you know? And, and that's my best guess, 'cause it, it, you're right, it's absolutely underserved and an afterthought, just like security was and often still is even today. Um, and it's, it's such a pity because, um, it, it's super powerful. That concludes part one. Tune in for the next episode, where we finish our discussion on Zeek's 30th anniversary. Thank you for joining us on the Network Defenders. podcast, sponsored by Corelight.

We will see you on the network. You've been listening to Corelight. Defenders. To stay informed with expert intelligence on today's cybersecurity challenges, please subscribe to ensure you never miss an episode.

We'll see you on the network.