Episode 26 - Zeek at 30: The Measure-First Mindset That Shaped Zeek (Part 2)

Welcome to Corelight Defenders. I'm Richard Bejtlich, strategist and author in residence at Corelight. In each episode, we explore insights from the front lines of NDR, network detection and response. This is part two of my interview with Vern Paxson, chief scientist and co-founder at Corelight.

We've been discussing Zeek's 30th anniversary. I just asked Vern about the tendency for software and hardware vendors to not build visibility into their products.

First, it's under-taught, just the power of measurement and, and the like. And I'm not sure why that is, but it's probably because it always takes extra effort.

It's kind of like, you know, why didn't you build it securely? Well, it would've taken more time and cost more, you know? And, and that's my best guess,

'cause it, it-- You're right, it's absolutely underserved and an afterthought, just like security was and often still is even today.

Um, and it's, it's such a pity because, um, it, it's super powerful. In my travels, I've come across people who have either a configuration mindset or a data mindset, and the configuration mindset pops up when someone says, "I have a, I have a problem. I can't get this to work." And the configuration mindset says, "Well, do you have the following configured? Tell me what your configuration is." And the data mindset says, "Tell me what's happening."

You know, so if I, if I have an issue, I'll start, like, uh, you know, collecting traffic or whatever it is. Yes. In, in, in fact, I mean, it, it's reflective in my whole research, um, trajectory, which is I'm an empiricist fundamentally.

It's like, uh, sure, I care what should be happening, but I don't believe it till I see that it's happening. And that got cemented in my early years, and still does all the way up through today, which is often what is actually happening is not what's you think is supposed to happen. And, and I love that moment of, like, aha or surprise or puzzle or whatever, that, that mismatch between a mental model and, and the reality. And, and you only get it by measuring. Um, as we were talking, I was reflecting on how, you know, I, I mentioned earlier the group I was in did BPF, Libpcap, TCP Dump. Those are all measurement tools.

So, you know, um, it, it's kind of striking that they had to be developed sort of post facto, but they did.

And Van had trained as a physicist, experimental I'm pretty sure. Um, and so I think he, through physics, uh, where they take data very seriously if they're on the experimental side, not, not necessarily the theorists, um, really values measurement. That-that's sort of the q- that's what experimental physics is. Um, so that, that-- It could be that, that that's the trajectory. Coming out of physics, this guy does this measurement stuff. It's like, geez, why can't you measure these things?

Okay, let's do it, and let's do it efficiently. And then that leads to the follow-on of the appreciation of the data and better measurement tools. Hmm.

You know, my opinion is that no two networks are alike. Like- ... every network is a snowflake. Uh, do, do you have any, any thoughts on that? Oh, totally.

Um, I-- You, you know, you know that, that, uh, aphorism, you know, a person who has a watch knows what time it is, a person who has two watches is never sure.

Right. Um, it, it's, it's exactly that. And this was so eye-opening in my early work, which is, um, I would measure the same phenomena, and, you know, you can measure it really in, in depth and come up with all these sort of models of it and, and then you'd measure it somewhere else in a different context where you thought it would be kind of the same, and it was just totally different. Hmm. And, and so that goes to this, um, uh, the, you know, the spark I have for measurement, which is I, I love it when I learn my mental model's wrong. But it's also humbling.

And it meant you eventually-- So I eventually, um, distilled this into a, a pithy way of, of capturing it, whic-which was for a lot of internet characterization work I would do, which is there is no such thing as typical. Mm-hmm.

And the importance of it was not-- Was we want to build typical in our heads really badly. It's, it's how we abstract things. And once you learn there's no such thing as it, then you're like, oh my goodness, what do I do about that fact?

You know? And, and it's so easy to just ignore it and get your one thing and, um, not appreciate. Anyway, uh, you're absolutely right. They're, they're all different, and it is surprising how different they can be about just the simplest things even. Are you surprised that, uh, what you learned about, say, TCP/IP and Ethernet back in the '90s is what ate the world and became, you know, the, the dominant protocols?

Onl- No, no. I, I was initially, right? I was like, um, it-- I distinctly remember Van saying to me at one point, "This internet thing, it's gonna be big." And I remember thinking, no, it's not.

And, and then, then I started measuring it, and then, um, and I have, uh, I have this, um, one of my very early papers, Growth Trends in Wide-Area TCP

Conversations, something like that, and it spanned the rise of Port 80 traffic.

Hmm. And I had these data points at LBL, and while I only had, like, four, um, across a couple years, it, it was-

It's doubling every six weeks. And, and it was just like, it was right there in the data. It was like there was no 80, and then there was some 80, and then there was quite a bit of 80, and then there was a huge amount of 80, and then there was even more

80. And it was just like, wow. Um, so that was really the first taste of, okay, this is totally gonna really matter. Um, too bad I didn't know more about, okay, how do you invest if you have that perspective?

Yeah. If we think about when you were developing the software for the first time, there was another extremely important project that was happening, uh, you know, Linux, and I think a lot of people tend to sort of conflate the idea of open source with Linux-

Yeah ... and the whole development of that ecosystem. When did you sort of become aware of that, and did it have any effect on the development of Zeek?

Yeah, um, so un- unfortunately it was actually a negative. So I was doing my dissertation work analyzing TCP behavior, and the Linux TCPs were horrible.

Mm-hmm. They, they just had such pathological behavior. It's like, how does anyone get this thing to at all communicate? Hmm. And, and so I just knew that from analyzing, and it was like, well, geez, BSD totally nails this, and System V is pretty good at it, and

Linux sucks. And so I had a not good sort of, you know, um, view of the Linux kernel and networking and the like. In addition, uh, something else I should have framed is, is that the network research group was sort of in the heart of open source already, was BSD was open source.

Yeah. And, uh, Linux struck me as sort of a Johnny-come-lately to the whole open source world, and, uh, it, and of course had huge success at it, and like Snort sort of found the right sweet spot that, um, the more sophisticated, at the time, uh, approaches never really figured out. So it, it, it's pretty fascinating in retrospect, but, uh, it wasn't very influential on me at, at the time. Did you always think of Zeek as an open source project, or was there a point at which you thought, "I need to start organizing this into a project that potentially could outlast me"? I mean, I always thought of it as open source. That was the culture, and so of course it was gonna be that.

Um, the trajectory of so many of these open source projects was modest, and so I, I had no thinking that this thing needs to actually be, you know, industrial strength and self-sustaining, as I... None of that was in the picture until, for about a decade at least.

And then we, um, started thinking about it as sort of a community resource and the like, and part of that was driven by just needing to fund things. So we would, we would need to make pitches to do funding so we could work on it, and, um, a pitch that resonated a fair amount was, "Hey, this is a great resource for research R&D, for the educational and, uh, research, um, labs and the like. It's, it's, it's a really valuable resource."

And, and that's when we first started thinking about we need to it needs to actually be u- usable, you know? It needs documentation that's actually clear. And, and, and over time other things evolved like, hey, we need, you know, notions of stable releases and stuff like that. Speaking of releases, um,

I think everyone... We know, well, well, we know the latest release of Zeek, uh, patched a bunch of security issues that were discovered through, uh, various uses of, of certain AI models. Do you have any, I guess, sort of two-part question. One would be, what's your sense of, of that, uh, development?

And then secondly, your own use of AI has been, uh, a topic within the company, uh, as you've gone through this journey. Uh, my boss, Greg Bell, quite often has said, "You'll never believe what Vern said about whatever AI." So can you talk a little bit about that, the AI and the Zeek project, and then your own use of AI? Sure, yeah, yeah.

Um, for the Zeek project, it's not that different than, hey, there's a new fuzzer and it's really good. And, and so we've gone through some of this in the past when, um, the folks in the project who deal with our, um, code scanning and the like would do a significant upgrade of, of, in particular the fuzzer. "Hey, it should know how to generate this, this type of protocols, packets, and so forth," and then we'd suddenly get a whole bunch of bugs.

Um, the current batch from the sort of AI discovered is, uh, larger than that, but it's not super larger. It's, it's kind of a drag in that, um, there's a s- subset of the folks reporting these who are basically, as far as I can tell, just in it to be able to say they've found a thing. You know, they, they don't actually care specifically about the project-

Yeah ... and they don't put much care into does this one actually matter. And so the, the volume has been large, and there's a lot of work the project's having to do to deduplicate things and to figure out what, which ones really matter, and then to figure out... A, a whole lot of them are, uh, state-holding attacks.

And, um, I mean, that's been the bugaboo of Zeek from the beginning. I've always been aware of, hey, you can make a whole lot of state and then you're, you're gonna be screwed. It's gonna crash.

Yeah. And it's had that problem for 30 years, I would say. Um, and, and a lot of them are that, and it, it, it has been useful in thinking about, um, how do we- In general, have a better approach to state management and the like, um, 'cause these, these attacks will be used down the road. Yeah.

You know, I tried Claude Code late last year to do some development work, and was unimpressed. It just wasn't that good.

And then I returned to it around March after it'd been gone through some updates, thanks to the fast pace of innovation, and, and now it was quite good. And I started using it, and I tried, um, a few things.

I tried making, "Hey, I've got a, a change I wanna make to Zeek," and I would do it with Claude, and it just took care of all the grunt work really.

And, um, uh, and, and that was quite good. I tried vibe coding a thing I wanted in-- written in a language I don't know, and, uh, it made a hash of it eventually. It, it kept mushing it towards working, and then I could realize we got to the point where every time we fixed a thing, something else would break and, you know, it, it had no good architectural sense. And then I've done more ambitious things with it since, and where I'm currently at is it's certainly a tool I wouldn't, uh, wanna stop using. It would-- that would be, um, frustrating.

And my guess is on my current project, which is large, it's probably around 50,000 lines of code now, um, it's been a four to five X accelerator.

Wow. So, so that's terrific. But I've also found that you still really do need to know what you're doing. Yeah. And w- as I'm watching it, you know, and it makes decisions and so forth, like 80% of the time they're, they're fine, they're good. And, uh, but I would say another, you know, 15% of the time I'm like, "Wait, what are you doing? Why are you doing it that way?" You know? Or, "Don't do that, that'll paint us in a corner," or whatever, and, uh, have to back things out. And then maybe five I have a, "What are you doing? That's wrong."

And then I-- it explains it and I'm like, "Oh, it's right. I'm wrong." Oh. It, it's kind of weird. I, I feel like you have to watch it like a hawk, and, and so I'm still figuring out good workflows for that where essentially

I'm interrupted every two or three minutes by it, "Hey, I'm ready to build," or, "Hey, I got a question," whatever. How do I make use of that time, those, those little snippets, um, effectively?

I know I have colleagues who are just like going from Claude to Claude to Claude on different projects. Mm. Uh, I haven't yet tried that. Yeah.

So I don't do anywhere near as much on the code side because it's, you know, I'm not a developer. But anytime it's recommending something and I get that little voice inside that says, "Uh, I don't quite know about that," and I'll ask like, "Is there any downside to what you're, you're proposing?" Or, "Is there an alternative?" Or, "What could be the negative effects?" Or, "How could I, uh, you know, back out of this?" And it'll say, "Oh well, I'll break your system if this doesn't work-" "... or I'll screw up your Python environment," or- Yes ... and I'll say, "Okay, that's what I was thinking of. Is there a way to do this without screwing up my Python environment?" "Oh yeah, do, do this," da da da. Yes. That, that part is amazing, where sometimes it's like, uh, "Oh, you're right. That's totally busted. Uh, why don't I do it this other way?" Yeah. Especially if it- Like, "Dude" ... it devotes a lot of time to a certain approach and i-if, you know, if you're lucky, along the way you'll realize, "I think we're going down a wrong path. We need to get out of this-" Yes

... um, before, you know... And it will, it'll, it just, it's like it's committed to it. It will keep going until you say, "Nah, let's get away from that." Absolutely does that in coding as well, and I have found sort of two key things which usually I have to do, it doesn't think about it, which is, um, could we add trace information that will help us diagnose this? That's one.

Mm. And sometimes I'll even ask it to write a program to analyze the trace information. Mm. And the second is, let's step back and think about the abstractions, you know? Maybe we should have this abstraction to deal with this whole nexus of stuff, and, and that, it, it, it'll appreciate those, you know? And for me, it's not very sycophantic, but often it'll like the abstractions and, and I can see why, and then it'll go off and do 'em. And yeah, the problems just, um, are much easier to solve. But it's not gonna come up with that. It's gonna do what, what you were framing. It gets fixated.

Yeah. Must do the immediate task. Yeah. Well, let's turn back to Zeek for a second. I'm wondering, in the many years that the project's been around, are there any, uh, high points where you're, where you think, "Wow, I can't believe we did that," or, "I knew Zeek was used for this great use," or, or any sort of success stories like that? So the National Science

Foundation put a lot of money into making Zeek be a community resource. And in fact, you know, we would not be here at Corelight and talking about it like this had they not done that. And, uh, you know, it was around, um, more than $6 million, and, and that enabled creating team and, and documentation and so forth. In their, I think it was the FY17 request written in 2016 to Congress for their budget, they highlighted eight achievements of the National Science Foundation in 2016, o-one being like gravity waves, detecting gravity waves, and another being what, what's called at the time BRO.

Defending supercomputers was how they put it- Wow ... even though it was really broader than that. Yeah. And so, th- I mean, that, that was pretty lovely. I'm not quite sure, I still wonder whether they really felt it was one of their top eight, or they felt, "That one goes up the food chain really readily. Let's just use it."

That's cool I've never asked Well, I know it was a core component of Security Onion when Security Onion- Yes ... was first released. Yes. And so millions of people are using it now because it's in Security Onion.

Yep. Uh, and before that, uh, we deployed it at General Electric, and we even got to the point where I think we technically hired Seth for maybe a week before he got- Yes ... the job with you all. Um, so we found tons of stuff at, at GE, so thank you for that.

Appreciate that. Um, I guess my last question, 'cause we're, um, covered so much ground, there's probably tons of more stuff we could talk about, but I'm just wondering if you have advice for anyone who's maybe in your shoes, uh, now, right?

They're, they're looking to maybe work on something around instrumentation, or they are looking for a research project. Do you have any advice for, for, uh, new project creators?

I can give the advice I've generally given to my PhD students and other grad students. My entire career path has been driven by, what am I passionate about?

And, and I find that for most researchers, passion, you really wanna listen to that. Th- that, that is what gets you to, "I really want this thing," and gets you through all the headaches of attaining that thing. Um, so I, I've been a big proponent of that, you know. That you need to just listen to that, and, and you need to listen when it's not there, and try to understand what that means.

And I don't mean it as a sort of simple signpost. I just mean it as a voice to consider. And, uh, in particular, my passion voice, for example, said I would never wanna be a professor, because in part I didn't really know what being a professor was like, and I didn't know that I would actually love teaching, which, um, I only discovered when I had to do it. Um, similarly, my, my passion voice was, "I'm not gonna be in a company." I mean,

I like the freedom of academia, and it's because I didn't realize how fascinating being in a company could be. So, so it, it's tricky in that regard, but it's still, it's a great barometer, and I love the energy that it, uh, unleashes. That- that's really what's, what's so great about it.

I think it's interesting you mentioned what is there and what isn't there. And also you gave, you gave it a shot, right? You tried these other things and found out y- you like them, and maybe there's other things you've tried and you didn't, and, you know, hey, that's a great way to try it. If you, if you, uh, if you don't give it a shot, oh, you know, you'll never know,

I suppose. Y- y- here's the humbling thing, though. I only gave them shots when I was forced to. Oh. It wasn't like I'm like, "You know, I should double-check that view." N- no, I was more or less backed into it.

So- ... uh, uh, it, it makes me appreciate the difficulties of all of us sort of navigating our way. Yeah.

Well, Vern, uh, thank you so much for this really insightful conversation. Congratulations on 30 years of Zeek. And boy, if we were to get 30 more, I can't even imagine what the project-

... will be like at that time. That blows my mind. It's been really fun. Thanks, Richard. Thank you for joining us on the Network Defenders. podcast, sponsored by Corelight. We will see you on the network.

You've been listening to Corelight. Defenders. To stay informed with expert intelligence on today's cybersecurity challenges, please subscribe to ensure you never miss an episode. We'll see you on the network.