Episode 23 - Inside Zeek 9: Modernizing Open Source Network Monitoring & Agentic Security Scanning

Welcome to Corelight Defenders. I'm Richard Bejtlich, strategist and author in residence at Corelight. In each episode, we explore insights from the front lines of NDR, network detection and response. Today, I'm speaking with Christian Kreibich, principal engineer and Zeek technical lead.

Welcome, Christian. Thank you, Richard. Thanks for having me. I'm really glad you're here. The-- Whenever we do anything that talks about Zeek, we haven't done it on the podcast yet, but we have in some other media, people are always very interested, so I'm glad you're here to talk about Zeek.

And, uh, specifically, I'd like to begin by talking about Zeek 9, and I kind of have a two-part question about that. And the first part is, can you discuss how the Zeek project handles releases?

And then after that, can you tell me a little bit about what's in Zeek 9? So, um, in general, the way we structure our releases is that we aim to do three big releases over the course of a year. We don't pin those closely to specific dates, but sort of take some, some liberty to release when we feel like the, the feature set is right. And one of those releases, the .0 release, is our long-term support release. So when we talk about Zeek 9, that is basically 9.0, scheduled to come out at the end of August, and that's our next upcoming long-term support release, which usually matters a lot for folks like, you know, Corelight, because that's the version they prefer to ship. And so accordingly, the current long-term support release that we actively have maintained over the past year is 8.0, and its, you know, patch releases. Over the course of that year, we basically do these two additional, you know, uh, releases that in, in this instance is 8.1 and 8.2, and this is basically sort of, um, more or less like checkpoints along the road to the next long-term support release. So we work on feature sets. We sometimes take some liberties to, to iterate on features if they perhaps didn't land exactly as we had hoped, or because we get user feedback that some, you know, modest change to them would be more useful.

Um, but, but in the greater scheme of things, sort of that's how we operate. So we're pretty excited right now because, you know, whenever the new LTS release comes out, that's sort of a big deal, and we hope we nailed it. If you followed our recent release work, um, a little bit, then you know that we're sort of in this big effort to essentially just make Zeek more modern, is how I would summarize it quickly. So that's, this particularly means, um, adopting more modern ways of doing things with Zeek, so like cluster orchestration, for example. Um, but also just acknowledging that the things we've built over the years are sometimes from an era where simply there wasn't a lot of choice for the things we needed, but nowadays there is. So that's in, in many ways the reason why we've been doing all this work to, um, shift to ZeroMQ for our cluster messaging, so basically how, you know, the nodes in a Zeek cluster talk to each other, and sort of several other things. So this has been sort of an effort for at least, you know, the past three releases or so. And Zeek 9 is really the, the first one where all of this is sort of pretty rounded out and, and on by default. And so, so it's a big release for us. Like, we're really hoping that, that we've nailed it there.

Um, and it's looking pretty good. And the only other thing I would add, and then I'm sort of done with a quick summary of Zeek 9, is just that, um, this release has fallen squarely into that window of folks getting interested in agentic security scanning of open source software. And so we've had various offers. Actually, it's pretty interesting. We're sort of humbled by this, um, to have our code scanned, including, in-including an effort that we're doing ourselves. And, and so that in itself has kept us really busy this time, so which, you know, is just yet another reason why we haven't really prioritized new feature work for this.

So let me go back to the release just for a second. So when you have a .0 release, is that supported for the duration of when that is the LTS and then the... When 9.0 comes out, is 8.0 retired? Is that how that works? When 9.0 comes out, we continue to maintain

8.0 until the next point release. So when 9.1 comes out, we stop maintaining 8.0. So you get about a third of a year of overlap and time to upgrade out.

I see. Mm-hmm. And, and how much of what's in 9.0 w-did you introduce in, say, 8.1 and 8.2? Yeah, a lot. So, so, um, uh, that's, that's essentially a sort of, you know, checkpoint mentality.

So in the past, I think we, we weren't as, um, I'm gonna call it adventurous, sort of in these point releases, but especially for the, for the .1, we sort of try to do a lot of big lifts, maybe the more risky things sort of, because we then have one more point release, the .2, to correct or iron things out.

And then for the 9.0, for the following LTS release, we really are risk-off. We try to not change things too much that really could affect the way the release has held up so far. So we don't want to put at risk sort of the, the feedback that's come in for the .2, for the 8.2 release in this case, right, by making lots of changes. Well, you mentioned the team. Can you talk a little bit about who or, you know, even just the number of people that are working on Zeek at this point?

Yeah. So, so there are two things to keep in mind. So it's an open source project, but all of the core developers that get paid to work on Zeek work at Corelight. We're eight folks, uh, and each of us is in a different city on the planet and almost in a different time zone. But we're all somewhere between the

US West Coast and, uh, Germany. And so basically that just makes, you know, meetings feasible still. I think if we branched out further, that would start to get difficult. Um, but that still works. And then the other side is the community.

Um, and so, uh, you know, because we're an open source project, anybody is, is, is, is welcome to contribute. Um, and we've recently been doing a lot of work on that front to basically be more, uh, inviting, more welcoming, to give more guidance, uh, to potential contributors, to existing contributors, where to go next, and so forth. Hey, can you talk a little bit about how you manage the code? Like, is this something that people can check out of GitHub, or do you have some kind of private repo, or how does that work? Thanks for asking, 'cause I often want to mention this because I think we're sort of unique in that, particularly at Correlate, in that you can basically see everything we do, like all day, with some minor exceptions, sort of like just on GitHub, just like you just mentioned. So if you go to github.com/zeek/zeek, um, that's the main Zeek repository. All of our other ones are, are there as well. And in essence, almost everything there is, is publicly visible.

Um, if you want to contribute to the code, you need to have an account there to send us poll requests and so forth. But the only things, um, since I keep talking about this, like the only things you can't see by default is essentially security issues, because they get submitted to us via our security process, and we work these out with the contributor essentially in private until we have a fix in place. And then there is a process for how we communicate that to our user base, and then that goes out. But other than that, you can basically see everything we do, uh, by default, which is really cool. We're pretty, pretty proud of that. Can you talk a little bit more about, um, what's been going on with the security side of it? 'Cause I, I assume like every other op-- Well, like m-many popular open source projects, you've been inundated by people running various models against the code base and saying, "Hey, look what

I found." So we're not new to this world of having our code scanned, so, so there have been many established techniques for many years that, that, you know, you can use, and, and, and we have. And this ranges from, you know, static code analysis where, you know, there, there is a whole industry that has been building this out for literally decades, where you can basically, you know, like point a tool at a code and it understands the code and understands, you know, the code paths and so forth, and can find vulnerabilities in there. And then there's sort of a, uh, a flip side to it, which is called fuzzing, which is this pretty crazy but super cool idea, um, since computers are fast, of just relying on computers to throw lots and lots of inputs at your program, more or less randomly generated, but with some guardrails to hopefully unearth bugs in your code.

And that is, you know, shockingly effective. Like, it's really cool, and we've been using it heavily, so that's a really interesting way to find, you know, corner cases or outright bugs in, in, in our protocol parsers.

And, and Zeek by definition processes untrusted input off the wire, so, so that stuff is exactly the kind of thing that, that fuzzing is good at sort of un-unearthing. So that's another big source. And then, um, we do various internal sort of standard security practices for code review and, and, and so forth. And so, so, so this, this LLM-enabled, this agentic security scanning is sort of the newest member in that, in that family. And of course, we've been really interested in it. And so there are multiple efforts afoot right now. Um, but the only one I really have hard numbers on right now is essentially our own. So, so via Correlate, we've been able to use GPT with the, with the cyber model to do, you know, security scanning.

Um, and we've basically sort of for the first time, um, uh, been building out our own process for doing this. So we basically came up with a, with a threat model and sort of, you know, an, an, an attack surface, if you will, for where the model should really sort of focus its effort. And for us, that initially is, at least initially, is the, uh, the packet input path, because again, that's where the untrusted input for Zeek comes from by definition.

And then basically go through, you know, the, the core and its analyzers and, and try to find vulnerabilities. Um, and that's been... This is a little hot-off-the-press sort of preliminary, but somewhere between 50 and 70 high-severity findings, which is excellent, um, for multiple reasons. Like first, that's a good number, and it's a good number in two ways. Like, it really found new things, we think. But it's also a good number in the sense that we can totally work with that.

So, so, you know, for comparison, when we enable a new fuzzer, usually that also triggers sort of a flood of issues. Maybe not 70, but, but like a, a few dozen is, is, is not crazy. Um, so it's sort of roughly still in the ballpark of that. Where it now gets interesting is, and like these models tend to be overconfident, so you really need to work them toward giving you reliable results. And so, so, you know, there's, uh, a, a clear requirement to ask for a reproducer for any claimed vulnerability, because otherwise it, it, it might simply not be true.

And then even if there is a reproducer, it actually has to reproduce the issue that this is supposedly about, and so forth. So, so we think that that number will shrink a little more still, but it's a little too early to talk about that.

Um, but what is really exciting and, and, and also humbling is that, I guess because we're an open source project, there are other initiatives that often, you know, have come our way even without us prompting for it. And so I wanna mention a couple of those. So one is, um, uh, OpenAI with Trail of Bits in this Patch the Planet initiative, where we basically get two of their engineers to go to town on the code base for, you know, one to two weeks, and then send us the findings. And this is underway right now. We're actually awaiting the findings sort of every day right now, so this is pretty exciting. And so it's gonna be super interesting to see, you know, how their methodology compares, um, how the findings compare, to which extent they're redundant and so forth. Um, and then there are a few others. So, so, uh, Anthropic has a similar one.

Those guys really just approached us the other day, and we didn't even really know about it. Uh, there are some Correlate customers who've offered resources for this, which is just fantastic.

And so, so we literally just had a conversation in the team whether we should be, you know, mindful of people's, you know, money and resources since we're already looking at this as well, to maybe sort of stagger these efforts a little bit so that they're not all going to town on the exact same code base right now, because that increases the odds of finding redundant results, uh, or, or, or, you know, um, uh, issues. There's been a lot of work in recent years to migrate and f- You know, correct me if I don't get the terminology right, but I think it was migrating away from Binpack and more towards Spicy, right? Both from a security standpoint, but also just in terms of trying to make it more accessible for people to write protocol analyzers. Do you think SPICE has helped with keeping some of the number down in terms of vulnerabilities that might've been found? Essentially, we know of no actual vulnerability in any SPICE-generated code so far, but we certainly know of vulnerabilities in Binpack-generated code. But I have to be specific there, because with Binpack, uh, which is sort of like a, like a very early version of what the SPICE was trying to do proper- properly then, um, um, several years later.

Um, with Binpack, you, you got fewer guarantees about, you know, the, the security of the generated code from the get-go and, and more of a, more of a, a harness to, to sort of, you know, hook up your own parsing code. And so a, a typical Binpack parser has a lot more handwritten C++ code in it still than you will find in SPICE code.

And, and so for that reason, there is just inherently more, you know, potential there to go wrong. And we've seen several findings where you go like, "Oh, ah, this is in Binpack, so let, let's see. Is it really the Binpack-generated code or more like the, the greater, you know, amount of manually written code that is still hooked into Binpack code?"

Mm. And it's the latter, we think. So, so Binpack, a- a- all in all, already did a pretty good job at this. It's just that SPICE was much, much more sort of methodical and, and sort of structured in how it goes about this. So from a security perspective, I think SPICE has been nailing it. I think several other aspects there are still TBD, like to the extent to which it really gets, you know, newcomers to write a parser, the speed of the parser in, in, in production, and so forth. I think that's all stuff where we're working on things still. So whenever I hear people talking about, um, C or C++ and s- code security, you know what I'm gonna say, right?

Uh, Rust is what often is mentioned next. Uh, I think the first time I ever even heard of Rust might have been with respect to Suricata, 'cause I think they... You know, initially there was like a Suricata dash, dash RS. project, like that was their Rust implementation, whatever. Has, has the use of Rust ever been something y- you've considered with, with Zeek, or are... Is there... You know, when I' think of Zeek, I think of C++ and SPICE and all that, but, um, are, are there other languages even involved? I think

Suricata made exactly the right choice for, you know, their project's needs with Rust. I think that was a good one. I think it was exciting to watch from a distance a little bit.

Um, and I think there are, there are many things there to keep in mind. So I think, um, historically in, in, in the Zeek project and in its sort of, you know, academic heritage, we've often tried to build the exact right tool for the job sort of as well as we possibly can, rather than going off the shelf. And, and, and so this is another instance, I think, where at the time, all this work that led to SPICE came about, um, like Rust wasn't really the thing that you would have just taken. But perhaps more importantly, like Rust is giving you the security guarantee, but we would argue you don't really want to write a parser in Rust. Like if you look at the SPICE code that you use to express the, the, the syntax and semantics of a protocol, I, I would argue, maybe this is subjective sort of, but, but, but we would argue that that just looks way nicer than Rust code.

Once you, you know, are, are knee-deep in Rust, then truly, like, you know, maybe you no longer think that way sort of. But I, I would actually argue that the, the SPICE code is more maintainable at that level than a bunch of Rust code. But the flip side to this is that SPICE is this completely custom-built thing, so it needs to mature, it needs to, it needs to be performance-tuned, all of these things sort of, and you don't have that aspect in Rust. Perhaps what's worth mentioning is w- we have almost like at, at least monthly, almost weekly discussions about where else in the project we could use Rust. So we're super interested at, uh, in it.

Um, but so basically in, in specific corners of the system, we're absolutely looking at, at, at Rust, just because it's such a, such a go-to today. Can you talk a little bit about the use of Zeek in Windows, which some of our listeners may be aware of, and probably a lot are not aware of it. Several years back, Microsoft announced that they were going to ship Zeek as a part of Defender for, for Endpoints to basically gain this ability to look at the traffic going by on the wire. Um, and that collaboration continues to this day. Um, what's interesting is, so in this, in this first version, this was basically all new, and we worked with them, uh, on their, on their change set that they were, um, willing to contribute to Zeek to make this, you know, support easier, which was super cool. This was during the Zeek 5 era, just to be specific.

And so, so that collaboration then gained a lot of momen- mo- momentum again about a year ago, where they basically started to consider sort of bigger upgrades to the Zeek version that Defender is running with. And, um, and there's all kinds of stuff there that I could talk about, but, but in essence, just like, for example, Corelight. has also figured out it is much better to think about such support, not in terms of like periodic big lifts to new versions, but as a continuous thing, where you just sort of always keep an eye on what works and what doesn't and what's new and what needs support. And, and essentially the technical solution for that is continuous integration. And, and so very recently, Microsoft have sent us a bunch of, you know, patches and so forth to build out our test support on Windows.

And so we're looking just as closely at the Windows results in our CI pipelines now as we do for Linux or BSD. Well, let's, um, let's finish by just maybe looking ahead a little bit. Can you tell me, what y'all are working on, uh, for, I guess, it would be Zeek, 9.1,

9.2, and, then, you know, eventually the 10 series? So on, on Linux, we're introducing just a completely Different way of maintaining or, or orchestrating, I should say, Zeek clusters, which is, just much more aligned with how things work on Linux these days, which is system

D. So, so, that might be sort of an, an, an abstract concept for, for, for many listeners, but it's basically the way things run on' modern

Linux distros, and it gives you a ton of control about, you know, what proce- processes are allowed to do and whatnot. Um, really cool control about, you know, bringing up a complex system and so forth. And, and it just, um, is our, you know, envisioned path forward to essentially get away from

Zeek Control, which is this, this, this much older but very powerful, you know, tool that we've had for a long- time, uh, to, to maintain a cluster. Folks who look closely at our documentation may have seen our, things like our UDP packet source. So this is this notion that in the cloud it makes a lot of sense to consume packets differently.

Um, and, and this notion of sniffing on an interface is, is not necessarily sort of the, the, the, the best suited technical approach to this. And, and so with this packet source, you can basically just sort of run Zeek as a packet sync. Like you can just send it packets over UDP and then we'll natively process Geneve and so forth right there. And we have some fun conversations about like where else you could take this, but sort of another, you know, a- another checkbox sort of in the modernization department. And then there's a lot of other stuff that we would like to do with protocols. At the end of the day, you know, protocols are, are Zeek's bread and butter. Um, there's been a lot of thinking about encapsulations, so tunnels historically speaking, you know, are, are, are everywhere now. Zeek is arguably not as flexible as it should be for handling this. So, so, you know, we've received some feedback that, for example, any tunneled flow, any Geneve encapsulated flow should be strictly separate in Zeek from, you know, anything that comes out somewhere else. So we need to be better at, at making that a thing. And so we're also finally returning to a packaging ecosystem, which has been sort of sitting there for a while, essentially, um, as is. So we've not made many changes to it, but, but the bits and pieces around it have moved a lot, and so we have some really nice ideas, I think, for how all of that can be much simpler today. So, so ZKG, the package manager itself, but also the notion of a, of a native code plugin in a package, which has been really complex in the past, we can do much more simply right now. So that's, that's all stuff that's gonna keep us busy for the next sort of several releases. So I think un- until 10, the road is pretty clear right now, and then beyond that it's sort of opening up pretty widely. So we'll see what, what time brings, but it's, it's fun. There is really a lot going on right now, so it's, it's a good time to look at Zeek. That sounds great. Sounds like there's no shortage of cool things that you all are working on. I appreciate you taking some time to, out of your, all that development work to talk to us today at the, uh, Corelight

Podcast. Awesome. Thank you so much, Richard. Great fun. Thank you for joining us on the Network Defenders Podcast, sponsored by Corelight. We will see you on the network.

You've been listening to Corelight Defenders. To stay informed with expert intelligence on today's cybersecurity challenges, please subscribe to ensure you never miss an episode. We'll see you on the network.