Zeek on Windows

Editor's note: This post was originally published on the Zeek.org blog on Nov. 28, 2022. Reposted here in full with permission as a courtesy.

As we shared at ZeekWeek 2022 in October, we’re thrilled to announce emerging support for Zeek on Windows, thanks to an open-source contribution from Microsoft. Part of its integration of Zeek into its Defender for Endpoint security platform, this contribution provides fully-native build support for Windows platforms and opens up a range of future technical possibilities in this vast ecosystem. Make sure to check out Microsoft’s talks on the technical aspects of this integration as well as the detection capabilities this move enables.

In this blog post we’d like to recap the open-source work that’s unfolded since the beginning of our collaboration with Microsoft, summarize the current status, and outline next steps.

The road to Windows

Initiated by an emerging partnership between Corelight and Microsoft, the first concrete conversations about Zeek running natively on Windows and its potential for Defender happened early in the year. In mid-September the Zeek team, led by Tim Wojtulewicz, began to work closely with the Microsoft engineering team on actual code, and to map out a path to getting the contribution integrated into Zeek’s main line of development.

The first step on this path was to review an initial version of Microsoft’s contribution, based on their internally deployed Zeek build, to understand potential trouble spots, focus areas, and the extent to which changes would affect our source tree’s already complex submodule structure. To align the process with our regular GitHub-based workflows, this set of initial PRs and their review happened in private repositories that we forked from our public ones in the last week of September. This also gave our team an opportunity to set up local builds and deploy a Windows CI environment.

As a next step, the Microsoft team turned the needed submodule modifications into PRs (for bifcl, binpac, broker, cmake, gen-zam and paraglob) that we merged shortly thereafter.

The MIT-licensed libunistd provides a lot of the port’s underlying Windows/UNIX compatibility. In line with many of Zeek’s third-party dependencies, we added an internal fork of the library as an additional submodule to allow us to pin our build to specific commits, and we’re upstreaming any modifications we make to it. Many of the hiccups we identified at this stage were due to subtle discrepancies in this compatibility layer, such as one version adding a trailing slash while the other does not, or timestamps suddenly turning negative because of 32-bit/64-bit confusion. However, Zeek’s codebase has remained encouragingly free of platform-specific ifdefs and similar ugliness!

Finally, Microsoft’s main Zeek PR landed on October 31, consisted of 72 commits, and got merged 11 days later.

Current status and next steps

At this point we provide experimental support for the Windows platform, as follows:

• The Zeek build is functional and supports the libpcap API, as provided e.g. by npcap. Our testing has focused on pcap file processing, and we don’t yet have much practical experience with real-time packet capture.
• Zeek log output works as expected in our test environments.
• We provide full support for builds on Visual Studio 2019, with a Docker configuration available for use on Cirrus and other systems.
• All auxiliary tooling needed for builds (binpac, broker, bifcl, etc) is functional.
• Most unit tests function correctly, and the ones that don’t are accordingly documented.

However:

• The build does not yet support Spicy, though support is already in the works.
• We haven’t yet tackled add-on plugins, or Zeek’s package manager.
• We cannot currently run Zeek’s system-level testsuites, powered by btest. Much of the difficulty here stems from the fact that the btest toolchain is very “UNIX-y” in nature, with tests and helper scripts frequently implemented as shell scripts invoking typical UNIX tools such as grep, sed, and awk. Reliance on Python’s multiprocessing module further complicates the situation. We’re looking into ways to remedy this situation.
• Zeek’s performance on Windows isn’t well-tested and documented.

The Zeek 5.2 release, scheduled for early 2023, will be the first to feature Windows support. In the meantime, we encourage folks interested to check out the documentation, experiment, and share feedback in Slack and our Discourse forum.

The Zeek Project would like to thank the team at Microsoft and particularly Tomer Lev and Elad Solomon for this contribution, the late hours, and the fun and lively collaboration — it’s been a thrill and joy to work with you all.

By Tim Wojtulewicz