Writing a Zeek package in TypeScript with ZeekJS

Zeek^® is the world’s most widely used network security monitoring platform and is the foundation for Corelight network evidence. In this blog I share how to write a Zeek package in TypeScript with a new capability called ZeekJS that was released as part of Zeek 6.0. Packages to enable security teams to extend Zeek’s network security monitoring functionality through policy scripts, and that helps analysts, engineers and operators achieve a variety of goals—from new detections to changing the way Zeek outputs logs.

This blog is also a companion to Zeek Blog’s post introducing ZeekJS (JavaScript support for Zeek) and covers some of the details that may not be obvious to other Zeek users who are not also already experienced JavaScript (and TypeScript) hackers.

Scratching my HTTPS itch

I have been using Telegram Messenger for some years to chat with family and friends (much like WhatsApp or iMessage). What I love about it is that it always had great multi-device (including Linux desktop) client support and an easy to use bot API.

Like many of my colleagues at Corelight, I run a Corelight@Home sensor sensor on my home network, so I was thrilled to discover that my Corelight Labs colleague Yacin Nadji implemented a Zeek Notice Telegram package and wrote an article about it. Soon after installing this on my sensor I noticed strange failures with the ActiveHTTP function call this package uses to reach out to the Telegram API.

ActiveHTTP is a wrapper around the curl utility. It is often described as a hack and it is no accident that the Zeek documentation complains about ActiveHTTP in the ZeekJS section. In my particular case, I found that the curl sub-process was returning to Zeek with a status code that made no sense (“Unsupported Protocol”) and this issue was not reproducible with curl outside of ActiveHTTP.

With the recent release of Zeek 6.0, ZeekJS (which adds JavaScript support in Zeek) is now a built-in plugin. While ZeekJS is still experimental (meaning the API is subject to change), I figured now is a good time to kick the tires by re-writing the telegram notice package in JavaScript.

While the official documentation provides an overview of the API (including examples) I didn’t have a good background in using JavaScript specifically with Node.js (the JavaScript runtime that ZeekJS embeds into Zeek).

In this article we will walk through my implementation and then explore using TypeScript to add type checking as well as writing unit tests with btest and ZeekJS.

Interlude: why JavaScript?

Zeek already has a built-in domain-specific scripting language designed specifically for the task of analyzing network traffic. It is a fair question to ask: why also support JavaScript?

According to the 2023 Stack Overflow Developer Survey, it has been the most commonly used programming language for eleven years in a row.

The survey shows that JavaScript is also the most desired language (by measuring the proportion of respondents who want to use it). There are other languages that are more admired (the proportion of respondents who use it, and would like to keep doing so) but JavaScript still beats out traditional heavyweights like Java, C and C++ on this measure, and TypeScript comes third (after Rust and Elixir, and well ahead of Javascript itself).

Now consider the library and tooling ecosystem:

• JavaScript engines like Google’s V8 have been optimized to the hilt with JIT compilers so they are fast;
• Runtimes that embed JavaScript engines (like Node.js) provide a rich API and standard library for network, database and file interaction, etc.;
• The number of JavaScript libraries is enormous (the npm registry has over two million packages), with options from image processing to interacting with Kafka and everything in between;
• There is a proliferation of source-to-source compilers for languages that target JavaScript (notably: adding functionality like TypeScript, or implementing a completely different programming style like the functional Elm language);
• WebAssembly is breaking new ground with a portable instruction set that allows existing compilers for languages like C, C++, Go and Rust to target all the major JavaScript engines.

ZeekJS enables the Zeek project to access both the pool of programmer talent and the vast capabilities of the JavaScript ecosystem. I can’t predict what this means for the next 20 years of Zeek, but I do have a short term need: reliable outgoing HTTPS requests.

Implementation details

My objective was to maintain the API of Yacin’s original Zeek implementation. This is relatively simple, the package requires the user to:

• Redefine two configuration variables that identify the destination Telegram user or chat group, and a Telegram bot API token;
• Use Zeek’s Notice framework policy hook and (based on some logic) choose when to add the Notice::ACTION_TELEGRAM enum value to the actions vector in the Notice::Info record.

Here is an example of how you might (in Zeek script) use the package to be notified (via Telegram!) when Zeek detects a successful SSH login:

@load base/frameworks/notice

export {
    redef enum Notice::Type += {
        Notice::SSHLoginSuccessful,
    };

    // NOTE: you must define these!
    redef Notice::telegram_token = "xxx";
    redef Notice::telegram_chat_id = "yyy";
}

event ssh_auth_successful(c: connection, auth_method_none: bool)
    {
    NOTICE([$note=Notice::SSHLoginSuccessful, $conn=c, $msg="Login detected"]);
    }

hook Notice::policy(n: Notice::Info)
    {
    if ( n$note == Notice::SSHLoginSuccessful )
        {
        add n$actions[Notice::ACTION_TELEGRAM];
        }
    }

Now let’s quickly run through the JavaScript implementation (for readability, I have linked to the source code on Github rather than reproduce it all here).

The mechanism by which we test notices for the presence of the Notice::ACTION_TELEGRAM action is via another Notice framework hook (now in JavaScript):

zeek.hook('Notice::notice', { priority: 0 }, function (n) {
    if (!n.actions.includes('Notice::ACTION_TELEGRAM')) {
        return true;
    }
    sendTelegramNotice(n);
    return true;
});

While working on this I discovered that Yacin’s Telegram package took inspiration from zeek-notice-slack by Pierre Gaulon who has since written his own ZeekJS version of that package (zeekjs-notice-slack). I was able to get some advice from Pierre on the Zeek slack and used one of his packages’ functions in my code.

The sendTelegramNotice() function is 38 lines but most of it is setup and error handling. The work is done by:

• generating the Telegram message string by calling a function that takes the Notice::notice object (translated from a Zeek record to a JavaScript object) and substitutes pertinent details into an HTML template;
• building a URL containing the (URL encoded) HTML message string in a JSON object expected by the Telegram API;
• calling Node.js’s https.get() API function to trigger the Telegram API.

The devil and the angels are in the details

The first JS-noob hurdle I hit is the dreaded “Cannot use import statement outside a module”. To fix this I created a package.json file containing {"type": "module"}. This felt a bit like cargo cult programming but it came in useful later when I had a clearer understanding of what I was doing.

Next, I found that I still needed a small Zeek script in the ZeekJS-flavoured package to define the API and (as the Zeek Blog post points out) to redefine existing Zeek types.

Finally, I found that when testing with a PCAP file as input (running zeek -r ssh.pcap for example) instead of having a long-running Zeek process, Zeek would exit before the code was done running. This was because the callbacks I passed to https.get() were run asynchronously and Zeek wasn’t sticking around for them to complete. To be fair, this would also happen with Zeek’s when statement, but those are less common (whereas asynchronous calls are used very often on Node.js). The fix was to run test scripts with redef exit_only_after_terminate = T; and then explicitly call terminate() when the test is done.

With these issues out the way, I get a ding on my phone and we’re in business:

The only significant functional difference between my reimplementation and Yacin’s original package is that I provide three Zeek hooks, one is called when sending the Telegram message succeeds, and two for handling different types of errors (with default implementations to raise Reporter errors).

Translating it to TypeScript (so it can be “transpiled” to JavaScript)

As I mentioned above, TypeScript is a language that is a superset of JavaScript. It adds type annotations that allow static type checking (something I’m fond of!). The official implementation is a source-to-source compiler (or “transpiler”) that emits readable JavaScript if the type checking passes.

Adding the annotations is not hard. I moved the original telegram.js file to telegram.ts then repeatedly ran tsc, noting the errors it emits and inserting appropriate types to eliminate them. Slightly less simple was dealing with so-called type definition files. These are a bit like C/C++ header files: they define the types in data structures and function signatures separately from the code where those things are declared. The compiler needs such type definitions to check the usage of libraries and other code not written in TypeScript.

First off the bat is the ZeekJS API itself, for which the documentation helpfully points to a PR containing a draft of zeek.d.ts.

Next, we need to fetch the type definitions for Node.js itself, which can be done with: npm install --save @types/node

Finally, we need type definitions for the Zeek records that are translated to JavaScript objects by ZeekJS. Here I crafted my own and put it in zeektypes.d.ts.

Originally I used a Makefile but thanks to a PR from Benjamin Bannier, now npm install && npm run build regenerates telegram.js from telegram.ts (I decided to commit the former into the repository to reduce “build time” dependencies for users who aren’t planning to modify the package).

Development-driven testing

Since it isn’t practical to use the Telegram API in a test, I started out writing a mock API server using ZeekJS to run a simple Node.js web application and a test client written in Zeek script.

To make the code testable in this manner I had to make two changes. First it is necessary for the Telegram API endpoint URL to be re-configured for the test environment. This is done by checking an environment variable:

const telegram_endpoint = process.env.TELEGRAM_ENDPOINT || "https://api.telegram.org/bot";

The second change is to allow the embedded Node.js runtime to trust the TLS CA certificate of our mock API service. I preferred to do this over switching between HTTPS for the real API and HTTP for the mock API since that would create a rich habitat for new (untestable) bugs. Normally this can be done by setting a NODE_EXTRA_CA_CERTS environment variable, but the way ZeekJS initializes the embedded Node.js prevents this from working. To work around this limitation I added the following to the mock API server:

if (process.env.NODE_EXTRA_CA_CERTS) {
    https.globalAgent.options.ca = fs.readFileSync(process.env.NODE_EXTRA_CA_CERTS, 'ascii');
}

The server certificate and key are issued for 127.0.0.1 (the IP used in the test) and (along with the CA certificate) have validities of 100 years to prevent tests from breaking (at least while I’m around to worry about it).

There are five tests—one to test the success case and then four error cases.

Here is an example of error.sh, the test for what would happen if the Telegram API returned an error (for example, if our token was invalid, or we hit a rate limit):

# @TEST-PORT: ZEEK_MOCK_PORT
# @TEST-EXEC: btest-bg-run api "MOCK_PORT=$(echo ${ZEEK_MOCK_PORT} | sed 's|/tcp||') zeek ${SCRIPTS}/mock_server.js > server.out"
# @TEST-EXEC: MOCK_PORT=$(echo ${ZEEK_MOCK_PORT} | sed 's|/tcp||') NODE_EXTRA_CA_CERTS=${ETC}/ca.pem TELEGRAM_ENDPOINT="https://127.0.0.1:${MOCK_PORT}/bad" zeek $PACKAGE ${SCRIPTS}/mock_client.zeek > client.out
# @TEST-EXEC: btest-bg-wait 10
# @TEST-EXEC: btest-diff client.out
# @TEST-EXEC: btest-diff api/server.out

The baselines for this test are what we expect the server and the client to output (respectively):

request url = /badxxx/sendMessage?chat_id=-123&text=%3Cb%3ETest%3A%3C%2Fb%3E%20Test%20notice%0A&parse_mode=HTML
responding with error

and

Test script api error hook:, Notice::Test, [statusCode=404, responseObject={
[ok] = false
}]

Finally, since I wanted to know if (or when?) a change breaks the package, I implemented a github actions test matrix that triggers on push, pull requests and on a schedule using the zeek/zeek:lts, zeek/zeek:latest, zeek/zeek-dev:latest docker containers (covering all the bases for versions of Zeek and ZeekJS that people are likely to care about).

Conclusion

I hope this article has provided some useful guidance for writing a real-world Zeek package in JavaScript or TypeScript, beyond the hello-world example.

The best part of this little project was interacting with Yacin (who kindly reviewed my code), Benjamin Bannier and Arne Welzel (the primary author of ZeekJS) - both from Corelight Open Source, and Seth Hall (who encouraged me to try out ZeekJS). I’m very interested to see what the community uses ZeekJS for and where Zeek goes in this brave new multi-language world.

To learn more about Zeek and how it supports Corelight network evidence, visit our website.