Corelight’s recent contributions to open-source Bro

When we founded Corelight in 2013, one of our goals was to build an organization that could sustain open-source Bro development long term. At that time, the core team behind Bro was still funded primarily through grants from the National Science Foundation. One of the underlying assumptions coming with that funding was that, with our work, Bro would become self-supporting: production-quality open-source software could generate a revenue stream to support its own development. Today, five years later, with many of the people who created and maintained Bro over the last two decades working at Corelight, that vision is becoming reality. Corelight is bridging the gap between the open-source software and enterprise environments looking for professional, supported products—they get the expertise of Bro’s creators packaged into a high-quality solution. In return, the success of Corelight enables us to invest heavily into advancing open-source Bro. You can rest assured that our team remains as committed to Bro as ever—no change there. In that spirit, I want to take the opportunity here to talk about a few of our more recent contributions to open-source Bro.

Getting Broker Ready for Production

One of Corelight’s main focus areas over recent months has been the Broker transition for Bro 2.6. While a series of Broker prototypes have existed for a while, the current Bro 2.5 still relies on a decade-old communication framework not designed for today’s network loads. Corelight’s Jon Siwek worked hard to tie together all the loose ends of this work, getting Broker ready for 2.6. He moved all of Bro’s standard scripts over to using Broker, and, during that process, improved many aspects of Broker’s API, implementation, documentation, and regression testing. Internally, Corelight is performing a series of tests to ensure that Bro & Broker, and hence the community’s Bro clusters, operate as expected.

Improving CAF

Jon also spent significant time on the innards of CAF, an actor framework library providing the low-level foundation for Broker. He tracked down and fixed a number of issues that were critically affecting stability & performance of Bro clusters. As Bro is gearing up for 2.6, CAF will be soon releasing a new stable version that incorporates all these changes, so that the trio of Bro, Broker, and CAF will smoothly work together. For easier installation, we also integrated CAF into Bro’s source code distribution, so that users don’t need to worry about getting a non-standard dependency in place before building Bro.

Supporting Dynamic Reconfiguration

Corelight recently contributed Johanna Amann’s Configuration Framework to Bro, which fills a long-time gap by providing an easy-use-to, unobtrusive infrastructure for changing Bro’s script-level tuning options dynamically at run-time.

Expanding Logging

We’ve made a number of logging improvements: restructuring DHCP logs (based on a community contribution); expanding connection histories that now capture TCP window closures, and also indicate repeats for a number of situations; and adding rate-limitations of “weirds” to address performance issues when your network exhibits a high level of non-conforming protocol usage. For geo-location, Bro now supports MaxMind’s new GeoIP2 databases, replacing their discontinued legacy format. For Bro’s TLS analysis, we reimplemented OCSP support, which was still missing from the recently contributed port of the analyzer to OpenSSL 1.1. We also added support for Cisco’s FabricPath and PPPoE over QinQ.

Improving Bro’s Scripting Language

We have been working on the scripting language as well: regular expressions can now be case-insensitive, and they can be created dynamically at runtime (which meant tracking down several memory leaks that had previously prevented Bro from allowing this). Sets and vectors have gained new operators, there’s support for bitwise operations, and we wrapped up & merged new functionality for type checking and type-safe casting.

Building & Installing Bro

Bro’s build system now knows to install Bro’s header files so that developers can build plugins without needing a copy of a Bro source tree laying around. We also added better support for cross-compiling Bro.

Bro Packages

As part of the broader Bro ecosystem, Corelight continues to maintain the increasingly popular Bro Package Manager.  We have open-sourced a number of new Bro packages as well:

• QUIC analyzer/detector parses and detects Google’s implementation of QUIC.
• Community ID provides a standardized way of labeling traffic flows in network monitors—an approach championed by the Bro and Suricata communities to enable correlation of flows across tools.
• HTTP Stalling Detector finds stalling DoS attacks taking advantage of web servers’ inability to differentiate legitimate client connecting over slow links from attackers deliberately sending data slowly to cause extra work.
• JSON Streaming Logs lets Bro write out JSON logs with additional attributes making life easier for external log shippers such as filebeats, logstash, and splunk_forwarder.

BroCon 2018 & Project Leadership

Speaking of community work, the Bro Leadership Team asked Corelight to host BroCon 2018, a task which our Events team has been happy to take on. As Bro’s NSF funding winds down, it is becoming quite challenging for the open-source project to organize large events on its own. Corelight employees are also active members of the Bro Leadership Team. In that role, Corelight staff have focused particularly on finding a new name for the project (still ongoing); moving the project back from Software Freedom Conservancy to ICSI; and being a liaison between the open-source project and the BroCon event team, providing history and context.

And all the little things that make Bro great

While not especially visible, Corelight has probably spent the greatest portion of its time on all the routine maintenance work that—while often hard to notice—is critical for any popular open-source project: fixing bugs & security issues; shepherding and merging community contributions; improving documentation & regression testing; and, crucially, providing users & developers with answers to questions. You can track much of this work on Bro’s public channels, such as mailing lists, the issue tracker, and Git repositories. Some work has to remain behind the scenes, however, such as discussion of security issues as well as interactions involving specifics of peoples’ environments.

We have never been more excited about the project, and are continually gratified and amazed at the way it has grown. Bro has always been popular among its fans, and we strongly believe that as it gets more usable and capable, deployment of Bro will continue to accelerate and really become a fundamental part of the modern security stack.