Don’t overlook the obvious: the answer may be you
Let me explain, because I’ve watched the following story unfold many times. A curious person gets super excited about Bro, deploys it widely in their organization, and makes a big impact on the local SOC. Everyone on that team becomes more effective, because Bro data helps them understand and respond to security incidents so much faster. Over time, this Bro advocate becomes the local Bro expert – responsible for configuration, tuning, documentation, patching, integration, etc. It’s a full-time job. And that’s OK, until the local Bro expert is hired away for the experience he or she just acquired!
It happens. Just think of all the skills that person gained along the way: about Bro itself, specialized network cards, BIOS/UEFI firmware options, network stack tuning, file systems, memory allocators, etc. This means your ‘local Bro expert’ is an asset but also a risk. Because so many companies are looking for Bro experts now, it cuts both ways. I’ve seen wonderful Bro deployments fall into disrepair when a key person leaves.
In fact, it was watching that pattern unfold several times that led us to develop the enterprise ready, turn-key Corelight Sensor about 18 months ago because we had identified that just creating Bro wasn’t quite enough. We sweated many, many details so that customers could confidently deploy Bro in less than 30 minutes, focusing effort on incident response, forensics, and threat hunting.
As a very small example of the tiny details we take care of for you on the Corelight Sensor and because I’d like to provide some useful tidbit for people running Bro on their own, I’d like to finish the post with a note about using tcmalloc. Tcmalloc is an alternative memory allocator that was originally created by Google as part of their Google Perftools package for memory debugging. The package has since been renamed to gperftools (found here: https://github.com/gperftools/gperftools) and is no longer officially maintained by Google. It’s intended to perform especially well in multithreaded applications and it has a number of other tweaks that make it an appealing choice as a memory allocator. A number of years ago we discovered that Bro performs noticeably better when tcmalloc is the memory allocator. This led to a change in the build system to use tcmalloc by default on Linux if it is discovered. Bro has been doing this for a long time but we’ve never publicly told everyone that they should be using it.
You should use whatever package system your OS uses to install gperftools and tcmalloc. On CentOS, it’s named “gperftools” and on Ubuntu it’s named “google-perftools”. After you install the package, you will want to reconfigure Bro with whatever configure arguments you used previously. If tcmalloc was found, you will see the following toward the end of the configure output:
If it show that gperftools is found and tcmalloc is found then you’re all set to build and reinstall. If you’ve had trouble getting rid of the last few percentage points of packet loss in your own Bro deployment, this easy change could possibly get rid of it right away! As you remove more and more of these small problems and Bro’s output becomes better, all of your downstream analysis is improved. Better data in equals better data out.
On the Corelight Sensor we are already using tcmalloc along with many other specialized configurations and an accelerated FPGA network card. This is all maintained and updated with zero effort from you so that you can focus on data and discovering intrusions.
And that’s just one example of how you’re covered if your Bro expert disappears one day.