September 14, 2017 by Vince Stoffer
You probably know that Bro generates real-time data about network flows, highly valued by threat hunters & incident responders around the world. But Bro can do a lot more, and in this blog series, we’ll highlight lesser-known features from time to time.
First the problem statement: how do you monitor the files that go back and forth across your network? Of course, there are logs for some of your enterprise services, and maybe you’re getting info in the form of URLs or hashes from your proxies or other security tools…but what about everything else? If you were given the hash of a file that you knew was malicious, how would you figure out if it had ever been on your network? What if that file never triggered an alert or system log?
Visibility into all files – not just network flows – is a powerful, under-appreciated feature of open-source Bro. Bro’s file analysis capabilities are pretty amazing, and the data it captures is a great resource for detection, response, and prevention.
Here’s how the feature works: whenever a file is transferred over the network using a protocol that Bro knows about, the file is tracked, hashes are created, and detailed data is logged to the file and its associated connections.
As an example, here’s a visit to the Slashdot web page by a browser, including an AJAX post as recorded by Bro’s files.log:
This is all part of a single HTTP connection and includes the HTML, favicon, some plain text, plus the JSON. All these components have been recorded in the Bro logs with a number of important details:
A few other interesting fields include file type, file name (if available), byte counts of various types, calculation of the entropy of the file, and hashes – MD5, SHA1, and optionally on the Corelight Sensor, SHA256.
Without delving into all the details of what’s available from Bro’s file analyzer, we see that a whole lot of actionable info is created for each of these files. Remember that this same detail is recorded for EVERY FILE on your network. And even better, it doesn’t matter what protocol the file was transferred over… as long as Bro can decode it, the file can be extracted – that includes HTTP, SMTP, FTP, IRC, SMB, etc. In fact, Bro has 50 protocol analyzers. You can perform indicator matching and hunting across everything from web traffic to email attachments.
That’s an amazing amount of data, and as an incident responder, I relied heavily on the files log to help paint a picture of what might have happened for a particular event or series of file transfers.
But what if you need more than just the derived data about the transfer? The Corelight Sensor can also extract all of the associated files and export them to a file server. You can leave them for future investigations, and plumb them into a static or dynamic analysis pipeline – providing not just data about the connection and transfer but indicators and data extracted from the file itself.
Bro doesn’t stop there. The same level of forensic detail is available for individual protocols as well…we’ll get into some of the other logs in a future blog post.
Do you have some unique ways you use the files.log or questions about how it could help your security team? Drop us a line – firstname.lastname@example.org.