Another cool thing about Bro: tracking files!

You probably know that Bro generates real-time data about network flows, highly valued by threat hunters & incident responders around the world.  But Bro can do a lot more, and in this blog series, we’ll highlight lesser-known features from time to time.

Today: tracking files!

First the problem statement: how do you monitor the files that go back and forth across your network? Of course, there are logs for some of your enterprise services, and maybe you’re getting info in the form of URLs or hashes from your proxies or other security tools…but what about everything else?  If you were given the hash of a file that you knew was malicious, how would you figure out if it had ever been on your network? What if that file never triggered an alert or system log?

Visibility into all files – not just network flows – is a powerful, under-appreciated feature of open-source Bro. Bro’s file analysis capabilities are pretty amazing, and the data it captures is a great resource for detection, response, and prevention.
Here’s how the feature works: whenever a file is transferred over the network using a protocol that Bro knows about, the file is tracked, hashes are created, and detailed data is logged to the file and its associated connections.

As an example, here’s a visit to the Slashdot web page by a browser, including an AJAX post as recorded by Bro’s files.log:

This is all part of a single HTTP connection and includes the HTML, favicon, some plain text, plus the JSON.  All these components have been recorded in the Bro logs with a number of important details:

• The first field is the UNIX timestamp.  The Corelight Sensor outputs this in a standard ISO 8601 date/time format and it’s super precise, helping to pinpoint exactly when a specific event happened in regards to a file.
  
  

• The second field is the file UID.  This is a unique ID/string generated per file seen.  You can reference this to look up other connections which transferred the exact same file.

• Third and fourth fields are the transmit and receive hosts for the file.

• The fifth field is a list of all the connections UIDs which this file was transferred over, often it’s just one but it could be part of a series of connections.  This same UID can be used to track an individual connection across any of Bro’s logs.
• The sixth field is the protocol source that Bro’s analyzers saw the file and extracted it from.

A few other interesting fields include file type, file name (if available), byte counts of various types, calculation of the entropy of the file, and hashes – MD5, SHA1, and optionally on the Corelight Sensor, SHA256.

Without delving into all the details of what’s available from Bro’s file analyzer, we see that a whole lot of actionable info is created for each of these files.  Remember that this same detail is recorded for EVERY FILE on your network.  And even better, it doesn’t matter what protocol the file was transferred over… as long as Bro can decode it, the file can be extracted – that includes HTTP, SMTP, FTP, IRC, SMB, etc.  In fact, Bro has 50 protocol analyzers. You can perform indicator matching and hunting across everything from web traffic to email attachments.

That’s an amazing amount of data, and as an incident responder, I relied heavily on the files log to help paint a picture of what might have happened for a particular event or series of file transfers.

But what if you need more than just the derived data about the transfer?  The Corelight Sensor can also extract all of the associated files and export them to a file server.  You can leave them for future investigations, and plumb them into a static or dynamic analysis pipeline – providing not just data about the connection and transfer but indicators and data extracted from the file itself.

Bro doesn’t stop there. The same level of forensic detail is available for individual protocols as well…we’ll get into some of the other logs in a future blog post.  
Do you have some unique ways you use the files.log or questions about how it could help your security team?  Drop us a line – info@corelight.com.