Reduce network log volume by 50-80% | Corelight

According to Forrester Research, “How do we reduce our SIEM ingest costs?” is one of the top inquiries they receive from clients. Many security organizations rely on SIEMs for their detection, investigation, and response workflows, ingesting critical security information and events to detect and respond to threats. However, the large volume of logs — including endpoint, identity, cloud apps, email, and network logs — ingested, processed, and stored in the SIEM can result in high costs.

We're introducing data aggregation to our sensors, empowering organizations to dramatically reduce the volume of network logs sent to their SIEMs while preserving critical security insights. By reducing network log volume by 50-80%, data aggregation not only lowers operational costs but also reduces the data set, resulting in faster, more efficient searches and threat hunting.

What is data aggregation?

Data aggregation summarizes various network logs while retaining critical security insights. Similar network logs – such as conn, DNS, HTTP, SSL, files, and weird — generated within a specific timeframe are grouped together and summarized through logical operations such as sums, unions, or first or last values. Data aggregation sends log summaries to SIEM solutions, which reduces SIEM ingestion and, in turn, helps reduce SIEM costs, a major pain point for security leaders.

The value of network logs

Network logs provide detailed records of network traffic, including connections, applications, and protocols across the network. However, network logs produce large volumes of data because:

1. Networks handle large amounts of data traffic
2. Network logs capture detailed information about events
3. Networks are made up of various log-producing devices (e.g., routers, switches, firewalls, and servers)
4. Network devices perform various operations that generate logs constantly

Some organizations may choose to filter network logs to reduce the volume and address the cost concerns or performance issues of ingesting network logs in a SIEM. However, filtering correctly is often time-consuming and risks losing important data. Yet, network logs are necessary to detect malicious activity. They can audit activities like user logins and logouts, source and destination IP addresses, and protocols. Networking logs can also help identify lateral movement and detect protocol deception and errors. When paired with additional security log data like endpoint or email, logs help connect evidence of malicious actions and build timelines of threats within their organization.

The value of data aggregation

Data aggregation helps solve the problem of excessive network log volume through:

• Reduction of SIEM costs
  Aggregation can reduce volume by 50-80% without sacrificing security fidelity. Instead of filtering out repetitive logs, data aggregation summarizes these logs to condense the network data sent to SIEM. Customers in our early access program have seen an average of a 7:1 log consolidation.
• Acceleration threat hunting and investigations
  Network log summaries from data aggregation improve operational efficiency, accelerating threat hunting and investigations. The CPU reduction from fewer logs boosts detection accuracy and speeds up queries. Customers can experience faster Threat Intel Retro-matching and up to 5X longer look-back for forensics — and realize an overall improvement in their ability to hunt and investigate threats.

The Corelight Open NDR Platform

Corelight transforms network and cloud activity into evidence that security teams use to proactively hunt for threats, accelerate response to incidents, gain complete network visibility, and create powerful analytics. The Corelight Open NDR platform combines dynamic network detections, AI, intrusion detection (IDS), network security monitoring (NSM), packet capture (PCAP), and file analysis in a single security tool that’s powered by proprietary and open-source technologies such as Zeek, Suricata, and YARA.

Adding data aggregation to Corelight Open NDR platform with YARA enables security teams to realize a large reduction of network logs without sacrificing security fidelity, aiding in their ability to threat hunt and investigate pervasive threats to their environments without experiencing large costs to their SIEMs.

To learn more about sensors, read our datasheet.