Hardware-Based Deduplication for NDR: Cleaner Data, Lower Costs | Corelight

In today's complex network environments, ensuring complete visibility while optimizing resource utilization is paramount. Duplicate network traffic can overwhelm your monitoring infrastructure, create redundant alerts for SecOps, consume valuable storage, and obscure critical insights, making it harder for Network Detection and Response (NDR) solutions to spot genuine threats or anomalies. Network Packet Brokers often offer deduplication as a feature but it can add complexity and cost.

We're thrilled to announce a powerful new capability for Corelight’s physical network sensors equipped with our FPGA-based network cards: advanced hardware-based deduplication. This feature is designed to deliver cleaner, more efficient NDR detections, allowing you to focus on what truly matters for your security and operations.

All of this at no additional cost!

What is deduplication and how does it work?

Deduplication is a functionality that optimizes network traffic handling by discarding duplicate frames after parsing IP packet header and label or tunnel information contained within that frame.

You might wonder, "Aren't duplicate frames identical?" Not always! Duplicate frames typically traverse the network along different routes, meaning they might arrive with slight variations, such as different MAC source and destination addresses or even different VLAN tags. Corelight's deduplication is configured to compare specific parts of a frame to identify duplicates. When evaluating network traffic, a frame is considered a duplicate if it meets the following criteria:

• Same traffic segment: The frame must belong to the same designated network segment.
• Same correlation key and group ID: The frame must possess the same unique identifiers, a correlation key and group ID, as another frame within that segment. Correlation IDs are unique identifiers that help track requests or messages as they traverse through systems.
• Within the deduplication window: The time elapsed between the reception of the two frames cannot exceed a predetermined "deduplication window". This window defines the timeframe within which duplicate frames are considered valid duplicates and are subsequently removed.

This sophisticated approach ensures that the system efficiently identifies and processes only the unique data you need for analysis.

The value in enabling deduplication

The primary benefit of enabling deduplication is to optimize network traffic handling. By intelligently discarding redundant data, Corelight sensors can:

Optimize performance:

• Free up valuable resources on the Sensor for deeper analysis of unique and actionable traffic.
• Reduce unnecessary data processing of your network data, making it easier to pinpoint security threats, operational issues, and critical network events.

Improved detection:

• A cleaner and more relevant stream of network traffic means minimizing the risk of duplicate logs, alerts, or false positives.
• Reduction in volume of data processed also cascades down to reduction in volume of data stored for evidence using our Smart PCAP functionality.

Reduce cost:

Beyond performance, deduplication can also translate into direct cost efficiencies for Network Detection and Response (NDR) deployments. Crucially, our licensing model applies only to traffic after filtering is performed by our Corelight sensor NICs.

• Corelight's licensing model is capacity-based, meaning pricing is directly tied to the total monitored network traffic.
• Deduplication can lead to a lower overall volume of monitored traffic, which can directly result in potential cost savings on your Corelight licenses.

In contrast, software-driven NDR solutions may utilize:

• Software-based packet deduplication, checking for duplicates against the immediately preceding packet within a predetermined window.
• Deduplication based on the 5-tuple as a function of their detection and analysis engine.

While many NDR solutions can leverage external Network Packet Brokers (NPBs), Corelight incorporates this crucial optimization directly within its appliance architecture. This integrated hardware approach offers advantages by not only retaining valuable CPU cycles but by saving additional complexity and licensing costs.

Corelight hardware support

Deduplication is now available in our Corelight software v28.3 release.

This powerful deduplication feature is available on Corelight sensors equipped with an FPGA NIC that has dedup-capable firmware in its active bank. Many of our popular appliances are compatible, including:

• AP 3200, AP 5000, AP 5002 when equipped with the NT200A02 NIC.
• AP 1100, AP 1200, AP 3100, AP 3200 when equipped with the NT100A01 NIC.
• AP 1001, AP 3000 when equipped with the NT40E3 NIC.
• AP 200 when equipped with the NT40A01 NIC.

The integrated approach of hardware-based deduplication not only provides cleaner data for analysis but also aligns directly with Corelight's capacity-based licensing model, potentially leading to cost savings by reducing the actual monitored traffic volume. Ultimately, Corelight's method ensures a more predictable and efficient network security monitoring experience at scale.

Corelight customers can access configuration details through the deduplication topic on the Corelight documentation site.