Monitoring AWS networks at scale

Corelight is pleased to announce our integration with AWS’s Traffic Mirroring to Gateway Load Balancer (GWLB) Endpoint as a Target. This integration simplifies the monitoring of network traffic and generating Corelight data in massively scaled-out public cloud environments.

When it comes to monitoring network traffic today, we see two primary deployment patterns, each with their own pain points.

Distributed monitoring - Customers deploy a single sensor or a cluster of sensors in every AWS Virtual Private Cloud (VPC) they want to monitor. This deployment model works well for customers that are looking to surgically get visibility into specific chokepoints (e.g. monitoring the egress traffic in the outbound VPC only). However, for customers that are looking to monitor a large number of VPCs, deploying and managing these security tools across their environment is challenging. They have to deploy capacity in each VPC and do not get the value of an elastically scalable solution in a centralized deployment. Additionally, security teams lose control of the tools and policies when they are deployed in VPCs administered by other organizations.

Centralized monitoring - Customers may choose to mirror the traffic and send it to a central VPC (over peered VPC connections or AWS Transit Gateway). A centralized network monitoring infrastructure allows security teams to exert better control in deploying and managing the security tools they need. However, peering a large number of VPCs to a central VPC can be challenging, since this requires setting up a peering connection for each new VPC that is created and managing CIDR overlap across VPCs. Additionally, the customer has to ensure that the traffic mirrored in a certain Availability Zone (AZ) stays within the same AZ to not incur cross-AZ data transit charges. While our customers prefer the centralized monitoring architecture, the deployment complexity has been challenging to overcome. 

With AWS’s latest announcement, mirrored traffic can be sent to a central VPC using Gateway Load Balancer (GWLB) endpoint as a target. This enables the centralized deployment of the network monitoring tools and eliminates much of the deployment complexity of peered VPC connections. With Corelight’s latest software release, our customers can now deploy Corelight sensors behind a GWLB and take advantage of this architecture. 

Centralized monitoring with GWLB

In a centralized security VPC, the customer sets up the GWLB and an auto-scale group of Corelight sensors as the target group behind the GWLB. Traffic is mirrored from EC2 ENIs of interest across AZs / VPCs in a customer’s cloud environment. The destination of the mirrored traffic is set at the GWLB endpoint resident in that AZ (to avoid cross-AZ charges). Once the traffic hits the GWLB endpoint, traffic is delivered securely and privately to the GWLB without the need to configure any other route tables. The GWLB ensures that each flow is directed to the same Corelight VM.

Advantages of using GWLB for centralized monitoring 

• Centralized infrastructure – Security teams can deploy and manage the security tools, and enforce policies within the Security VPC without relying on other organizations within the enterprise.
• No overprovisioning – A centralized design allows an elastically scalable design that eliminates the over-provisioning needed in a distributed architecture. Additionally, GWLB session affinity will ensure that the traffic does not cross AZ boundaries.
• On demand monitoring – In certain instances (e.g., to monitor an ongoing incident or audit an account) security teams could enable network monitoring of certain assets on an on-demand basis without needing a full stack deployment.

However, this solution includes both the GWLB and the GWLB endpoint to the overall architecture. So as customers consider both options for centralized monitoring, it is best to model out the total costs to compare both the GWLB and VPC Peering based solutions.

Extracting network metadata in AWS

Analysis of mirrored traffic in AWS is used to generate evidence that fuels threat detection, investigation and hunting for sophisticated Security Operations Center (SOC) teams. Corelight’s AWS Sensor turns mirrored traffic into comprehensive logs, extracted files, and custom insights via Zeek, a powerful, open-source network security monitoring framework used by thousands of organizations worldwide to accelerate incident response and unlock new threat hunting capabilities.

How can SOC teams use this data to secure their cloud? A great place to start is reading the MITRE ATT&CK Cloud Matrix for enterprises. This matrix covers the cloud-based TTPs that adversaries employ in the cloud. Additionally, we have put together a tool that identifies TTPs in the ATT&CK matrix where Corelight data can be used to discover and thwart attackers. Here are some real world examples of SOC teams utilizing network monitoring in the cloud.

Malware C2 using DGA: Evidence to investigate alerts

Malware uses Domain Generation Algorithms (DGA) as a common evasion technique to evade sinkholing. It works by generating new domains that the malware reaches out to for Command and Control (C2). 

Some cloud-based threat detections have built in detections for DGA but there are several challenges with these. Typically, they are signature oriented and can only look for known malicious domains. When they do alert, they generate a large number of false positives in some environments. If you have an application that reaches out to CDNs it could look like a DGA access (a random string of characters in the FQDN). These solutions require a lot of tuning and whitelisting to manage the false positives, and log evidence to triage some of these alerts. This is compounded by the fact that the context associated with these alerts is extremely shallow. You might have DNS lookup, DNSresponse, and some details around the host and the IP address – the actual correlation to hosts to figure out if this was a real alert is missing. 

Corelight’s data allows you to triage alerts by looking at the dns.log for random/long domain lookups. Are you seeing weird, random, long strings in your domain lookups? Are the domains themselves weird? Which of these DNS lookups are being resolved? 

Corelight’s DGA detection in dga.log uses known generators to flag malicious domains. It allows you to pinpoint what kind of DGA activity is going on, by what generator, from what kind of host. Once you have that alert, you have the entire data set of other logs available to pivot across and stitch together the narrative for the DGA origin. 

Behavioral detection of malware

There are lots of ways to get malware into an environment, for example unintentionally downloading an infected AMI from AWS Community AMI, or an intentional attack that drops malware into your environment. Malware can be designed to evade the usual signatures (malware hashes or C2 to known malicious domains).

One of our customers encountered a novel malware that was in the environment for a while. The first indication was that they saw weird things in HTTP.log: anomalous URIs and user agents, anomalous HTTP headers - things that are hard to find in VPC flow logs or cloud API logs.

Corelight can help detect payload drops by extracting and analyzing files and identifying mime types. Use HTTP.log to look for specific stuff in the POST body, anomalous URI and user-agents (not in VPC Flow Logs), or odd response MIME types. These can be stitched together and cross referenced across multiple logs (application logs) to tell the story of what happened: find what systems were infected and how the attacker traversed the environment. Additionally, the Corelight C2 collection contains numerous packages developed by the Corelight Labs research team focused on identification and detection of network C2 components. These packages deliver high-fidelity detections for known malware tools as well as highlight unknown C2 behaviors, allowing Corelight customers to uncover conventional and targeted malware communication.

As enterprise adoption of cloud explodes, there is an increasing awareness that security in the cloud cannot be taken for granted. We believe AWS’s support of traffic mirroring and expanded offering is a step in the right direction and we look forward to collaborating with AWS to empower SOC teams to accelerate detection and investigations. If you’d like more details on our public cloud offering and what’s coming in the future, please reach out.

By Vijit Nair, Senior Director of Product Management, Corelight