Editor’s note: This is the latest in a series of posts we have planned over the next several weeks where we explore topics such as network monitoring in Kubernetes, using sidecars to sniff and tunnel traffic, show a real-world example of detecting malicious traffic between containers, and more! Please subscribe to the blog, or come back for more each week.
In the first blog post in this series (Deeper visibility into Kubernetes environments with network monitoring) we showed how network monitoring solutions can overcome the security visibility blind spots in Kubernetes environments and in blog 2 (Sidecars for network monitoring) we looked at a specific example of using a sidecar-sniffer to sniff container-container traffic.
In this post, we demonstrate how the visibility of network traffic passing between pods and containers within the K8s network, can be utilized to detect a log4j exploit that occurred from a vulnerable Apache web application.
This use case was adapted from the POC exploit found here: https://github.com/cyberxml/log4j-poc. The architecture of the log4j demo environment is as follows:
The demo environment contained 4 K8s pods. Those pods where network traffic monitoring is required, have two containers, the service itself and vxlan. The vxlan container encapsulates all network traffic to and from the pod into a vxlan and forwards that traffic to the sensor-core pod. This traffic forwarding could also take place to an external network sensor.
When the exploit starts, the attacker uses the web application, hosted on the Apache web app pod, to send the exploit string:
to the log4j POC listener.
In this scenario, a special listener was created that had vulnerabilities built in and would perform the ldap lookup using the ldap server pod. Under normal circumstances, the traffic between the pods (ldap traffic in this case) would not be visible above the container system itself (e.g. to a higher level network sensor). Having the per pod sensor in place forwarding all traffic to the sensor-core, we were able to detect the exploit and generate a zeek notice.log entry on the sensor-core pod.
When this exploit succeeded, the call back from the ldap server provided a shell back to the attacker and the attacker was able to issue commands on the exploited pod.
The C2 communications from the shell created during the exploit were also captured and logged within the conn.log by the sensor-core pod.
By Stan Kiefer, Corelight Security Product Researcher