How Zeek can provide insights despite encrypted communications

Overview

Encrypted communications are ubiquitous. While encryption provides confidentiality, it cannot prevent all means of traffic analysis. Certain protocols, such as SSH and TLS, ensure contents are not directly readable by monitoring systems. However, analysis of size and order of transmitted data can provide grounds for inference. This post will outline some methods Zeek employs to provide visibility into SSH connections.

Anyone who has observed traffic from a network interface with a publicly routable IP address has seen the variety of inbound connections the Internet has to offer. Unsolicited connections from scanners, crawlers, worms, and backscatter are commonplace. Services like Shodan, Greynoise, and Censys have created businesses around Internet-wide scan data, providing historic data sets for forensic and intelligence investigations. While these services are benign and accommodate requests to not be scanned, other entities are not so cordial. For example, consider the original variant of Mirai. The bot would scan for SSH and telnet services and attempt to log in using a list of credentials with the intent of using unauthorized access for propagation.

SSH Bruteforce Detection in Zeek

Detecting endpoints conducting bruteforce SSH scans is something Zeek does “out of the box”. You simply have to turn it on. By enabling this policy script Zeek tracks successful and failed authentication attempts per host in order to detect SSH bruteforcing. The logic within the script detects SSH bruteforce authentication attempts, also known as  credential stuffing if the credentials are stolen.

The way this script works is by using the SumStats framework to count ssh_auth_successful and ssh_auth_failed events. Each time the ssh_auth_failed event occurs, a counter is incremented. If the counter crosses the configured password_guesses_limit threshold, the script raises a notice. The script will also raise a notice if a password guessing client eventually successfully logs into a server it was seen submitting guesses to. If the same originating and responding hosts are seen from an ssh_auth_successful event after both were seen from a ssh_auth_failed event, Zeek can infer a guessing client authenticated to a server successfully. To comprehend what causes the ssh_auth_successful and ssh_auth_failed events to trigger an understanding of the SSH protocol is necessary.

The SSH Protocol(s)

SSH is best described by RFC4251: “The Secure Shell (SSH) Protocol is a protocol for secure remote login and other secure network services over an insecure network.” The protocol is composed of three sub-protocols, with differing messages types, to convey control information between a client and a server. The three sub-protocols are responsible for different tasks and include:

• The Transport Layer Protocol – used to establish encryption and for facilitating the other two sub-protocols
• The Authentication Protocol – used to authenticate endpoints of an SSH connection
• The Connection Protocol – used to provide interactive sessions, command execution, and forwarded X11 or port connections

Figure 1 (below) attempts to illustrate the exchanged messages of a typical SSH connection. Initially, a TCP connection is established. Both endpoints send their version (or identification) string, which is somewhat comparable to an HTTP user-agent except both endpoints provide one. After version strings are exchanged, an initial key exchange occurs which facilitates the creation of a symmetric key encrypted tunnel. The cipher suites, compression methods, and other configuration options are negotiated in the clear, before encryption begins. This exchange is visualized in figure 1 within the top box labeled “In the clear.” As an aside, the information exchanged in the clear is used by fingerprinting techniques such as HASSH.

The SSH protocol does not mandate what occurs after encryption begins. However, most client implementations will send a service request message to the server with type ssh-userauth (the first message sent within the green box of figure 1 labeled ‘Encrypted’) indicating the client would like to authenticate to the server. The server responds to the client’s service request with either an accept message, SSH_MSG_SERVICE_ACCEPT, or a disconnect message, SSH_MSG_DISCONNECT. The disconnect message would cause the TCP connection to terminate. If the TCP connection did not terminate, it is safe for an observing entity to assume that the server sent an accept message to the client. Measuring the size of the server’s accept message is an example of information leakage. It is key to how Zeek infers successful SSH authentication and thus raises ssh_auth_successful events.

After authentication is complete, the client sends another service request to the server. Unlike the first service request, which had a type of ssh-authuser, this second service request has a type of ssh-connection. Again, if the server accepts the client’s service request it will send a service accept message as opposed to a disconnect message. Connection protocol messages can then be exchanged as indicated in the purple box of figure 1. These messages are used to issue channel-specific requests such as a PTY session used for interactive logins, an X11 sessions used for X11 forwarding, or a sub-system session for sftp. Measuring packet sizes at this state in the SSH protocol can also be used to infer the type of session established between a client and server. For example, file transfers exhibit patterns in packet sizes which can be recognized easily.

SSH-Related Events in Zeek

With an understanding of the different states of the SSH protocol and packet sizes comes an understanding of when Zeek generates the ssh_auth_successful and ssh_auth_failed events. The current commit of master on GitHub shows the logic for when the SSH protocol analyzer raises the events with names beginning with “generate_ssh_auth_”, including ssh_auth_successful.

Observing previous commits reveals sources of inspiration for previous heuristics. As Zeek is an open source project, its source is auditable. The history of the SSH.cc file shows the current inferences were added over 4 years ago. The logic within the ProcessEncrypted function can be summarized as follows:

1. Identify the size of the server’s SSH_MSG_SERVICE_ACCEPT packet, labeled “Auth service accept” in figure 1.
2. Determine if the client’s (optional but typical) None authentication, labeled “Optional auth request (none)” in figure 1, results in an SSH_MSG_USERAUTH_SUCCESS or SSH_MSG_USERAUTH_FAILURE response from the server.
   1. An authentication success will be 16 bytes smaller than the size of the packet from step 1.
   2. A failure would be of a different size.
3. Determine if the server sent an optional banner, labeled “Optional banner” in figure 1, before sending the success message in step 2a to the client.
4. Check the size of the server response to the client’s second authentication attempt, labeled “Auth request (publickey)” in figure 1.
   1. An authentication success will be 16 bytes smaller than the size of the packet from step 1.
   2. A failure will be the same size as step 2b.

This summarizes how authentication successes are inferred, but how are authentication failures inferred? The ssh_auth_failed event never gets raised from the SSH protocol analyzer, and instead is raised from within Zeek’s scriptland. This line shows that ssh_auth_failed events are raised once an SSH connection is about to expire within Zeek’s core. Any completed SSH connection which did not result in an ssh_auth_successful event being raised will result in an ssh_auth_failed event being raised. Recall from the previous section (“SSH Bruteforce Detection in Zeek”) that the ssh_auth_failed event is hooked to count the originating and responding hosts (i.e. endpoints) of SSH  connections for comparison against a threshold.

Back to Bruteforcing

Recently, Corelight Labs analyzed a set of details from SSH packets which traversed a modestly sized network. The dataset included anonymized remote and local host numbers, remote and local port numbers, a shifted timestamp, and a packet size. During the course of our analysis, we quickly discovered that simply by examining packet sizes and ordering, we can make inferences about connections, even though they may be encrypted.

To depict this finding, we first munged the anonymized dataset into a series of conversations and then generated graphical representations of the conversations. These graphical representations are somewhat similar, in concept, to the SPLT analysis used by Joy. Example graphs and comments follow.

Figure 2 is a representation of an authentication guessing attempt. The red bars indicate packet sizes sent by local (L) systems while the blue bars indicate packet sizes sent by remote (R) systems. The x-axis indicates time. Figure 2 demonstrates a remote host, 1, attempting to bruteforce authenticate to a local host, 66. Local host 66 is likely configured to ensure some time passes before responding to client authentication attempts resulting in failures, as suggested by the temporal gaps between small packet exchanges. The large packets at the beginning of the conversation likely reflect the initial key exchange, labeled “kex_algorithms…” in figure 1.

Figure 3 shows the visual representation of a successful SSH conversation. A local system successfully authenticated to a remote system and began transferring data to the server. It is likely that PDU sizes generated by the system process employing SSH exceeded the MTU for Ethernet, indicated by packet sizes reaching 1500. After hypothesizing conversations which resembled this one were all successful, we learned that remote system 78 belongs to GitHub’s infrastructure. This SSH conversation was likely used to transfer a few small files.

Figures 4 and 5 are graphical representations of successful authentication resulting in large amounts of data over a long period of time. Figure 4 illustrates an SSH conversation with the majority of the data transmitted from the client, while figure 5 illustrates an SSH conversation with the majority of the data transmitted from the server.

Wrapping Up

This post outlined one of the many ways Zeek provides visibility into encrypted communications. Just because a connection’s content is encrypted doesn’t stop all means of analysis. Many of the concepts discussed in this post could also be applied to TLS. Some research makes this claim for HTTPS. Additionally, other researchers have claimed packet analysis can be used to identify the size of packets containing the SSH_MSG_CHANNEL_REQUEST containing requests for shell or pty access.

SSH is everywhere. If it’s used in your environment and you’d like to learn more about the insights Zeek can provide around SSH connections, see the HASSH Zeek package and the SSH policy scripts which ship, by default, with Zeek. If you’re interested in hearing more about this and other analyses Corelight Labs conducts on network data, reach out to us!

Special thanks go to Vlad Grigorescu for his contributions to the SSH analyzer as well as his assistance in understanding the heuristics within it.