Investigating the effects of TLS 1.3 on Corelight logs, part 1

Introduction

I’ve written previously about Corelight data and encryption. I wanted to know how TLS 1.3 would appear in Corelight data, and compare the same network conversation over clear-text HTTP, TLS 1.2, and TLS 1.3. In this first of three parts, I will introduce TLS and demonstrate a clear-text HTTP session as interpreted by Corelight logs. In part 2 I will cover TLS 1.2, and in part 3 I will finish with TLS 1.3 and an overall conclusion.

If you’re not familiar with TLS 1.3, it’s the next version of the Secure Sockets Layer / Transport Layer Security mechanism used to encrypt traffic on the Web and elsewhere. According to the Internet Engineering Task Force,

In contrast to TLS 1.2, TLS 1.3 provides additional privacy for data exchanges by encrypting more of the negotiation handshake to protect it from eavesdroppers. This enhancement helps protect the identities of the participants and impede traffic analysis. TLS 1.3 also enables forward secrecy by default which means that the compromise of long term secrets used in the protocol does not allow the decryption of data communicated while those long term secrets were in use. As a result, current communications will remain secure even if future communications are compromised.

Last year the IETF published Request for Comment (RFC) 8446, which standardized TLS 1.3. The protocol is now available in widely used Web browsers such as Firefox and Chrome, and is in development for Apple’s Safari and Microsoft’s Edge browsers. Web servers are also offering the protocol, which is why the combination of TLS 1.3-enabled clients and servers could challenge traditional network security approaches.

As network security analysts, we recognize the need to both protect users and data, but also detect and respond to intruders who seek to abuse those resources. TLS 1.3, in seeking to further “protect from eavesdroppers,” has reduced the amount of data available to those inspecting network traffic.

For example, TLS 1.3 introduces perfect forward secrecy, eliminating the ability to decrypt traffic stored offline with a key saved for that purpose. That capability had been used by organizations such as those in the financial services industry. As a result, the Bank Policy Institute (previously the Financial Services Roundtable) and its BITS technology division proposed a “middlebox security protocol,” (PDF) initially called eTLS, that would avoid perfect forward secrecy and other anti-visibility features of TLS 1.3. BPI later changed eTLS to “Enterprise Transport Security,” or ETS, after resistance from the IETF. ETS has not gained the acceptance enjoyed by TLS 1.3, so this article examines TLS 1.3 and does not examine ETS.

Current statistics about TLS 1.3 implementations are difficult to find, but one paper noted that as of December 2018, approximately 10% of a sample set of servers surveyed by Netcraft offered TLS 1.3 connectivity. Therefore, it pays to understand how TLS 1.3 will affect security operations in the coming years. Corelight has conducted extensive research on TLS, and one of our programmers, Johanna Amann, is the primary maintainer of the Zeek TLS analyzer. While TLS 1.3 is a challenge to traditional visibility tools, Corelight continues to innovate, and the results documented in this post should be considered a snapshot taken at the time of writing, and not the final word on encryption.

The Experiments

The following experiment shows how these changes manifest in Corelight data. I used a Corelight AP200 sensor in my lab to collect the transaction logs. You may see extra entries in certain logs, such as the conn.log, thanks to the data enrichment offered by the Corelight sensor.

The test platform was a FreeBSD 12 virtual machine. FreeBSD 12 incorporates support for TLS 1.3 in the native OpenSSL library, which must be at least 1.1.1 for TLS 1.3. I used Curl to access a Web site, enabled.tls13.com, which offers the same test page whether the visitor uses no encryption, TLS 1.2, or TLS 1.3. The site enabled.tls13.com is hosted on several IP addresses, which required me to pass extra parameters to Curl in order to connect to a single IP address for each test. I typed bold italicized text as commands. Output which is worth extra notice is simply in bold.

First, I report the tools in use.

root@freebsd12:~ # date
Mon Apr 8 10:49:37 EDT 2019

root@freebsd12:~ # openssl version
OpenSSL 1.1.1a-freebsd 20 Nov 2018

root@freebsd12:~ # curl –version
curl 7.64.1 (i386-portbld-freebsd12.0) libcurl/7.64.1 OpenSSL/1.1.1a zlib/1.2.11 nghttp2/1.37.0
Release-Date: 2019-03-27
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets

No Encryption

In the first experiment, I visited the enabled.tls13.com Web site with no encryption. I told Curl to connect to port 80 where a HTTP Web server was listening.

root@freebsd12:~ # curl –resolve enabled.tls13.com:80:104.16.125.34 -v http://enabled.tls13.com

* Added enabled.tls13.com:80:104.16.125.34 to DNS cache
* Hostname enabled.tls13.com was found in DNS cache
*   Trying 104.16.125.34…
* TCP_NODELAY set
* Connected to enabled.tls13.com (104.16.125.34) port 80 (#0)
> GET / HTTP/1.1
> Host: enabled.tls13.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 08 Apr 2019 14:50:07 GMT
< Content-Type: text/html
< Content-Length: 140
< Connection: keep-alive
< Set-Cookie: __cfduid=d89c44d9f2a2a277333cefdf259d81a0f1554735007; expires=Tue, 07-Apr-20 14:50:07 GMT; path=/; domain=.tls13.com; HttpOnly
< x-amz-id-2: zr0BZddKmlY6oivxm4U6ZtAHcp0p57pMU0fzQ8QQEBuEiG5omYFmdORDNg6AZqep8JL7ByOHLDI=
< x-amz-request-id: 9FCC1A2D114AC667
< Last-Modified: Wed, 29 Jun 2016 00:38:23 GMT
< ETag: “31e34918942105b9adeff6485029054c”
< CF-Cache-Status: HIT
< Expires: Mon, 08 Apr 2019 16:50:07 GMT
< Cache-Control: public, max-age=7200
< Accept-Ranges: bytes
< Server: cloudflare
< CF-RAY: 4c450d41bb58cf42-IAD
<
<html xmlns=”http://www.w3.org/1999/xhtml” >
<head>
   <title>Test</title>
</head>
<body>
 <h1>Test</h1>
 <p>Testing</p>
</body>
</html>
* Connection #0 to host enabled.tls13.com left intact
* Closing connection 0

As you can see, the Web page is a simple “Test” site.

Let’s look at the Corelight logs, starting with the conn.log.

{“ts”:”2019-04-08T14:50:06.915521Z”,”uid”:”CV5Sp942eAcJwA6Yo5“,”id.orig_h”:”192.168.40.82″,”id.orig_p”:21229,”id.res

p_h”:”104.16.125.34″,”id.resp_p”:80,”proto”:”tcp“,”service”:”http“,”duration”:1.605495,”orig_bytes”:81,”resp

_bytes”:791,”conn_state”:”SF”,”local_orig”:true,”local_resp”:false,”missed_bytes”:0,”history”:

”ShADadFf”,”orig_pkts”:6,”orig_ip_bytes”:341,”resp_pkts”:4,”resp_ip_bytes”:963,”tunnel_parents”:

[],”resp_cc”:”US”,”orig_l2_addr”:”08:00:27:ac:ed:82″,”resp_l2_addr”:”fc:ec:da:49:e0:10″,”id.orig_h.name.src”:”

DNS_PTR”,”id.orig_h.name.vals”:

[“freebsd12.localdomain”],”id.resp_h.name.src”:”DNS_A”,”id.resp_h.name.vals”:[“enabled.tls13.com”]}

I bolded the connection identifier, the destination port, the IP protocol, and the service. We can compare these in later experiments. I next use the connection ID to find linked Corelight logs for the same session. I omit the conn.log in the output because we already viewed it.

$ zgrep CV5Sp942eAcJwA6Yo5 *

files_20190408_07:00:00-08:00:00-0700.log.gz:{“ts”:”2019-04-08T14:50:07.033531Z”,”fuid”:”FosDpFTdEwzJZEeMb“,”tx_hosts”:[“104.16.125.34″],”rx_hosts”:[“192.168.40.82″],”conn_uids”:[“CV5Sp942eAcJwA6Yo5″],”source”:”HTTP“,”depth”:0,”analyzers”:[“MD5″,”SHA256″,”SHA1″],”mime_type”:”text/html“,”duration”:0.0,”local_orig”:false,

”is_orig”:false,”seen_bytes”:140,”total_bytes”:140,”missing_bytes”:0,”overflow_bytes”:0,

”timedout”:false,”md5″:”31e34918942105b9adeff6485029054c”,”sha1″:

”fec36798c3fca94a29579a5d27e731f6e2af64a1“,”sha256″:

”815dbb80d430917a243e5895ccd6cc02aa462804f267aa42013b56ea350c3d0f”}

In the files.log, I bolded the file identifier, which does not have a follow-on in this example but will be important in the TLS 1.2 and TLS 1.3 experiments. We see the HTTP analyzer recognizes the payload as text/html. I next bolded the sha1 hash, which we could use to find the corresponding text/html file for the connection if we so choose. I am not extracting HTTP on this sensor, so I do not have a corresponding file.

http_20190408_07:00:00-08:00:00-0700.log.gz:{“ts”:”2019-04-08T14:50:07.008185Z”,”uid”:”CV5Sp942eAcJwA6Yo5″,”id.orig_h”:”192.168.40.82″,

”id.orig_p”:21229,”id.resp_h”:”104.16.125.34″,”id.resp_p”:80,”trans_depth”:1,”method”:”GET“,”host”:”enabled.tls13.com”,

”uri”:”/“,”version”:”1.1″,”user_agent”:”curl/7.64.1“,”request_body_len”:0,”response_body_len”:

140,”status_code”:200,”status_msg”:”OK“,”tags”:[],”resp_fuids”:[“FosDpFTdEwzJZEeMb”],

”resp_mime_types”:[“text/html”]}

In the http.log, I highlighted the GET / request, the Curl user agent, and the Web server’s 200 OK reply. We could only see this information because the traffic was passed in the clear.

Remember that I was able to easily connect the files.log and http.log entries to the original connection thanks to the connection ID, or uid as it is represented in the logs.

Conclusion

In part 1 of this series, I introduced TLS and explained how Corelight creates logs for clear-text HTTP traffic. In part 2 I will perform the same analysis for TLS 1.2, and in part 3 I will return with a comparison of TLS 1.3. See you then!