Over the last six months, we’ve read in the security press about a variety of managed service providers (MSPs) being compromised by nation-state and criminal actors. Some examples:
- December 2018 – The United States Department of Justice indicted two individuals associated with APT10 for their role in compromising major managed security service providers.
- April 2019 – The actors behind the intrusion at the Indian information technology outsourcing provider Wipro, reported by Brian Krebs remain at large.
- May 2019 – Similarly, the culprits behind the compromise of the German IT provider Citycomp, are also still at large according to recent reporting.
The intruders who compromised these targets are ultimately responsible for the security breaches and their consequences. It would be natural to write about how the MSPs could have better detected and responded to these intrusions. However, in this post I’m putting on my CSO hat, and pretending that I’m a client of the MSPs. I want a better idea of the scope of the incident, and any effect it might have had on my organization.
There are two “relationships” I care about when I consider assessing the risk of compromise to my organization, based on my dependency on a MSP. First, I wonder about the relationship between intruders and the MSP, and second, I wonder about the relationship between the MSP and my organization. By relationship, I mean interactions that represent threat actor activity that could affect the security of my digital resources.
These relationships could have manifested in several combinations:
- An insider at the MSP works with an insider at my organization to compromise my organization.
- An insider at the MSP uses privileged network access to compromise my organization.
- An outsider compromises the MSP via the Internet and works with an insider at my organization to compromise my organization.
- An outsider compromises the MSP via the Internet and uses privileged network access to compromise my organization.
In the first case, two insiders will probably take care to avoid creating network evidence. In the last three cases, network evidence will be created, with the last case creating the most network evidence.
As a CSO and client of an MSP, I would like to see the MSP use a network security monitoring (NSM) technology like Corelight to create Zeek logs for cases 2, 3, and 4 should the insiders from case 1 decide to use the network for some reason, I’d like that captured too). The Zeek Network Security Monitor summarizes conversations between parties in transaction logs, such as the conn.log (which collects metadata about TCP connections), that are perfect for these cases. Remember that the first questions asked by any incident response professional involve determining the scope of an intrusion. These questions include:
- When did the activity take place? (When did it start, and when did it end? Is it ongoing?)
- What digital resources are affected? (What systems, accounts, and data are involved?)
- How did the intruders interact with the digital resources?
There are many more questions to be asked during incident detection and response, but transaction logs as provided by Corelight are a great place to start. Because they are lightweight relative to full packet capture (PCAP), they can be stored for a very long time – months or even years – without severely impacting storage budgets.
Once investigators understand the infrastructure used by a threat actor to interact with a target, they can search Corelight logs for evidence of those relationships and conversations. Investigators can then review interactions between the MSP and my organization to better understand when those conversations represented threat activity, rather than legitimate MSP actions on my network.
MSPs could easily collect these Corelight logs and expose them on a per-customer basis to their constituency. They should be willing to expose logs of activity between the Internet and their customer management infrastructure, and between their customer management infrastructure and individual clients, limiting views on a per-customer basis.
This setup does not show activity between the Internet and general MSP computers, as that might reveal activity by MSP employees unrelated to threat actor activity. This limitation would fail to show the likely ways that MSP employees were compromised, but the MSP is probably unwilling to share that level of communication to their customers.
Incidentally, this sort of network security monitoring has been the norm for forward-leaning organizations who work with many third parties. The difference is that those organizations instrumented their ends of the gateways accessed by third parties, such as MSPs. This post proposes that the MSPs add NSM to their infrastructure in two cases which will assist with building confidence in their client bases. It goes without saying that I recommend MSPs employ NSM to protect their organization and network! MSPs must protect their employees, who are likely to be targeted and victimized, as intruders pivot to their real targets — the MSP customers.
Of course, we should not ignore the fact that the MSPs must properly architect, deploy, and protect any NSM infrastructure. As a largely passive approach, NSM can be operated in a quiet manner, likely to evade easy identification by the intruder. Defenders should be able to find and counter an adversary prior to the point where the intruder has discovered NSM infrastructure. Even if that fails to happen, engineers can build NSM infrastructures that can resist many forms of adversary scrutiny. That could be a future article!
I anticipate one objection to this scheme is that MSPs might choose to filter what records are available to their clients. Perhaps they might doctor or filter records associated with intruder activity in order to reduce the perceived scope of an intrusion, or to deny it altogether? Again, there are ways to architect a solution to avoid this problem. In extremis, the MSP could provide a feed of logs to a collection platform maintained by the customer, or by the customer’s log collection vendor of choice.
Customers are in many ways at the mercy of their MSPs. Currently their primary “protection” comes from the contracts they sign. This is increasingly untenable, as these breach reports demonstrate. Looking ahead, it may be necessary to consider a similar arrangement for cloud vendors. For now, I hope MSPs read this post and consider integrating NSM to meet the use cases described herein.
