This article was originally featured in TechBeacon.
What happens at the point in time when an organization’s information security is compromised?
This blog explores an important paradigm shift that occurs at the very moment of compromise and that can be leveraged to turn the tables on the attackers.
Spy novels often involve operatives who have long been trying to infiltrate an organization. Waiting, learning, picking their mark, not being too rash, being very patient and strategic until finally they see their opportunity, make their play and have a breakthrough. The play worked perfectly, they have now gained trust and are now “on the inside.” What happens next? Well, the stakes are raised significantly for the operative, who must evade being discovered, keeping their cover lock-tight while at the same time carrying out their mission. All that’s needed for the cover to be blown is one misstep, one quirk in body language at a critical moment, anything that seems out of the ordinary and raises suspicion. A top operative may remain uncovered, coming and going without raising any eyebrows, but even the best can make missteps. Only one error is required, and suddenly the tables are tilted in favor of the defender, not the infiltrator.
Let’s take this same paradigm shift into information security in its simplest form:
needs one good day to succeed
needs only one bad day to fail
needs only one bad day to fail
needs one good day to succeed
Let’s now illustrate this with an extreme, but nonetheless actual real world example. The moral of the story could well be “You only need to catch them once!”
A world-class red team was conducting an operation against an organization that, at the time, was under rapid growth and had reasonable security visibility, but was still building out components and improving its detection and response capabilities. After a significant recon phase and trying a few fruitless avenues of attack, the red team was able to determine the software version of an externally facing system, for which they then built out a lab replica and developed a Zero day exploit—tailor-made for that target. At a time of their choosing, the Zero-day was launched at the externally facing system, and the system was compromised. The red team had a good day and the defender just had their first bad day (but they didn’t know it quite yet). This is where the paradigm shift occurred, the red team then had to evade detection systems of which they didn’t even have knowledge. Just one mistake may bring them undone.
Meanwhile, the defenders were doing their best with what they had. They had been successful every day up to this point, fending off millions of attempts, including from the attacker. Quite separately, the defender had developed a “leetspeak” search, which would alert the defender if there were hacker “leetspeak” patterns in logs. It was never expected to work against a red team; instead, it was designed to catch script kiddies, proof of concept exploits, and commodity attackers. However, there’s one thing about red teams; sometimes they just can’t help being cute with their exploit attempts, and in this case they named a particular element something along the lines of "pwnedby31337". They perhaps forgot to rename it from their lab version to some innocuous looking name that would have easily gone unnoticed. When this element invariably appeared in logs, it was more than enough to trigger the leetspeak search alert, which ran every hour against the log repository. The red team had made one small OPSEC blunder, indeed they had their first bad day and they were detected, while the blue team had one very good day.
While this may appear to be a far-flung or a “lucky blue team” case, it is real and the wider reality is that many attackers have suboptimal operational security—even those with excellent OPSEC can make mistakes and oversights. Red teams are skilled in offense but do not focus primarily on defense and detection evasion. It makes sense that an attacker would make more errors in defense than a defender would. The good news is that taking advantage of this does not always involve a heavy or expensive lift. In fact, there are a number of tools that can easily provide defenders with comprehensive network data and alerts, sometimes even for free. These tools include free open source tools like Zeek and Suricata. These have thriving communities, constantly churning out improvements and detection content for significant events like SUNBURST/Solarigate and Log4Shell. Furthermore, defenders can use these detections to create bespoke detection content that’s specific to the unique environment that they’re defending.
Let’s look at two basic elements that enable detection.
We need the evidence/data/visibility. Call it what makes sense to you, the essence is we can’t detect what we can’t see. We need raw logs (such as the ones Zeek provides), not just more alerts that aren't tuned or targeted for the defender’s unique need state. As a defender, you need context to help you understand the thousands of alerts that come your way every day.
We need search logic. These can broadly be fitted into two types.
Anomalous behavior “We are looking for weird things”.
Knowing what is normal in your environment, which is much easier said than done.
Search for anything outside normal, examples:
SSH sessions from the laptop of someone in HR? That’s weird...
RDP to the domain controller by any machine that is NOT from the sysadmin group. That’s weird…
Brute force scanning from an internal desktop machine? That’s weird…
How might this play out in a real world example? After their initial compromise, the attacker has a foothold inside the victim network. The attackers then tend to transition to a comparatively hasty/noisy phase, as they need to learn about an environment they have compromised as quickly as they can (note that quickly is mostly orthogonal to stealth). For example, the attacker will generally want to know:
Where the most valuable data is located, and how/if it is protected.
What is the topology of network, services and software? Some attackers may even note the security technologies that are in place (if they are evident by the scanning) so as to avoid them, or even attempt to disable them.
Which users have privileged access with a view to compromise these accounts in order elevate privileges to the level required to access protected target data
As mentioned earlier, to discover these attributes the attacker will often use scanning tools, e.g., nmap, Bloodhound/SharpHound/AzureHound, AD_LDAP_dumper, and a vast array of scanning tools which you can read more about here https://attack.mitre.org/techniques/T1046/. The key for defenders is that when these network scans occur they generally leave lots of trails behind, and Zeek logs provide this visibility during the attacker’s critical early “discovery” phase—that is before they reach their ultimate goals.
It is often said that effective defense is harder to achieve than an attack, and this is mostly true. However, this is true only up to the point where most people focus, to the “left of boom,” the boom being the point of compromise. This is where the tables are turned and the defender now has an advantage. In an environment where compromises can and do occur despite best practice controls being in place, it behooves organizations to not only “protect against compromise,” but, in parallel, “assume compromise and detect it.” For the latter, the most fundamental requirement is that there must be artifacts/evidence/logs collected at intelligent levels of abstraction that scale, such as the metadata style logs produced by Zeek. None of it is easy, but without data: “no logs, no crime!”