MITRE 2: The Return

Detection and protection

For this second MITRE Evaluations, we tested both the detection and protection capabilities of our EDR.

Beyond going from 3 to 4 days of testing, with staggered working hours to improve our endurance, this expanded scope implies that MITRE evaluates not only the way our EDR detects security events, but also its ability to block threats.

Our CTI team is made up of approximately ten experts who, on the strength of last year’s pizzas, devoted the first day to detecting attacks by playing out two different scenarios; a third scenario was played out on Day 2; the three scenarios were replayed on Day 3 (are you still with us?).

As a reminder, for all intents and purposes, MITRE evaluates EDRs according to the level of detection applied to the identified events:

None: nothing has been detected;
N/A: the evidence provided does not meet the documented detection criteria, or no evidence has been provided;
General: the tool has detected something, but without specifying what was detected, e.g. “Malicious file” without further details;
Tactic: an alert has sounded and contains the tactic associated with the technique detected;
Technique: data available on the tactic and technique, i.e. the maximum level of detection.

What happens next? Once the scenarios have been played out, MITRE collects the detection traces and analyzes them. To do this, solution vendors provide proof of detection in the form of screenshots of their own interface, for all stages and sub-stages. Which means, in practice… almost 500 screenshots in all.

Finally, Day 4 was dedicated to testing protection capabilities, which are designed to answer a very simple question: is the EDR capable of blocking threats or not?

This was a first for us. For all participants, this year was also marked by other novelties: the evaluation of false positives and running tests on macOS.

But in concrete terms, what did all of this translate to?

After Turla, it’s time for a North Korean APT and ransomware

In 2023, the Turla group was emulated for the MITRE Engenuity tests. This group is known for its intrusion techniques and stealth. In 2024, we had to deal with:

DPRK – macOS focus
Emulation of actions that could be perpetrated by a feared North Korean actor whose field of action has expanded in recent years, targeting macOS systems among others.
The scenario: a backdoor executed by an initial malicious script from a supply chain attack, followed by persistence and a second-level backdoor that performs discovery, credential access, and exfiltration operations. Its tools include a malicious Ruby script and backdoors.

CL0P – Windows and Linux focus
Active since at least 2019, CL0P spreads ransomware, and like most other ransomware families, it uses the “Steal, Encrypt, Leak” technique.
The scenario: persistence and RAT payload in memory for discovery and exfiltration, then download and execution of ransomware that applies several evasion techniques in impact and defense. Finally, the attackers proceed to exfiltrate files and “detonate” the ransomware.
LockBit Ransomware – Windows and Linux focus
LockBit is a ransomware family infamous for its sophisticated tools, extortion methods, and high severity attacks. It is a “Ransomware-as-a-Service” (RaaS), which enables other threat actors to use its ransomware for their attacks. Also, LockBit affiliates can hijack legal open-source tools for malicious actions such as network reconnaissance, remote access, extraction of credentials or secrets, and file exfiltration.
The scenario: initial access via compromised credentials, followed by discovery, credential access, and persistence. The activity continues with malicious actions on a Linux KVM server, followed by the appearance of a new implant that downloads and executes an exfiltration tool and ransomware that propagates across the network and performs a defense evasion and impact. Finally, virtual machines are shut down and files exfiltrated and encrypted.

What we expected (and didn’t expect)

As we said in the introduction, even if the tests change every year, we thought we had gotten the hang of it thanks to the 2023 experience.

The 2024 edition proved more eventful than expected, with more than a few surprises along the way.

Firstly, there seemed to be a number of steps missing between the initial arrival at a position and the execution of the attack. In addition to depriving us of the value of our detections on these steps, the fact that these crucial steps for detection were missing made the scenarios less realistic – there are always recognitions and lateralizations in real attacks.

Also, almost a third of the steps were based on Windows API detection.We were then supposed to detect actions that can only be seen by security solutions that base their detection on userland hooking, but this method is not recommended by Microsoft as it can cause incompatibility problems with other software as well as instability on the system. In line with Microsoft’s recommendations, we do not use userland hooking, and like other Microsoft compliant solutions, this sometimes penalized us in these tests.

Furthermore, certain simulated attacker behaviors simply had no chance of occurring in a real compromise. For example, the scenario involving LockBit called for actions that in no way corresponded to the attackers’ modus operandi.

Despite these surprises, MITRE remains a test that keeps our feet to the fire! Our thanks to the MITRE teams, who challenge all industry players to strengthen global cybersecurity.

It’s an opportunity to review our detection rules in their entirety and develop new detection capabilities to continue to be state-of-the-art.

This 2024 edition was also an opportunity to continue perfecting the Static Analysis tools available in the console, and to apply new detection techniques that we are developing on an ongoing basis.

In the field, we encounter Windows and Linux more often, and being challenged on macOS has pushed us to develop new detection rules and improve our detection capabilities.

In addition, the false positives evaluation confirmed our performance in this crucial area, helping our users focus on security events that require special attention or investigation.

Last but not least, these tests enabled us to put Static Analysis to the test, one of our functionalities which proved its relevance and qualities as a detection assistant. Static Analysis can be used to associate contextual information and correlate files from different alerts to accelerate the detection process. This feature helps to quickly categorize alerts and the executable concerned, based on, for example, the character strings contained, embedded files, or the configuration of known malware such as CobaltStrike.

So, see you next year? You bet!