March 8, 2019 | Posted in Blue Teams, Purple Teams, GRC, and Strategy by Evan Perotti and Mike Pinch

 

Back in December 2018, MITRE released the first round of its evaluations on EDR tools, including Carbon Black, CounterTack, Crowdstrike, Endgame, RSA, Sentinal One, and Windows Defender.  Specifically, MITRE tested the APT3 threat group (https://attack.mitre.org/groups/G0022/) against the products and rated how well they performed.

Above: APT3 Tactics highlighted in green

 

Recently MITRE published the first phase of its “Rolling Admissions” program, which added vendors FireEye and Cybereason.  Last time around (http://securityriskadvisors.com/blog/a-closer-look-at-mitre-attck-evaluation-data/), SRA scraped all the test result data from the MITRE results, and published it in a more head-to-head view, so that you could see how each vendor did against one another.

We recently updated our dataset (stored here: https://github.com/SecurityRiskAdvisors/mitreevalsdb) and have re-run some of our favorite queries to see how the new additions faired against the first wave of competitors.  What did we find?  Excellent performance from FireEye, and mid-pack performance from Cybereason.  In any case, this is a high level summary and detailed results should be examined if you’re seriously considering any of these products.  We tend to give the most credit to those orgs that went into the first round of this test blindly, and it seems that the ‘rolling admissions’ participants have a leg up in that they are taking an open-book test now.  That being said, Crowdstrike continued its dominance in this test, even while being from the first wave of participants.  Details below:

Query: select vendor, count(vendor) as total_detections from edr WHERE General = ‘yes’ or Specific = ‘yes’ group by vendor ORDER BY total_detections DESC;

If you want to recreate these results yourself, visit our github page here https://github.com/SecurityRiskAdvisors/mitreevalsdb to download mitreevals.db, then load that sqlite database into a DBMS, such as the web based system here: http://inloop.github.io/sqlite-viewer/

For more information, view the data yourself here! https://attackevals.mitre.org/evaluations.html