Base-Rate Fallacy Redux and a Deep Dive Review in Cybersecurity

TL;DR

This paper critically reviews the application of the base-rate fallacy in cybersecurity, emphasizing the need for better false positive analysis, metric evaluation, and the applicability of attack graphs in constrained environments like IoT.

Contribution

It provides a comprehensive review of current cybersecurity research practices, highlighting logical fallacies and proposing improvements in evaluation methods and attack graph applicability.

Findings

False positives require deeper analysis similar to true positives.

Current metrics may hinder scientific progress in intrusion detection.

Online attack graphs have limited applicability in IoT and constrained environments.

Abstract

This paper examines the current state of the science underlying cybersecurity research with an emphasis on the non-signature-based intrusion detection domain. First, the paper re-examines the base-rate fallacy originally published by Axelsson, putting the impact of false positives into context. Given the relative high numbers of false positives, the paper argues for deeper analysis of false positives, akin to the analysis that true positives are treated to. The second section of the paper examines the metrics being used to analyze non-signature intrusion detection techniques, the current status quo of employed metrics, and the impact of the status quo on scientific advancement. Finally, the paper analyzes the use of online attack graphs and their applicability, especially in scenarios of constrained environments, such as Internet of Things devices. The use of offline attack graphs in such constrained environments is also examined. In essence, a deep dive review identified multiple areas throughout the field in which the effectiveness and validity of the scientific method can be greatly improved, e.g., through removal of logical fallacies.