A Controlled Experiment on the Impact of Intrusion Detection False Alarm Rate on Analyst Performance

TL;DR

This study investigates how different false alarm rates in intrusion detection systems affect cybersecurity analysts' accuracy, speed, and decision-making, revealing that higher false alarms decrease precision and increase analysis time without impacting detection sensitivity.

Contribution

It provides empirical evidence on the effects of false alarm rates on analyst performance, highlighting the importance of false alarm management in IDS systems.

Findings

Higher false alarm rate reduces analyst precision by 47%.

Increased false alarms lead to 40% longer analysis times.

No significant change in detection sensitivity between false alarm rates.

Abstract

Organizations use intrusion detection systems (IDSes) to identify harmful activity among millions of computer network events. Cybersecurity analysts review IDS alarms to verify whether malicious activity occurred and to take remedial action. However, IDS systems exhibit high false alarm rates. This study examines the impact of IDS false alarm rate on human analyst sensitivity (probability of detection), precision (positive predictive value), and time on task when evaluating IDS alarms. A controlled experiment was conducted with participants divided into two treatment groups, 50% IDS false alarm rate and 86% false alarm rate, who classified whether simulated IDS alarms were true or false alarms. Results show statistically significant differences in precision and time on task. The median values for the 86% false alarm rate group were 47% lower precision and 40% slower time on task than the 50% false alarm rate group. No significant difference in analyst sensitivity was observed.