SoK: Modeling Explainability in Security Analytics for Interpretability, Trustworthiness, and Usability

TL;DR

This paper critically analyzes explainability methods in security analytics, highlighting their limitations in trustworthiness and robustness across applications like anomaly detection, malware prediction, and adversarial image detection.

Contribution

It provides a comprehensive evaluation of current explanation methods in security, identifying key limitations and proposing prerequisites for trustworthy explanations.

Findings

Explanation methods often lack consistency and fidelity.

Current methods are vulnerable to adversarial manipulation.

Security-specific explanation requirements include stability and robustness.

Abstract

Interpretability, trustworthiness, and usability are key considerations in high-stake security applications, especially when utilizing deep learning models. While these models are known for their high accuracy, they behave as black boxes in which identifying important features and factors that led to a classification or a prediction is difficult. This can lead to uncertainty and distrust, especially when an incorrect prediction results in severe consequences. Thus, explanation methods aim to provide insights into the inner working of deep learning models. However, most explanation methods provide inconsistent explanations, have low fidelity, and are susceptible to adversarial manipulation, which can reduce model trustworthiness. This paper provides a comprehensive analysis of explainable methods and demonstrates their efficacy in three distinct security applications: anomaly detection using system logs, malware prediction, and detection of adversarial images. Our quantitative and qualitative analysis reveals serious limitations and concerns in state-of-the-art explanation methods in all three applications. We show that explanation methods for security applications necessitate distinct characteristics, such as stability, fidelity, robustness, and usability, among others, which we outline as the prerequisites for trustworthy explanation methods.