Explanation Method for Anomaly Detection on Mixed Numerical and Categorical Spaces

TL;DR

This paper introduces EADMNC, an explainable extension of the ADMNC anomaly detection model, providing global and local explanations while maintaining high accuracy and scalability for mixed numerical and categorical data.

Contribution

The work extends the ADMNC model to include explainability features using pre hoc and post hoc methods, with scalable implementation in Apache Spark.

Findings

EADMNC maintains the accuracy of the original ADMNC model.

The explainability methods are effective in real-world network intrusion detection.

Graphical and textual explanations improve understanding of anomaly detection results.

Abstract

Most proposals in the anomaly detection field focus exclusively on the detection stage, specially in the recent deep learning approaches. While providing highly accurate predictions, these models often lack transparency, acting as "black boxes". This criticism has grown to the point that explanation is now considered very relevant in terms of acceptability and reliability. In this paper, we addressed this issue by inspecting the ADMNC (Anomaly Detection on Mixed Numerical and Categorical Spaces) model, an existing very accurate although opaque anomaly detector capable to operate with both numerical and categorical inputs. This work presents the extension EADMNC (Explainable Anomaly Detection on Mixed Numerical and Categorical spaces), which adds explainability to the predictions obtained with the original model. We preserved the scalability of the original method thanks to the Apache Spark framework. EADMNC leverages the formulation of the previous ADMNC model to offer pre hoc and post hoc explainability, while maintaining the accuracy of the original architecture. We present a pre hoc model that globally explains the outputs by segmenting input data into homogeneous groups, described with only a few variables. We designed a graphical representation based on regression trees, which supervisors can inspect to understand the differences between normal and anomalous data. Our post hoc explanations consist of a text-based template method that locally provides textual arguments supporting each detection. We report experimental results on extensive real-world data, particularly in the domain of network intrusion detection. The usefulness of the explanations is assessed by theory analysis using expert knowledge in the network intrusion domain.