Data-Efficient and Interpretable Tabular Anomaly Detection

TL;DR

This paper introduces DIAD, a novel, interpretable, and data-efficient anomaly detection framework for tabular data that effectively incorporates small amounts of labeled data and provides explanations for detected anomalies.

Contribution

The paper proposes DIAD, a white-box model based on Generalized Additive Models, for anomaly detection that is both interpretable and capable of semi-supervised learning with minimal labeled data.

Findings

DIAD outperforms previous methods in unsupervised and semi-supervised settings.

Incorporating 5 labeled anomalies improves AUC from 86.2% to 89.4%.

DIAD provides meaningful interpretations of anomalies.

Abstract

Anomaly detection (AD) plays an important role in numerous applications. We focus on two understudied aspects of AD that are critical for integration into real-world applications. First, most AD methods cannot incorporate labeled data that are often available in practice in small quantities and can be crucial to achieve high AD accuracy. Second, most AD methods are not interpretable, a bottleneck that prevents stakeholders from understanding the reason behind the anomalies. In this paper, we propose a novel AD framework that adapts a white-box model class, Generalized Additive Models, to detect anomalies using a partial identification objective which naturally handles noisy or heterogeneous features. In addition, the proposed framework, DIAD, can incorporate a small amount of labeled data to further boost anomaly detection performances in semi-supervised settings. We demonstrate the superiority of our framework compared to previous work in both unsupervised and semi-supervised settings using diverse tabular datasets. For example, under 5 labeled anomalies DIAD improves from 86.2\% to 89.4\% AUC by learning AD from unlabeled data. We also present insightful interpretations that explain why DIAD deems certain samples as anomalies.