CSCAD: Correlation Structure-based Collective Anomaly Detection in Complex System

TL;DR

CSCAD is a novel anomaly detection framework for large, complex systems that leverages correlation structures and deep learning to identify collective anomalies effectively without extensive labeled data.

Contribution

The paper introduces CSCAD, a correlation structure-based model combining graph convolutional networks and autoencoders for high-dimensional anomaly detection in large systems, adaptable to semi-supervised settings.

Findings

Outperforms baseline methods on five public datasets.

Effectively detects collective anomalies in complex systems.

Utilizes a new EMI metric to enhance correlation mining.

Abstract

Detecting anomalies in large complex systems is a critical and challenging task. The difficulties arise from several aspects. First, collecting ground truth labels or prior knowledge for anomalies is hard in real-world systems, which often lead to limited or no anomaly labels in the dataset. Second, anomalies in large systems usually occur in a collective manner due to the underlying dependency structure among devices or sensors. Lastly, real-time anomaly detection for high-dimensional data requires efficient algorithms that are capable of handling different types of data (i.e. continuous and discrete). We propose a correlation structure-based collective anomaly detection (CSCAD) model for high-dimensional anomaly detection problem in large systems, which is also generalizable to semi-supervised or supervised settings. Our framework utilize graph convolutional network combining a variational autoencoder to jointly exploit the feature space correlation and reconstruction deficiency of samples to perform anomaly detection. We propose an extended mutual information (EMI) metric to mine the internal correlation structure among different data features, which enhances the data reconstruction capability of CSCAD. The reconstruction loss and latent standard deviation vector of a sample obtained from reconstruction network can be perceived as two natural anomalous degree measures. An anomaly discriminating network can then be trained using low anomalous degree samples as positive samples, and high anomalous degree samples as negative samples. Experimental results on five public datasets demonstrate that our approach consistently outperforms all the competing baselines.