A Latent Space Correlation-Aware Autoencoder for Anomaly Detection in Skewed Data

TL;DR

This paper introduces a kernelized autoencoder that uses a robust Mahalanobis distance and mutual information maximization to improve anomaly detection in skewed, high-dimensional data by capturing correlation information more effectively.

Contribution

It proposes a novel hybrid loss function combining a robust Mahalanobis distance with mutual information maximization for enhanced anomaly detection in skewed data.

Findings

Improved detection of near and far anomalies in skewed datasets.

Enhanced reconstruction accuracy by capturing correlation information.

Robustness to non-Gaussian, skewed data distributions.

Abstract

Unsupervised learning-based anomaly detection in latent space has gained importance since discriminating anomalies from normal data becomes difficult in high-dimensional space. Both density estimation and distance-based methods to detect anomalies in latent space have been explored in the past. These methods prove that retaining valuable properties of input data in latent space helps in the better reconstruction of test data. Moreover, real-world sensor data is skewed and non-Gaussian in nature, making mean-based estimators unreliable for skewed data. Again, anomaly detection methods based on reconstruction error rely on Euclidean distance, which does not consider useful correlation information in the feature space and also fails to accurately reconstruct the data when it deviates from the training distribution. In this work, we address the limitations of reconstruction error-based autoencoders and propose a kernelized autoencoder that leverages a robust form of Mahalanobis distance (MD) to measure latent dimension correlation to effectively detect both near and far anomalies. This hybrid loss is aided by the principle of maximizing the mutual information gain between the latent dimension and the high-dimensional prior data space by maximizing the entropy of the latent space while preserving useful correlation information of the original data in the low-dimensional latent space. The multi-objective function has two goals -- it measures correlation information in the latent feature space in the form of robust MD distance and simultaneously tries to preserve useful correlation information from the original data space in the latent space by maximizing mutual information between the prior and latent space.