Modeling Heterogeneous Statistical Patterns in High-dimensional Data by Adversarial Distributions: An Unsupervised Generative Framework

TL;DR

This paper introduces FIRD, an unsupervised generative framework using adversarial distributions to model and disentangle heterogeneous statistical patterns in high-dimensional data, improving anomaly detection and clustering.

Contribution

FIRD is a novel unsupervised generative model that effectively captures diverse statistical patterns across different dimensions in high-dimensional data.

Findings

FIRD outperforms state-of-the-art methods with over 5% average AUC improvement.

FIRD successfully distinguishes synchronized fraudsters from normal users.

The method demonstrates superior modeling of heterogeneous patterns in various datasets.

Abstract

Since the label collecting is prohibitive and time-consuming, unsupervised methods are preferred in applications such as fraud detection. Meanwhile, such applications usually require modeling the intrinsic clusters in high-dimensional data, which usually displays heterogeneous statistical patterns as the patterns of different clusters may appear in different dimensions. Existing methods propose to model the data clusters on selected dimensions, yet globally omitting any dimension may damage the pattern of certain clusters. To address the above issues, we propose a novel unsupervised generative framework called FIRD, which utilizes adversarial distributions to fit and disentangle the heterogeneous statistical patterns. When applying to discrete spaces, FIRD effectively distinguishes the synchronized fraudsters from normal users. Besides, FIRD also provides superior performance on anomaly detection datasets compared with SOTA anomaly detection methods (over 5% average AUC improvement). The significant experiment results on various datasets verify that the proposed method can better model the heterogeneous statistical patterns in high-dimensional data and benefit downstream applications.