Entity Embedding-based Anomaly Detection for Heterogeneous Categorical Events

TL;DR

This paper introduces a probabilistic model called APE that embeds entities into a latent space to effectively detect anomalies in heterogeneous categorical events, outperforming existing heuristics-based methods.

Contribution

The paper presents a unified probabilistic framework with entity embeddings for anomaly detection, addressing the lack of intrinsic distance measures and large event space challenges.

Findings

APE achieves higher detection accuracy than state-of-the-art methods.

The model efficiently learns from large event spaces using Noise-Contrastive Estimation.

Experimental results validate the effectiveness of the proposed approach.

Abstract

Anomaly detection plays an important role in modern data-driven security applications, such as detecting suspicious access to a socket from a process. In many cases, such events can be described as a collection of categorical values that are considered as entities of different types, which we call heterogeneous categorical events. Due to the lack of intrinsic distance measures among entities, and the exponentially large event space, most existing work relies heavily on heuristics to calculate abnormal scores for events. Different from previous work, we propose a principled and unified probabilistic model APE (Anomaly detection via Probabilistic pairwise interaction and Entity embedding) that directly models the likelihood of events. In this model, we embed entities into a common latent space using their observed co-occurrence in different events. More specifically, we first model the compatibility of each pair of entities according to their embeddings. Then we utilize the weighted pairwise interactions of different entity types to define the event probability. Using Noise-Contrastive Estimation with "context-dependent" noise distribution, our model can be learned efficiently regardless of the large event space. Experimental results on real enterprise surveillance data show that our methods can accurately detect abnormal events compared to other state-of-the-art abnormal detection techniques.