Accelerating Dependency Graph Learning from Heterogeneous Categorical Event Streams via Knowledge Transfer

TL;DR

This paper introduces ACRET, a transfer learning model that accelerates the learning of dependency graphs from heterogeneous event streams by transferring relevant entity and dependency knowledge, improving efficiency and detection accuracy.

Contribution

ACRET is the first model to effectively transfer knowledge for dependency graph learning from heterogeneous categorical data, addressing domain variety issues with entity filtering and dependency construction.

Findings

ACRET outperforms traditional methods in synthetic and real datasets.

It achieves at least 20 days lead time in intrusion detection.

It improves detection accuracy by over 70%.

Abstract

Dependency graph, as a heterogeneous graph representing the intrinsic relationships between different pairs of system entities, is essential to many data analysis applications, such as root cause diagnosis, intrusion detection, etc. Given a well-trained dependency graph from a source domain and an immature dependency graph from a target domain, how can we extract the entity and dependency knowledge from the source to enhance the target? One way is to directly apply a mature dependency graph learned from a source domain to the target domain. But due to the domain variety problem, directly using the source dependency graph often can not achieve good performance. Traditional transfer learning methods mainly focus on numerical data and are not applicable. In this paper, we propose ACRET, a knowledge transfer based model for accelerating dependency graph learning from heterogeneous categorical event streams. In particular, we first propose an entity estimation model to filter out irrelevant entities from the source domain based on entity embedding and manifold learning. Only the entities with statistically high correlations are transferred to the target domain. On the surviving entities, we propose a dependency construction model for constructing the unbiased dependency relationships by solving a two-constraint optimization problem. The experimental results on synthetic and real-world datasets demonstrate the effectiveness and efficiency of ACRET. We also apply ACRET to a real enterprise security system for intrusion detection. Our method is able to achieve superior detection performance at least 20 days lead lag time in advance with more than 70% accuracy.