Anomaly Detection for Aggregated Data Using Multi-Graph Autoencoder

TL;DR

This paper introduces MGAE, a novel multi-graph autoencoder for anomaly detection in aggregated system log data, significantly improving reconstruction accuracy by leveraging relationships between aggregated events.

Contribution

The paper proposes MGAE, a new convolutional graph autoencoder that exploits relationships in aggregated data for improved anomaly detection performance.

Findings

MGAE achieves 60% reduction in reconstruction error.

MGAE outperforms standard graph autoencoders in experiments.

Graph representations effectively model relationships in aggregated logs.

Abstract

In data systems, activities or events are continuously collected in the field to trace their proper executions. Logging, which means recording sequences of events, can be used for analyzing system failures and malfunctions, and identifying the causes and locations of such issues. In our research we focus on creating an Anomaly detection models for system logs. The task of anomaly detection is identifying unexpected events in dataset, which differ from the normal behavior. Anomaly detection models also assist in data systems analysis tasks. Modern systems may produce such a large amount of events monitoring every individual event is not feasible. In such cases, the events are often aggregated over a fixed period of time, reporting the number of times every event has occurred in that time period. This aggregation facilitates scaling, but requires a different approach for anomaly detection. In this research, we present a thorough analysis of the aggregated data and the relationships between aggregated events. Based on the initial phase of our research we present graphs representations of our aggregated dataset, which represent the different relationships between aggregated instances in the same context. Using the graph representation, we propose Multiple-graphs autoencoder MGAE, a novel convolutional graphs-autoencoder model which exploits the relationships of the aggregated instances in our unique dataset. MGAE outperforms standard graph-autoencoder models and the different experiments. With our novel MGAE we present 60% decrease in reconstruction error in comparison to standard graph autoencoder, which is expressed in reconstructing high-degree relationships.