Heterogeneous Anomaly Detection for Software Systems via Semi-supervised Cross-modal Attention

TL;DR

This paper introduces Hades, a semi-supervised, cross-modal attention-based method that effectively detects system anomalies by integrating heterogeneous logs and metrics, outperforming existing single-data-type approaches.

Contribution

The paper presents the first end-to-end semi-supervised approach that fuses logs and metrics using cross-modal attention for anomaly detection in software systems.

Findings

Hades outperforms baseline methods on large-scale datasets.

Integrating logs and metrics improves anomaly detection accuracy.

The approach is effective on real-world cloud data.

Abstract

Prompt and accurate detection of system anomalies is essential to ensure the reliability of software systems. Unlike manual efforts that exploit all available run-time information, existing approaches usually leverage only a single type of monitoring data (often logs or metrics) or fail to make effective use of the joint information among different types of data. Consequently, many false predictions occur. To better understand the manifestations of system anomalies, we conduct a systematical study on a large amount of heterogeneous data, i.e., logs and metrics. Our study demonstrates that logs and metrics can manifest system anomalies collaboratively and complementarily, and neither of them only is sufficient. Thus, integrating heterogeneous data can help recover the complete picture of a system's health status. In this context, we propose Hades, the first end-to-end semi-supervised approach to effectively identify system anomalies based on heterogeneous data. Our approach employs a hierarchical architecture to learn a global representation of the system status by fusing log semantics and metric patterns. It captures discriminative features and meaningful interactions from heterogeneous data via a cross-modal attention module, trained in a semi-supervised manner. We evaluate Hades extensively on large-scale simulated data and datasets from Huawei Cloud. The experimental results present the effectiveness of our model in detecting system anomalies. We also release the code and the annotated dataset for replication and future research.