Leveraging Log Instructions in Log-based Anomaly Detection

TL;DR

This paper introduces ADLILog, a novel anomaly detection method that leverages log instructions from source code to improve detection accuracy without extensive labeled data, suitable for industrial deployment.

Contribution

It presents a new approach using log instructions from GitHub projects to train anomaly detection models, reducing reliance on labeled data and enhancing performance.

Findings

ADLILog outperforms existing methods by up to 60% in F1 score.

The approach requires no manual labeling, making it practical for real-world use.

The model is efficient, small, and suitable for industrial deployment.

Abstract

Artificial Intelligence for IT Operations (AIOps) describes the process of maintaining and operating large IT systems using diverse AI-enabled methods and tools for, e.g., anomaly detection and root cause analysis, to support the remediation, optimization, and automatic initiation of self-stabilizing IT activities. The core step of any AIOps workflow is anomaly detection, typically performed on high-volume heterogeneous data such as log messages (logs), metrics (e.g., CPU utilization), and distributed traces. In this paper, we propose a method for reliable and practical anomaly detection from system logs. It overcomes the common disadvantage of related works, i.e., the need for a large amount of manually labeled training data, by building an anomaly detection model with log instructions from the source code of 1000+ GitHub projects. The instructions from diverse systems contain rich and heterogenous information about many different normal and abnormal IT events and serve as a foundation for anomaly detection. The proposed method, named ADLILog, combines the log instructions and the data from the system of interest (target system) to learn a deep neural network model through a two-phase learning procedure. The experimental results show that ADLILog outperforms the related approaches by up to 60% on the F1 score while satisfying core non-functional requirements for industrial deployments such as unsupervised design, efficient model updates, and small model sizes.