OMLog: Online Log Anomaly Detection for Evolving System with Meta-learning

TL;DR

OMLog introduces a semi-supervised online meta-learning approach for real-time log anomaly detection, effectively handling distribution shifts in evolving systems and outperforming state-of-the-art methods on public datasets.

Contribution

The paper proposes OMLog, a novel online meta-learning method with distribution shift detection for improved real-time log anomaly detection in dynamic systems.

Findings

Achieves F1-Score of 93.7% on HDFS dataset

Surpasses state-of-the-art LAD methods in detection efficiency

Effectively detects distribution changes in log sequences

Abstract

Log anomaly detection (LAD) is essential to ensure safe and stable operation of software systems. Although current LAD methods exhibit significant potential in addressing challenges posed by unstable log events and temporal sequence patterns, their limitations in detection efficiency and generalization ability present a formidable challenge when dealing with evolving systems. To construct a real-time and reliable online log anomaly detection model, we propose OMLog, a semi-supervised online meta-learning method, to effectively tackle the distribution shift issue caused by changes in log event types and frequencies. Specifically, we introduce a maximum mean discrepancy-based distribution shift detection method to identify distribution changes in unseen log sequences. Depending on the identified distribution gap, the method can automatically trigger online fine-grained detection or offline fast inference. Furthermore, we design an online learning mechanism based on meta-learning, which can effectively learn the highly repetitive patterns of log sequences in the feature space, thereby enhancing the generalization ability of the model to evolving data. Extensive experiments conducted on two publicly available log datasets, HDFS and BGL, validate the effectiveness of the OMLog approach. When trained using only normal log sequences, the proposed approach achieves the F1-Score of 93.7\% and 64.9\%, respectively, surpassing the performance of the state-of-the-art (SOTA) LAD methods and demonstrating superior detection efficiency.