Feature Selection for Fault Detection and Prediction based on Event Log Analysis

TL;DR

This paper proposes a feature selection method for log-based anomaly detection in complex systems, aiming to improve detection efficiency and effectiveness by reducing the number of features used.

Contribution

It introduces a novel feature selection approach tailored for log analysis in complex systems, addressing the challenge of high-dimensional feature spaces.

Findings

Enhanced detection accuracy with fewer features

Reduced computational cost in anomaly detection

Applicable to large-scale complex systems

Abstract

Event logs are widely used for anomaly detection and prediction in complex systems. Existing log-based anomaly detection methods usually consist of four main steps: log collection, log parsing, feature extraction, and anomaly detection, wherein the feature extraction step extracts useful features for anomaly detection by counting log events. For a complex system, such as a lithography machine consisting of a large number of subsystems, its log may contain thousands of different events, resulting in abounding extracted features. However, when anomaly detection is performed at the subsystem level, analyzing all features becomes expensive and unnecessary. To mitigate this problem, we develop a feature selection method for log-based anomaly detection and prediction, largely improving the effectiveness and efficiency.