Comparing AutoML and Deep Learning Methods for Condition Monitoring using Realistic Validation Scenarios

TL;DR

This paper compares traditional machine learning and deep learning methods for condition monitoring, highlighting their performance, interpretability, and feature selection needs in realistic validation scenarios with domain shifts.

Contribution

It provides a comprehensive benchmark of AutoML, deep learning, and conventional methods under realistic validation, emphasizing the importance of feature selection and interpretability.

Findings

High accuracy in cross-validation scenarios

No clear winner in leave-one-group-out validation

Low-complexity models often suffice for limited class variations

Abstract

This study extensively compares conventional machine learning methods and deep learning for condition monitoring tasks using an AutoML toolbox. The experiments reveal consistent high accuracy in random K-fold cross-validation scenarios across all tested models. However, when employing leave-one-group-out (LOGO) cross-validation on the same datasets, no clear winner emerges, indicating the presence of domain shift in real-world scenarios. Additionally, the study assesses the scalability and interpretability of conventional methods and neural networks. Conventional methods offer explainability with their modular structure aiding feature identification. In contrast, neural networks require specialized interpretation techniques like occlusion maps to visualize important regions in the input data. Finally, the paper highlights the significance of feature selection, particularly in condition monitoring tasks with limited class variations. Low-complexity models prove sufficient for such tasks, as only a few features from the input signal are typically needed. In summary, these findings offer crucial insights into the strengths and limitations of various approaches, providing valuable benchmarks and identifying the most suitable methods for condition monitoring applications, thereby enhancing their applicability in real-world scenarios.