Deep Learning Approach to Anomaly Detection in Enterprise ETL Processes with Autoencoders

TL;DR

This paper introduces a deep autoencoder-based anomaly detection method tailored for enterprise ETL processes, effectively identifying various anomalies and enhancing data processing reliability.

Contribution

It presents a novel deep autoencoder approach with regularization for robust anomaly detection in complex enterprise ETL data streams.

Findings

Achieves superior AUC, ACC, Precision, and Recall performance.

Effectively captures latent distribution patterns in ETL data.

Accurately identifies diverse anomalies in enterprise data streams.

Abstract

An anomaly detection method based on deep autoencoders is proposed to address anomalies that often occur in enterprise-level ETL data streams. The study first analyzes multiple types of anomalies in ETL processes, including delays, missing values, duplicate loading, and sudden abnormal changes, and applies data standardization and feature modeling to ensure stable and usable inputs. In the method design, the encoder-decoder structure compresses high-dimensional inputs into latent representations and reconstructs them, while reconstruction error is used to measure anomaly levels. Regularization constraints are introduced in the latent space to enhance feature sparsity and distribution learning, thereby improving robustness in complex data streams. Systematic analyses under different hyperparameter settings, environmental changes, and data characteristics show that the proposed method achieves superior performance in AUC, ACC, Precision, and Recall. The results demonstrate that the deep autoencoder-based detection mechanism can effectively capture latent distribution patterns in enterprise-level ETL data streams and accurately identify diverse anomalies, providing reliable support for enterprise data processing and intelligent analysis.