Dynamic Reward Scaling for Multivariate Time Series Anomaly Detection: A VAE-Enhanced Reinforcement Learning Approach

TL;DR

This paper introduces a novel deep reinforcement learning framework with dynamic reward scaling, combining VAE, LSTM-DQN, and active learning to improve multivariate time series anomaly detection, outperforming existing methods on benchmark datasets.

Contribution

The main contribution is the development of Dynamic Reward Scaling (DRSMT) that integrates VAE, reinforcement learning, and active learning for enhanced anomaly detection.

Findings

Outperforms existing baselines in F1-score and AU-PR on SMD and WADI datasets.

Effectively balances exploration and exploitation through dynamic reward shaping.

Reduces manual labeling needs via active learning for uncertain samples.

Abstract

Detecting anomalies in multivariate time series is essential for monitoring complex industrial systems, where high dimensionality, limited labeled data, and subtle dependencies between sensors cause significant challenges. This paper presents a deep reinforcement learning framework that combines a Variational Autoencoder (VAE), an LSTM-based Deep Q-Network (DQN), dynamic reward shaping, and an active learning module to address these issues in a unified learning framework. The main contribution is the implementation of Dynamic Reward Scaling for Multivariate Time Series Anomaly Detection (DRSMT), which demonstrates how each component enhances the detection process. The VAE captures compact latent representations and reduces noise. The DQN enables adaptive, sequential anomaly classification, and the dynamic reward shaping balances exploration and exploitation during training by adjusting the importance of reconstruction and classification signals. In addition, active learning identifies the most uncertain samples for labeling, reducing the need for extensive manual supervision. Experiments on two multivariate benchmarks, namely Server Machine Dataset (SMD) and Water Distribution Testbed (WADI), show that the proposed method outperforms existing baselines in F1-score and AU-PR. These results highlight the effectiveness of combining generative modeling, reinforcement learning, and selective supervision for accurate and scalable anomaly detection in real-world multivariate systems.