Enhancing Q-Value Updates in Deep Q-Learning via Successor-State Prediction

TL;DR

This paper introduces SADQ, a novel deep Q-learning method that models environment dynamics to improve value update stability and efficiency, outperforming traditional DQNs in various benchmarks.

Contribution

SADQ explicitly incorporates successor-state distributions into Q-value updates, reducing variance and enhancing learning stability compared to existing methods.

Findings

SADQ achieves higher stability in training.

SADQ outperforms DQN variants in benchmark tasks.

SADQ demonstrates improved learning efficiency.

Abstract

Deep Q-Networks (DQNs) estimate future returns by learning from transitions sampled from a replay buffer. However, the target updates in DQN often rely on next states generated by actions from past, potentially suboptimal, policy. As a result, these states may not provide informative learning signals, causing high variance into the update process. This issue is exacerbated when the sampled transitions are poorly aligned with the agent's current policy. To address this limitation, we propose the Successor-state Aggregation Deep Q-Network (SADQ), which explicitly models environment dynamics using a stochastic transition model. SADQ integrates successor-state distributions into the Q-value estimation process, enabling more stable and policy-aligned value updates. Additionally, it explores a more efficient action selection strategy with the modeled transition structure. We provide theoretical guarantees that SADQ maintains unbiased value estimates while reducing training variance. Our extensive empirical results across standard RL benchmarks and real-world vector-based control tasks demonstrate that SADQ consistently outperforms DQN variants in both stability and learning efficiency.