GB-DQN: Gradient Boosted DQN Models for Non-stationary Reinforcement Learning

TL;DR

GB-DQN introduces an adaptive ensemble approach using gradient boosting to effectively handle non-stationary environments in reinforcement learning, leading to faster adaptation and increased robustness.

Contribution

The paper presents GB-DQN, a novel ensemble method that incrementally learns residuals to adapt to environment changes, outperforming traditional DQN in non-stationary settings.

Findings

Faster recovery from environment changes.

Enhanced stability and robustness.

Theoretical proof of residual reduction and convergence.

Abstract

Non-stationary environments pose a fundamental challenge for deep reinforcement learning, as changes in dynamics or rewards invalidate learned value functions and cause catastrophic forgetting. We propose \emph{Gradient-Boosted Deep Q-Networks (GB-DQN)}, an adaptive ensemble method that addresses model drift through incremental residual learning. Instead of retraining a single Q-network, GB-DQN constructs an additive ensemble in which each new learner is trained to approximate the Bellman residual of the current ensemble after drift. We provide theoretical results showing that each boosting step reduces the empirical Bellman residual and that the ensemble converges to the post-drift optimal value function under standard assumptions. Experiments across a diverse set of control tasks with controlled dynamics changes demonstrate faster recovery, improved stability, and greater robustness compared to DQN and common non-stationary baselines.