Momentum-based Accelerated Q-learning

TL;DR

This paper introduces a momentum-inspired acceleration scheme for Q-learning that improves convergence rates and performance in both discrete and continuous state-action spaces, validated through theoretical analysis and experiments.

Contribution

It proposes a novel acceleration method for Q-learning inspired by optimization momentum techniques, applicable to both finite and continuous spaces.

Findings

Accelerated Q-learning converges to the global optimum at a rate of O(1/√T).

The proposed method outperforms SpeedyQ in the FrozenLake game.

Accelerated algorithms improve convergence in LQR and Atari 2600 tasks.

Abstract

This paper studies accelerated algorithms for Q-learning. We propose an acceleration scheme by incorporating the historical iterates of the Q-function. The idea is conceptually inspired by the momentum-based acceleration methods in the optimization theory. Under finite state-action space settings, the proposed accelerated Q-learning algorithm provably converges to the global optimum with a rate of $\mathcal{O}(1/\sqrt{T})$. While sharing a comparable theoretic convergence rate with the existing Speedy Q-learning (SpeedyQ) algorithm, we numerically show that the proposed algorithm outperforms SpeedyQ via playing the FrozenLake grid world game. Furthermore, we generalize the acceleration scheme to the continuous state-action space case where function approximation of the Q-function is necessary. In this case, the algorithms are validated using commonly adopted testing problems in reinforcement learning, including two discrete-time linear quadratic regulation (LQR) problems from the Deepmind Control Suite, and the Atari 2600 games. Simulation results show that the proposed accelerated algorithms can improve the convergence performance compared with the vanilla Q-learning algorithm.