Double Q($\sigma$) and Q($\sigma, \lambda$): Unifying Reinforcement Learning Control Algorithms

TL;DR

This paper introduces the Q(σ, λ) algorithm, an extension of Q(σ) with eligibility traces, and Double Q(σ), unifying and improving reinforcement learning control methods with experimental validation.

Contribution

It presents the novel Q(σ, λ) algorithm with eligibility traces and Double Q(σ), unifying and extending existing TD control algorithms.

Findings

Q(σ, λ) outperforms classical TD methods in experiments.

Double Q(σ) reduces overestimation bias in Q-learning.

The unified framework simplifies understanding of TD algorithms.

Abstract

Temporal-difference (TD) learning is an important field in reinforcement learning. Sarsa and Q-Learning are among the most used TD algorithms. The Q($\sigma$) algorithm (Sutton and Barto (2017)) unifies both. This paper extends the Q($\sigma$) algorithm to an online multi-step algorithm Q($\sigma, \lambda$) using eligibility traces and introduces Double Q($\sigma$) as the extension of Q($\sigma$) to double learning. Experiments suggest that the new Q($\sigma, \lambda$) algorithm can outperform the classical TD control methods Sarsa($\lambda$), Q($\lambda$) and Q($\sigma$).