A Variance Minimization Approach to Temporal-Difference Learning

TL;DR

This paper proposes a variance minimization approach for reinforcement learning, introducing new algorithms that focus on reducing variance to improve convergence speed and stability in value-based methods.

Contribution

It introduces a novel variance minimization framework with new objectives and algorithms, offering convergence proofs and demonstrating improved performance over traditional error minimization methods.

Findings

Proposed VBE and VPBE objectives for variance minimization.

Derived algorithms VMTD, VMTDC, and VMETD with proven convergence.

Experimental results show enhanced convergence and stability.

Abstract

Fast-converging algorithms are a contemporary requirement in reinforcement learning. In the context of linear function approximation, the magnitude of the smallest eigenvalue of the key matrix is a major factor reflecting the convergence speed. Traditional value-based RL algorithms focus on minimizing errors. This paper introduces a variance minimization (VM) approach for value-based RL instead of error minimization. Based on this approach, we proposed two objectives, the Variance of Bellman Error (VBE) and the Variance of Projected Bellman Error (VPBE), and derived the VMTD, VMTDC, and VMETD algorithms. We provided proofs of their convergence and optimal policy invariance of the variance minimization. Experimental studies validate the effectiveness of the proposed algorithms.