Analysis of Off-Policy $n$-Step TD-Learning with Linear Function Approximation

TL;DR

This paper investigates the convergence properties of off-policy n-step TD-learning algorithms with linear function approximation, demonstrating convergence as the sampling horizon increases, and bridges model-based and model-free RL methods.

Contribution

It provides a theoretical analysis showing convergence of n-step TD algorithms in off-policy settings as n grows large, linking deterministic and stochastic approaches.

Findings

n-step TD algorithms converge with sufficiently large n

Analysis connects model-based and model-free RL methods

Provides theoretical guarantees for off-policy TD learning

Abstract

This paper analyzes multi-step temporal difference (TD)-learning algorithms within the ``deadly triad'' scenario, characterized by linear function approximation, off-policy learning, and bootstrapping. In particular, we prove that $n$-step TD-learning algorithms converge to a solution as the sampling horizon $n$ increases sufficiently. The paper is divided into two parts. In the first part, we comprehensively examine the fundamental properties of their model-based deterministic counterparts, including projected value iteration, gradient descent algorithms, which can be viewed as prototype deterministic algorithms whose analysis plays a pivotal role in understanding and developing their model-free reinforcement learning counterparts. In particular, we prove that these algorithms converge to meaningful solutions when $n$ is sufficiently large. Based on these findings, in the second part, two $n$-step TD-learning algorithms are proposed and analyzed, which can be seen as the model-free reinforcement learning counterparts of the model-based deterministic algorithms.