Accelerated and instance-optimal policy evaluation with linear function approximation

TL;DR

This paper introduces an accelerated, variance-reduced algorithm for policy evaluation with linear function approximation that achieves optimal error bounds and instance-optimality, improving upon existing methods.

Contribution

The paper develops a new accelerated, variance-reduced temporal difference algorithm that matches fundamental lower bounds and extends to Markovian observation settings.

Findings

The proposed VRFTD algorithm achieves optimal deterministic and stochastic error bounds.

Existing algorithms like variance-reduced TD fail to reach the lower bounds.

Numerical experiments confirm the theoretical optimality of VRFTD.

Abstract

We study the problem of policy evaluation with linear function approximation and present efficient and practical algorithms that come with strong optimality guarantees. We begin by proving lower bounds that establish baselines on both the deterministic error and stochastic error in this problem. In particular, we prove an oracle complexity lower bound on the deterministic error in an instance-dependent norm associated with the stationary distribution of the transition kernel, and use the local asymptotic minimax machinery to prove an instance-dependent lower bound on the stochastic error in the i.i.d. observation model. Existing algorithms fail to match at least one of these lower bounds: To illustrate, we analyze a variance-reduced variant of temporal difference learning, showing in particular that it fails to achieve the oracle complexity lower bound. To remedy this issue, we develop an accelerated, variance-reduced fast temporal difference algorithm (VRFTD) that simultaneously matches both lower bounds and attains a strong notion of instance-optimality. Finally, we extend the VRFTD algorithm to the setting with Markovian observations, and provide instance-dependent convergence results. Our theoretical guarantees of optimality are corroborated by numerical experiments.