Stochastic Variance Reduction Methods for Policy Evaluation

TL;DR

This paper introduces stochastic variance reduction algorithms for efficient policy evaluation in reinforcement learning, transforming the problem into a saddle point formulation and demonstrating linear convergence and scalability.

Contribution

It presents novel stochastic variance reduction methods for policy evaluation, achieving linear convergence without requiring strong convexity in primal variables.

Findings

Algorithms scale linearly with sample size and feature dimension

Methods achieve linear convergence under weak convexity assumptions

Numerical experiments confirm effectiveness on benchmark problems

Abstract

Policy evaluation is a crucial step in many reinforcement-learning procedures, which estimates a value function that predicts states' long-term value under a given policy. In this paper, we focus on policy evaluation with linear function approximation over a fixed dataset. We first transform the empirical policy evaluation problem into a (quadratic) convex-concave saddle point problem, and then present a primal-dual batch gradient method, as well as two stochastic variance reduction methods for solving the problem. These algorithms scale linearly in both sample size and feature dimension. Moreover, they achieve linear convergence even when the saddle-point problem has only strong concavity in the dual variables but no strong convexity in the primal variables. Numerical experiments on benchmark problems demonstrate the effectiveness of our methods.