The Role of Lookahead and Approximate Policy Evaluation in Reinforcement Learning with Linear Value Function Approximation

TL;DR

This paper analyzes how lookahead and m-step rollout techniques influence the stability and convergence of approximate dynamic programming in reinforcement learning with linear function approximation, providing quantitative insights.

Contribution

It provides the first quantitative characterization of the effects of lookahead and rollout on approximate DP with function approximation, including convergence and performance impacts.

Findings

Without sufficient lookahead and rollout, approximate DP may not converge.

Lookahead and rollout improve convergence rates of approximate DP.

Lookahead mitigates the effects of function approximation errors and discount factors.

Abstract

Function approximation is widely used in reinforcement learning to handle the computational difficulties associated with very large state spaces. However, function approximation introduces errors which may lead to instabilities when using approximate dynamic programming techniques to obtain the optimal policy. Therefore, techniques such as lookahead for policy improvement and m-step rollout for policy evaluation are used in practice to improve the performance of approximate dynamic programming with function approximation. We quantitatively characterize, for the first time, the impact of lookahead and m-step rollout on the performance of approximate dynamic programming (DP) with function approximation: (i) without a sufficient combination of lookahead and m-step rollout, approximate DP may not converge, (ii) both lookahead and m-step rollout improve the convergence rate of approximate DP, and (iii) lookahead helps mitigate the effect of function approximation and the discount factor on the asymptotic performance of the algorithm. Our results are presented for two approximate DP methods: one which uses least-squares regression to perform function approximation and another which performs several steps of gradient descent of the least-squares objective in each iteration.