Policy Mirror Descent with Temporal Difference Learning: Sample Complexity under Online Markov Data

TL;DR

This paper analyzes the sample complexity of policy mirror descent with temporal difference learning in reinforcement learning under Markovian data, introducing algorithms with provable efficiency guarantees.

Contribution

It introduces Expected TD-PMD and Approximate TD-PMD algorithms and establishes their sample complexity bounds under Markovian sampling.

Findings

Expected TD-PMD and Approximate TD-PMD achieve $ ilde{O}( ext{} ext{varepsilon}^{-2})$ sample complexity.

Adaptive step sizes improve sample complexity to $O( ext{varepsilon}^{-2})$ for last-iterate optimality.

The analysis extends understanding of policy mirror descent with TD learning in Markovian environments.

Abstract

This paper studies the policy mirror descent (PMD) method, which is a general policy optimization framework in reinforcement learning and can cover a wide range of policy gradient methods by specifying difference mirror maps. Existing sample complexity analysis for policy mirror descent either focuses on the generative sampling model, or the Markovian sampling model but with the action values being explicitly approximated to certain pre-specified accuracy. In contrast, we consider the sample complexity of policy mirror descent with temporal difference (TD) learning under the Markovian sampling model. Two algorithms called Expected TD-PMD and Approximate TD-PMD have been presented, which are off-policy and mixed policy algorithms respectively. Under a small enough constant policy update step size, the $\tilde{O}(\varepsilon^{-2})$ (a logarithm factor about $\varepsilon$ is hidden in $\tilde{O}(\cdot)$) sample complexity can be established for them to achieve average-time $\varepsilon$-optimality. The sample complexity is further improved to $O(\varepsilon^{-2})$ (without the hidden logarithm factor) to achieve the last-iterate $\varepsilon$-optimality based on adaptive policy update step sizes.