Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach

TL;DR

This paper introduces a martingale-based framework for policy evaluation and TD methods in continuous time and space, revealing new insights and algorithms with proven convergence properties.

Contribution

It develops a unified martingale approach to policy evaluation, connecting classical TD algorithms to continuous-time martingale conditions and providing new algorithms with convergence guarantees.

Findings

Martingale characterization of policy evaluation in continuous settings

New algorithms based on martingale loss and orthogonality conditions

Convergence of discretized algorithms to continuous-time solutions

Abstract

We propose a unified framework to study policy evaluation (PE) and the associated temporal difference (TD) methods for reinforcement learning in continuous time and space. We show that PE is equivalent to maintaining the martingale condition of a process. From this perspective, we find that the mean--square TD error approximates the quadratic variation of the martingale and thus is not a suitable objective for PE. We present two methods to use the martingale characterization for designing PE algorithms. The first one minimizes a "martingale loss function", whose solution is proved to be the best approximation of the true value function in the mean--square sense. This method interprets the classical gradient Monte-Carlo algorithm. The second method is based on a system of equations called the "martingale orthogonality conditions" with test functions. Solving these equations in different ways recovers various classical TD algorithms, such as TD($\lambda$), LSTD, and GTD. Different choices of test functions determine in what sense the resulting solutions approximate the true value function. Moreover, we prove that any convergent time-discretized algorithm converges to its continuous-time counterpart as the mesh size goes to zero, and we provide the convergence rate. We demonstrate the theoretical results and corresponding algorithms with numerical experiments and applications.