PhiBE: A PDE-based Bellman Equation for Continuous Time Policy Evaluation

TL;DR

This paper introduces PhiBE, a PDE-based Bellman equation for continuous-time policy evaluation in RL, offering more accurate value function approximation and improved sample complexity by leveraging the dynamics' smoothness.

Contribution

The paper develops PhiBE, a novel PDE-based Bellman equation for continuous-time RL, with theoretical guarantees and a model-free algorithm that improves sample complexity and handles model misspecification.

Findings

PhiBE provides a more accurate approximation than traditional discrete Bellman equations.

The model-free algorithm for PhiBE converges with finite-sample guarantees.

Sample complexity is improved to O(Δt^{-1}) by exploiting dynamics' smoothness.

Abstract

In this paper, we study policy evaluation in continuous-time reinforcement learning (RL), where the state follows an unknown stochastic differential equation (SDE), but only discrete-time data are available. We first highlight that the discrete-time Bellman equation (BE) is not always a reliable approximation to the true value function because it ignores the underlying continuous-time structure. We then introduce a new Bellman equation, PhiBE, which integrates the discrete-time information into a continuous-time PDE formulation. By leveraging the smooth structure of the underlying dynamics, PhiBE provides a provably more accurate approximation to the true value function, especially in scenarios where the underlying dynamics change slowly or the reward oscillates. Moreover, we extend PhiBE to higher orders, providing increasingly accurate approximations. We further develop a model-free algorithm for PhiBE under linear function approximation and establish its convergence under model misspecification, together with finite-sample guarantees. In contrast to existing continuous-time RL analyses, where the model misspecification error diverges as the sampling interval $\Delta t\to 0$ and the sample complexity typically scales as $O(\Delta t^{-4})$, our misspecification error is independent of $\Delta t$ and the resulting sample complexity improves to $O(\Delta t^{-1})$ by exploiting the smoothness of the underlying dynamics. Moreover, we identify a fundamental trade-off between discretization error and sample error that is intrinsic to continuous-time policy evaluation: finer time discretization reduces bias but amplifies variance, so excessively frequent sampling does not necessarily improve performance. This is an insight that does not arise in classical discrete-time RL analyses. Numerical experiments are provided to validate the theoretical guarantees we propose.