Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation

TL;DR

This paper introduces a theoretically grounded, sample-efficient, and computationally practical continuous-time reinforcement learning algorithm that leverages general function approximation, optimism-based confidence sets, and structured policy updates, with empirical validation on control tasks.

Contribution

It provides the first sample complexity guarantee for CTRL with general function approximation and proposes structured policy updates to enhance efficiency.

Findings

Achieves near-optimal policy with suboptimality gap of rac{}{}(+d_{}+d_{})N^{-1/2}

Reduces number of policy updates and rollouts while maintaining performance

Demonstrates competitive results on continuous control and diffusion model tasks

Abstract

Continuous-time reinforcement learning (CTRL) provides a principled framework for sequential decision-making in environments where interactions evolve continuously over time. Despite its empirical success, the theoretical understanding of CTRL remains limited, especially in settings with general function approximation. In this work, we propose a model-based CTRL algorithm that achieves both sample and computational efficiency. Our approach leverages optimism-based confidence sets to establish the first sample complexity guarantee for CTRL with general function approximation, showing that a near-optimal policy can be learned with a suboptimality gap of $\tilde{O}(\sqrt{d_{\mathcal{R}} + d_{\mathcal{F}}}N^{-1/2})$ using $N$ measurements, where $d_{\mathcal{R}}$ and $d_{\mathcal{F}}$ denote the distributional Eluder dimensions of the reward and dynamic functions, respectively, capturing the complexity of general function approximation in reinforcement learning. Moreover, we introduce structured policy updates and an alternative measurement strategy that significantly reduce the number of policy updates and rollouts while maintaining competitive sample efficiency. We implemented experiments to backup our proposed algorithms on continuous control tasks and diffusion model fine-tuning, demonstrating comparable performance with significantly fewer policy updates and rollouts.