Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes

TL;DR

This paper introduces an optimistic actor-critic algorithm with parametric policies for linear MDPs, achieving state-of-the-art sample complexity while being more practical than previous methods.

Contribution

It proposes a tractable logit-matching actor and uses Langevin Monte Carlo for the critic, enabling efficient exploration in linear MDPs.

Findings

Achieves $ ilde{O}( ext{epsilon}^{-4})$ sample complexity on-policy.

Achieves $ ilde{O}( ext{epsilon}^{-2})$ sample complexity off-policy.

Aligns theoretical guarantees with practical algorithm design.

Abstract

Although actor-critic methods have been successful in practice, their theoretical analyses have several limitations. Specifically, existing theoretical work either sidesteps the exploration problem by making strong assumptions or analyzes impractical methods with complicated algorithmic modifications. Moreover, the actor-critic methods analyzed for linear MDPs often employ natural policy gradient and construct "implicit" policies without explicit parameterization. Such policies are computationally expensive to sample from, making the environment interactions inefficient. To that end, we focus on the finite-horizon linear MDPs and propose an optimistic actor-critic framework that uses parametric log-linear policies. In particular, we introduce a tractable $\textit{logit-matching}$ regression objective for the actor. For the critic, we use approximate Thompson sampling via Langevin Monte Carlo to obtain optimistic value estimates. We prove that the resulting algorithm achieves $\widetilde{\mathcal{O}}(\epsilon^{-4})$ and $\widetilde{\mathcal{O}}(\epsilon^{-2})$ sample complexity in the on-policy and off-policy setting, respectively. Our results match prior theoretical work in achieving the state-of-the-art sample complexity, while our algorithm is more aligned with practice.