Provably Efficient RL under Episode-Wise Safety in Constrained MDPs with Linear Function Approximation

TL;DR

This paper introduces a computationally efficient RL algorithm for linear constrained MDPs that guarantees zero constraint violations per episode and achieves near-optimal regret bounds, advancing theoretical understanding in function approximation settings.

Contribution

It presents the first RL algorithm for linear CMDPs with both zero episode-wise constraint violation and $ ilde{O}( oot{K})$ regret, scalable with problem parameters.

Findings

Achieves $ ilde{O}( oot{K})$ regret in linear CMDPs.

Guarantees zero episode-wise constraint violation.

Scales polynomially with problem parameters, independent of state space size.

Abstract

We study the reinforcement learning (RL) problem in a constrained Markov decision process (CMDP), where an agent explores the environment to maximize the expected cumulative reward while satisfying a single constraint on the expected total utility value in every episode. While this problem is well understood in the tabular setting, theoretical results for function approximation remain scarce. This paper closes the gap by proposing an RL algorithm for linear CMDPs that achieves $\tilde{\mathcal{O}}(\sqrt{K})$ regret with an episode-wise zero-violation guarantee. Furthermore, our method is computationally efficient, scaling polynomially with problem-dependent parameters while remaining independent of the state space size. Our results significantly improve upon recent linear CMDP algorithms, which either violate the constraint or incur exponential computational costs.