Sample Complexity Bounds for Linear Constrained MDPs with a Generative Model

TL;DR

This paper develops sample complexity bounds for solving linear constrained MDPs with a generative model, providing near-optimal guarantees for both relaxed and strict feasibility cases using a primal-dual approach.

Contribution

The paper introduces a primal-dual framework leveraging any black-box unconstrained MDP solver for linear CMDPs and derives near-optimal sample complexity bounds for different feasibility settings.

Findings

Sample complexity for relaxed feasibility: rac{d^2}{(1-\u03b3)^4\u03b5^2} samples.

Sample complexity for strict feasibility: rac{d^2}{(1-b3)^6b5^2b0^2} samples.

Lower bound matches upper bounds up to logarithmic factors.

Abstract

We consider infinite-horizon $\gamma$-discounted (linear) constrained Markov decision processes (CMDPs) where the objective is to find a policy that maximizes the expected cumulative reward subject to expected cumulative constraints. Given access to a generative model, we propose to solve CMDPs with a primal-dual framework that can leverage any black-box unconstrained MDP solver. For linear CMDPs with feature dimension $d$, we instantiate the framework by using mirror descent value iteration (\texttt{MDVI})~\citep{kitamura2023regularization} an example MDP solver. We provide sample complexity bounds for the resulting CMDP algorithm in two cases: (i) relaxed feasibility, where small constraint violations are allowed, and (ii) strict feasibility, where the output policy is required to exactly satisfy the constraint. For (i), we prove that the algorithm can return an $\epsilon$-optimal policy with high probability by using $\tilde{O}\left(\frac{d^2}{(1-\gamma)^4\epsilon^2}\right)$ samples. For (ii), we show that the algorithm requires $\tilde{O}\left(\frac{d^2}{(1-\gamma)^6\epsilon^2\zeta^2}\right)$ samples, where $\zeta$ is the problem-dependent Slater constant that characterizes the size of the feasible region. Furthermore, we prove a lower-bound of $\Omega\left(\frac{d^2}{(1-\gamma)^5\epsilon^2\zeta^2}\right)$ for the strict feasibility setting. We note that our upper bounds under both settings exhibit a near-optimal dependence on $d$, $\epsilon$, and $\zeta$. Finally, we instantiate our framework for tabular CMDPs and show that it can be used to recover near-optimal sample complexities in this setting.