Efficiently Solving Discounted MDPs with Predictions on Transition Matrices

TL;DR

This paper explores how predictions of transition matrices can improve the sample efficiency of solving discounted Markov Decision Processes, providing new algorithms and bounds that outperform previous methods.

Contribution

It introduces a novel framework leveraging transition matrix predictions to enhance sample complexity bounds in DMDPs, with theoretical analysis and improved algorithms.

Findings

Impossibility result without prior prediction accuracy knowledge

Proposed algorithm achieves better sample complexity bounds

Numerical experiments support theoretical improvements

Abstract

We study infinite-horizon Discounted Markov Decision Processes (DMDPs) under a generative model. Motivated by the Algorithm with Advice framework Mitzenmacher and Vassilvitskii 2022, we propose a novel framework to investigate how a prediction on the transition matrix can enhance the sample efficiency in solving DMDPs and improve sample complexity bounds. We focus on the DMDPs with $N$ state-action pairs and discounted factor $\gamma$. Firstly, we provide an impossibility result that, without prior knowledge of the prediction accuracy, no sampling policy can compute an $\epsilon$-optimal policy with a sample complexity bound better than $\tilde{O}((1-\gamma)^{-3} N\epsilon^{-2})$, which matches the state-of-the-art minimax sample complexity bound with no prediction. In complement, we propose an algorithm based on minimax optimization techniques that leverages the prediction on the transition matrix. Our algorithm achieves a sample complexity bound depending on the prediction error, and the bound is uniformly better than $\tilde{O}((1-\gamma)^{-4} N \epsilon^{-2})$, the previous best result derived from convex optimization methods. These theoretical findings are further supported by our numerical experiments.