Off-Policy Risk Assessment in Markov Decision Processes

TL;DR

This paper extends off-policy risk assessment methods from contextual bandits to Markov decision processes, introducing a doubly robust estimator that reduces variance and improves accuracy in estimating return distributions.

Contribution

It develops the first doubly robust CDF estimator for MDPs, providing theoretical guarantees and practical improvements over importance sampling methods.

Findings

Doubly robust estimator significantly reduces variance.

Estimator achieves Cramer-Rao lower bound with well-specified models.

Experimental results confirm high precision of the proposed method.

Abstract

Addressing such diverse ends as safety alignment with human preferences, and the efficiency of learning, a growing line of reinforcement learning research focuses on risk functionals that depend on the entire distribution of returns. Recent work on \emph{off-policy risk assessment} (OPRA) for contextual bandits introduced consistent estimators for the target policy's CDF of returns along with finite sample guarantees that extend to (and hold simultaneously over) all risk. In this paper, we lift OPRA to Markov decision processes (MDPs), where importance sampling (IS) CDF estimators suffer high variance on longer trajectories due to small effective sample size. To mitigate these problems, we incorporate model-based estimation to develop the first doubly robust (DR) estimator for the CDF of returns in MDPs. This estimator enjoys significantly less variance and, when the model is well specified, achieves the Cramer-Rao variance lower bound. Moreover, for many risk functionals, the downstream estimates enjoy both lower bias and lower variance. Additionally, we derive the first minimax lower bounds for off-policy CDF and risk estimation, which match our error bounds up to a constant factor. Finally, we demonstrate the precision of our DR CDF estimates experimentally on several different environments.