Percentile Criterion Optimization in Offline Reinforcement Learning

TL;DR

This paper introduces a new dynamic programming algorithm for offline reinforcement learning that optimizes the percentile criterion more efficiently by implicitly constructing smaller ambiguity sets, leading to less conservative policies.

Contribution

A novel Value-at-Risk based dynamic programming method that avoids explicit ambiguity set construction in percentile criterion optimization.

Findings

Implicitly constructs smaller ambiguity sets

Learns less conservative robust policies

Outperforms existing Bayesian credible region methods

Abstract

In reinforcement learning, robust policies for high-stakes decision-making problems with limited data are usually computed by optimizing the \emph{percentile criterion}. The percentile criterion is approximately solved by constructing an \emph{ambiguity set} that contains the true model with high probability and optimizing the policy for the worst model in the set. Since the percentile criterion is non-convex, constructing ambiguity sets is often challenging. Existing work uses \emph{Bayesian credible regions} as ambiguity sets, but they are often unnecessarily large and result in learning overly conservative policies. To overcome these shortcomings, we propose a novel Value-at-Risk based dynamic programming algorithm to optimize the percentile criterion without explicitly constructing any ambiguity sets. Our theoretical and empirical results show that our algorithm implicitly constructs much smaller ambiguity sets and learns less conservative robust policies.