On the Sample Complexity of Discounted Reinforcement Learning with Optimized Certainty Equivalents

TL;DR

This paper analyzes the sample complexity of risk-sensitive reinforcement learning in finite discounted MDPs using optimized certainty equivalents, providing bounds and characterizations for PAC-learnability.

Contribution

It offers an exact characterization of utility functions for PAC-learnability and derives tight PAC sample complexity bounds for value and policy learning under OCE risk measures.

Findings

PAC-learnability depends on the domain of utility functions.

Derived PAC sample complexity bounds for value and policy learning.

Established lower bounds demonstrating tightness and dependence on horizon and risk parameters.

Abstract

We study risk-sensitive reinforcement learning in finite discounted MDPs, where a generative model of the MDP is assumed to be available. We consider a family or risk measures called the optimized certainty equivalent (OCE), which includes important risk measures such as entropic risk, CVaR, and mean-variance. Our focus is on the sample complexities of learning the optimal state-action value function (value learning) and an optimal policy (policy learning) under recursive OCE. We provide an exact characterization of utility functions $u$ for which the corresponding OCE defines an objective that is PAC-learnable. We analyze a simple model-based approach and derive PAC sample complexity bounds. We establish that whenever $u$ does not have full domain $\text{dom}(u)\neq \mathbb{R}$, the corresponding problem is not PAC-learnable. Finally, we establish corresponding lower bounds for both value and policy learning, demonstrating tightness in the size $SA$ of state-action space, and for a more restricted class of utilities, we derive lower bounds that makes the dependence on the effective horizon $\frac{1}{1-\gamma}$ explicit. Specifically, for $\text{CVaR}_\tau$ we show that the correct dependence on $\tau$ is $\frac{1}{\tau^2}$, thus improving by a factor of $\frac{1}{\tau}$ over state-of-the-art although our bound has a suboptimal dependence on $\frac{1}{1-\gamma}$.