Pessimistic Risk-Aware Policy Learning in Contextual Bandits

TL;DR

This paper introduces a unified framework for risk-aware offline policy learning in contextual bandits, enabling the optimization of various risk measures with optimal statistical guarantees.

Contribution

It develops a distributional approach for optimizing Lipschitz-continuous risk functionals, providing minimax optimal bounds without restrictive assumptions.

Findings

Achieves an $ ilde{ ext{O}}(1/ ext{sqrt}(n))$ convergence rate for risk functional optimization.

Provides data-dependent suboptimality bounds for importance sampling estimators.

Shows no additional statistical cost compared to risk-neutral policy optimization.

Abstract

We study risk-aware offline policy learning, aiming to learn a decision rule from logged data that is optimal under general risk criteria. This problem is crucial in high-stakes domains where online interaction is infeasible and adverse outcomes must be carefully controlled. However, existing literature on offline contextual bandits either centers on expected-reward criteria or restricts risk considerations to policy evaluation instead of optimization. In this work, we propose a unified distributional framework for optimizing Lipschitz-continuous risk functionals, a broad class of risk measures encompassing mean-variance, entropic risk, and conditional value-at-risk, among others. By developing novel empirical concentration inequalities for importance sampling-based distributional estimators, our analysis derives data-dependent suboptimality bounds with an $\tilde{\mathcal{O}}(1/\sqrt{n})$ rate, without relying on restrictive uniform overlap assumptions. This rate is minimax optimal and matches that of risk-neutral offline policy optimization, indicating that optimizing general Lipschitz risk criteria incurs no additional statistical cost relative to the expected-reward.