Improving Offline Contextual Bandits with Distributional Robustness

TL;DR

This paper introduces a distributionally robust optimization framework for offline contextual bandits, providing a convex reformulation that improves confidence interval calibration and is suitable for large-scale data, with promising preliminary results.

Contribution

It develops a convex reformulation of the Counterfactual Risk Minimization principle using DRO, enabling scalable and hyper-parameter free policy optimization.

Findings

Effective confidence interval calibration for offline bandits

Compatibility with stochastic optimization for large datasets

Preliminary empirical results show promising performance

Abstract

This paper extends the Distributionally Robust Optimization (DRO) approach for offline contextual bandits. Specifically, we leverage this framework to introduce a convex reformulation of the Counterfactual Risk Minimization principle. Besides relying on convex programs, our approach is compatible with stochastic optimization, and can therefore be readily adapted tothe large data regime. Our approach relies on the construction of asymptotic confidence intervals for offline contextual bandits through the DRO framework. By leveraging known asymptotic results of robust estimators, we also show how to automatically calibrate such confidence intervals, which in turn removes the burden of hyper-parameter selection for policy optimization. We present preliminary empirical results supporting the effectiveness of our approach.