Bayesian Counterfactual Risk Minimization

TL;DR

This paper introduces a Bayesian perspective on counterfactual risk minimization, deriving a new generalization bound and proposing a data-dependent regularization method that improves offline learning from logged bandit feedback.

Contribution

It presents a PAC-Bayesian analysis for CRM, leading to a novel regularization technique that outperforms standard methods in offline bandit learning.

Findings

The new regularization technique outperforms standard $L_2$ regularization.

It is competitive with variance regularization in performance.

The method is simpler to implement and more computationally efficient.

Abstract

We present a Bayesian view of counterfactual risk minimization (CRM) for offline learning from logged bandit feedback. Using PAC-Bayesian analysis, we derive a new generalization bound for the truncated inverse propensity score estimator. We apply the bound to a class of Bayesian policies, which motivates a novel, potentially data-dependent, regularization technique for CRM. Experimental results indicate that this technique outperforms standard $L_2$ regularization, and that it is competitive with variance regularization while being both simpler to implement and more computationally efficient.