A General Framework for Off-Policy Learning with Partially-Observed Reward

TL;DR

This paper introduces HyPeR, a novel framework for off-policy learning that effectively leverages secondary rewards to improve policy optimization when target rewards are only partially observed, with strong theoretical and empirical support.

Contribution

The work proposes a new method, HyPeR, that utilizes secondary rewards to enhance off-policy learning under partial target reward observation, and provides theoretical analysis and empirical validation.

Findings

HyPeR outperforms existing methods in synthetic data scenarios.

Leveraging secondary rewards improves target reward optimization.

Jointly optimizing target and secondary rewards can be beneficial.

Abstract

Off-policy learning (OPL) in contextual bandits aims to learn a decision-making policy that maximizes the target rewards by using only historical interaction data collected under previously developed policies. Unfortunately, when rewards are only partially observed, the effectiveness of OPL degrades severely. Well-known examples of such partial rewards include explicit ratings in content recommendations, conversion signals on e-commerce platforms that are partial due to delay, and the issue of censoring in medical problems. One possible solution to deal with such partial rewards is to use secondary rewards, such as dwelling time, clicks, and medical indicators, which are more densely observed. However, relying solely on such secondary rewards can also lead to poor policy learning since they may not align with the target reward. Thus, this work studies a new and general problem of OPL where the goal is to learn a policy that maximizes the expected target reward by leveraging densely observed secondary rewards as supplemental data. We then propose a new method called Hybrid Policy Optimization for Partially-Observed Reward (HyPeR), which effectively uses the secondary rewards in addition to the partially-observed target reward to achieve effective OPL despite the challenging scenario. We also discuss a case where we aim to optimize not only the expected target reward but also the expected secondary rewards to some extent; counter-intuitively, we will show that leveraging the two objectives is in fact advantageous also for the optimization of only the target reward. Along with statistical analysis of our proposed methods, empirical evaluations on both synthetic and real-world data show that HyPeR outperforms existing methods in various scenarios.