DOLCE: Decomposing Off-Policy Evaluation/Learning into Lagged and Current Effects

TL;DR

DOLCE is a novel method for off-policy evaluation and learning that effectively handles support violations by decomposing effects into lagged and current components, improving bias correction and estimate accuracy.

Contribution

DOLCE introduces a support-robust decomposition approach using lagged data, with a moment-based training procedure for unbiased and consistent off-policy evaluation and learning.

Findings

DOLCE achieves substantial improvements in evaluation accuracy.

It remains unbiased under idealized conditions.

It performs well even with support violations.

Abstract

Off-policy evaluation and learning in contextual bandits use logged interaction data to estimate and optimize the value of a target policy. Most existing methods require sufficient action overlap between the logging and target policies, and violations can bias value and policy gradient estimates. To address this issue, we propose DOLCE (Decomposing Off-policy evaluation/learning into Lagged and Current Effects), which uses only lagged contexts already stored in bandit logs to construct lag-marginalized importance weights and to decompose the objective into a support-robust lagged correction term and a current, model-based term, yielding bias cancellation when the reward-model residual is conditionally mean-zero given the lagged context and action. With multiple candidate lags, DOLCE softly aggregates lag-specific estimates, and we introduce a moment-based training procedure that promotes the desired invariance using only logged lag-augmented data. We show that DOLCE is unbiased in an idealized setting and yields consistent and asymptotically normal estimates with cross-fitting under standard conditions. Our experiments demonstrate that DOLCE achieves substantial improvements in both off-policy evaluation and learning, particularly as the proportion of individuals who violate support increases.