Offline Recommender System Evaluation under Unobserved Confounding

TL;DR

This paper investigates the impact of unobserved confounders on off-policy evaluation in recommender systems, highlighting biases and diagnostic limitations, and emphasizing the need for awareness and mitigation strategies.

Contribution

It characterizes the bias caused by unobserved confounders in off-policy estimators and demonstrates the limitations of existing diagnostics in detecting such biases.

Findings

Naive propensity estimation under confounding leads to severely biased metrics.

Existing diagnostics are ineffective at uncovering biases caused by unobserved confounders.

Bias depends on true, unobserved logging propensities, making it non-identifiable.

Abstract

Off-Policy Estimation (OPE) methods allow us to learn and evaluate decision-making policies from logged data. This makes them an attractive choice for the offline evaluation of recommender systems, and several recent works have reported successful adoption of OPE methods to this end. An important assumption that makes this work is the absence of unobserved confounders: random variables that influence both actions and rewards at data collection time. Because the data collection policy is typically under the practitioner's control, the unconfoundedness assumption is often left implicit, and its violations are rarely dealt with in the existing literature. This work aims to highlight the problems that arise when performing off-policy estimation in the presence of unobserved confounders, specifically focusing on a recommendation use-case. We focus on policy-based estimators, where the logging propensities are learned from logged data. We characterise the statistical bias that arises due to confounding, and show how existing diagnostics are unable to uncover such cases. Because the bias depends directly on the true and unobserved logging propensities, it is non-identifiable. As the unconfoundedness assumption is famously untestable, this becomes especially problematic. This paper emphasises this common, yet often overlooked issue. Through synthetic data, we empirically show how na\"ive propensity estimation under confounding can lead to severely biased metric estimates that are allowed to fly under the radar. We aim to cultivate an awareness among researchers and practitioners of this important problem, and touch upon potential research directions towards mitigating its effects.