Confounding is a Pervasive Problem in Real World Recommender Systems

TL;DR

This paper highlights that unobserved confounding is a widespread issue in real-world recommender systems, often introduced by standard practices like feature engineering and testing, which can bias causal estimates and reduce system effectiveness.

Contribution

It demonstrates how common recommender system practices can inadvertently introduce confounding, providing simulations and practical strategies to mitigate this problem.

Findings

Confounding affects many recommender system practices.

Simulations show confounding can bias recommendations.

Practical suggestions help reduce confounding effects.

Abstract

Unobserved confounding arises when an unmeasured feature influences both the treatment and the outcome, leading to biased causal effect estimates. This issue undermines observational studies in fields like economics, medicine, ecology or epidemiology. Recommender systems leveraging fully observed data seem not to be vulnerable to this problem. However many standard practices in recommender systems result in observed features being ignored, resulting in effectively the same problem. This paper will show that numerous common practices such as feature engineering, A/B testing and modularization can in fact introduce confounding into recommendation systems and hamper their performance. Several illustrations of the phenomena are provided, supported by simulation studies with practical suggestions about how practitioners may reduce or avoid the affects of confounding in real systems.