Choosing the Right Weights: Balancing Value, Strategy, and Noise in Recommender Systems

TL;DR

This paper analyzes how to optimally weight different user behaviors in recommender systems by balancing value, strategy, and noise, and demonstrates the benefits of these weights on user utility and societal outcomes using Facebook data.

Contribution

It introduces a theoretical framework for selecting behavior weights considering value, strategy, and noise, and applies it to improve Facebook's URL recommendation system.

Findings

Optimal weights increase user utility and societal benefits.

User-optimal weights outperform naive weighting schemes.

Applying the framework reduces misinformation and improves URL quality.

Abstract

Many recommender systems optimize a linear weighting of different user behaviors, such as clicks, likes, and shares. We analyze the optimal choice of weights from the perspectives of both users and content producers who strategically respond to the weights. We consider three aspects of each potential behavior: value-faithfulness (how well a behavior indicates whether the user values the content), strategy-robustness (how hard it is for producers to manipulate the behavior), and noisiness (how much estimation error there is in predicting the behavior). Our theoretical results show that for users, up-weighting more value-faithful and less noisy behaviors leads to higher utility, while for producers, up-weighting more value-faithful and strategy-robust behaviors leads to higher welfare (and the impact of noise is non-monotonic). Finally, we apply our framework to design weights on Facebook, using a large-scale dataset of approximately 70 million URLs shared on Facebook. Strikingly, we find that our user-optimal weight vector (a) delivers higher user value than a vector not accounting for variance; (b) also enhances broader societal outcomes, reducing misinformation and raising the quality of the URL domains, outcomes that were not directly targeted in our theoretical framework.