User Strategization and Trustworthy Algorithms

TL;DR

This paper models user-algorithm interactions as a game, revealing that user strategization can temporarily benefit platforms but ultimately harms data quality and decision accuracy, highlighting the importance of trustworthy algorithm design.

Contribution

It introduces a game-theoretic framework for understanding user strategization, showing its short-term benefits and long-term drawbacks for data integrity and decision-making.

Findings

User strategization can improve short-term platform outcomes.

Strategic behavior corrupts data and impairs counterfactual decision accuracy.

Designing trustworthy algorithms aligns with improving data reliability.

Abstract

Many human-facing algorithms -- including those that power recommender systems or hiring decision tools -- are trained on data provided by their users. The developers of these algorithms commonly adopt the assumption that the data generating process is exogenous: that is, how a user reacts to a given prompt (e.g., a recommendation or hiring suggestion) depends on the prompt and not on the algorithm that generated it. For example, the assumption that a person's behavior follows a ground-truth distribution is an exogeneity assumption. In practice, when algorithms interact with humans, this assumption rarely holds because users can be strategic. Recent studies document, for example, TikTok users changing their scrolling behavior after learning that TikTok uses it to curate their feed, and Uber drivers changing how they accept and cancel rides in response to changes in Uber's algorithm. Our work studies the implications of this strategic behavior by modeling the interactions between a user and their data-driven platform as a repeated, two-player game. We first find that user strategization can actually help platforms in the short term. We then show that it corrupts platforms' data and ultimately hurts their ability to make counterfactual decisions. We connect this phenomenon to user trust, and show that designing trustworthy algorithms can go hand in hand with accurate estimation. Finally, we provide a formalization of trustworthiness that inspires potential interventions.