People Can Accurately Predict Behavior of Complex Algorithms That Are Available, Compact, and Aligned

TL;DR

This study demonstrates that users can accurately predict the behavior of complex algorithms if they meet certain cognitive and alignment criteria, challenging assumptions about algorithm complexity and predictability.

Contribution

The paper introduces a theory identifying conditions under which complex algorithms can be predictably modeled by users, supported by experimental validation.

Findings

Complex algorithms can be predicted accurately when criteria are met.

Failure to meet any criterion reduces predictability.

Predictability influences mental model deployment.

Abstract

Users trust algorithms more when they can predict the algorithms' behavior. Simple algorithms trivially yield predictively accurate mental models, but modern AI algorithms have often been assumed too complex for people to build predictive mental models, especially in the social media domain. In this paper, we describe conditions under which even complex algorithms can yield predictive mental models, opening up opportunities for a broader set of human-centered algorithms. We theorize that users will form an accurate predictive mental model of an algorithm's behavior if and only if the algorithm simultaneously satisfies three criteria: (1) cognitive availability of the underlying concepts being modeled, (2) concept compactness (does it form a single cognitive construct?), and (3) high alignment between the person's and algorithm's execution of the concept. We evaluate this theory through a pre-registered experiment (N=1250) where users predict behavior of 25 social media feed ranking algorithms that vary on these criteria. We find that even complex (e.g., LLM-based) algorithms enjoy accurate prediction rates when they meet all criteria, and even simple (e.g., basic term count) algorithms fail to be predictable when a single criterion fails. We also find that these criteria determine outcomes beyond prediction accuracy, such as which mental models users deploy to make their predictions.