Priming, Path-dependence, and Plasticity: Understanding the molding of user-LLM interaction and its implications from (many) chat logs in the wild

TL;DR

This study analyzes large-scale real-world chat logs to understand how user interactions with LLMs develop, revealing rapid stabilization of interaction patterns and the influence of early exploration on long-term outcomes.

Contribution

It introduces a large-scale analysis of in-the-wild user-LLM interactions, highlighting the importance of early trajectories and dynamic patterns in shaping user behavior.

Findings

User interaction patterns stabilize quickly through early trajectories.

Early exploration strongly influences long-term outcomes like text patterns and retention.

Parallel dynamics include task-specific expressions and responses to model updates.

Abstract

User interactions with LLMs are shaped by prior experiences and individual exploration, but in-lab studies do not provide system designers with visibility into these in-the-wild factors. This work explores a new approach to studying real-world user-LLM interactions through large-scale chat logs from the wild. Through analysis of 140K chatbot sessions from 7,955 anonymized global users over time, we demonstrate key patterns in user expressions despite varied tasks: (1) LLM users are not tabula rasa, nor are they constantly adapting; rather, interaction patterns form and stabilize rapidly through individual early trajectories; (2) Longitudinal outcomes, such as recurring text patterns and retention rates, are strongly correlated with early exploration; (3) Parallel dynamics are present, including organizing expressions by task types such as emotional support, or in response to model-version updates. These results present an ``agency paradox'': despite LLM input spaces being unconstrained and user-driven, we in fact see less user exploration. We call for design consideration surrounding the molding procedure and its incorporation in future research.