Show or Tell? Modeling the evolution of request-making in Human-LLM conversations

TL;DR

This paper introduces a framework for analyzing user request-making in human-LLM conversations, revealing evolving interaction patterns and differences from human-human interactions, with implications for user studies and LLM alignment.

Contribution

It presents a new segmentation framework for user requests, a large real-world query dataset, and diachronic analysis revealing evolving and converging user interaction patterns.

Findings

Request patterns evolve from simple requests to more context-rich interactions

Users explore different expression patterns but tend to converge over time

Distinct task-related expression trends can be identified across users

Abstract

Designing user-centered LLM systems requires understanding how people use them, but patterns of user behavior are often masked by the variability of queries. In this work, we introduce a new framework to describe request-making that segments user input into request content, roles assigned, query-specific context, and the remaining task-independent expressions. We apply the workflow to create and analyze a dataset of 211k real-world queries based on WildChat. Compared with similar human-human setups, we find significant differences in the language for request-making in the human-LLM scenario. Further, we introduce a novel and essential perspective of diachronic analyses with user expressions, which reveals fundamental and habitual user-LLM interaction patterns beyond individual task completion. We find that query patterns evolve from early ones emphasizing sole requests to combining more context later on, and individual users explore expression patterns but tend to converge with more experience. From there, we propose to understand communal trends of expressions underlying distinct tasks and discuss the preliminary findings. Finally, we discuss the key implications for user studies, computational pragmatics, and LLM alignment.