LLMs and people both learn to form conventions -- just not with each other

TL;DR

This study investigates whether large language models (LLMs) and humans develop shared conversational conventions, finding that both form such conventions within their own types but struggle to align across human-AI pairs, highlighting differences in interpretative biases.

Contribution

The paper demonstrates that LLMs and humans form conventions independently, and that inducing LLMs to mimic human-like behavior does not fully bridge the communicative gap.

Findings

Humans and LLMs form conventions within their own pairs.

Heterogeneous human-AI pairs fail to develop shared conventions.

Prompting LLMs to behave more human-like does not improve alignment significantly.

Abstract

Humans align to one another in conversation -- adopting shared conventions that ease communication. We test whether LLMs form the same kinds of conventions in a multimodal communication game. Both humans and LLMs display evidence of convention-formation (increasing the accuracy and consistency of their turns while decreasing their length) when communicating in same-type dyads (humans with humans, AI with AI). However, heterogenous human-AI pairs fail -- suggesting differences in communicative tendencies. In Experiment 2, we ask whether LLMs can be induced to behave more like human conversants, by prompting them to produce superficially humanlike behavior. While the length of their messages matches that of human pairs, accuracy and lexical overlap in human-LLM pairs continues to lag behind that of both human-human and AI-AI pairs. These results suggest that conversational alignment requires more than just the ability to mimic previous interactions, but also shared interpretative biases toward the meanings that are conveyed.