Surprisal Minimisation over Goal-directed Alternatives Predicts Production Choice in Dialogue

TL;DR

This paper models dialogue utterance choices as probabilistic, cost-sensitive decisions over contextual alternatives, showing surprisal minimisation over goal-directed options best predicts production behavior.

Contribution

It introduces a framework using language models to generate alternatives and demonstrates surprisal minimisation over goal-directed choices as a key predictor in dialogue.

Findings

Surprisal minimisation over goal-directed alternatives predicts dialogue production choices effectively.

Uniform information density and length costs are less predictive across conditions.

Language model-generated alternatives provide a principled basis for studying speaker and listener pressures.

Abstract

We model utterance production as probabilistic cost-sensitive choice over contextual alternatives, using information-theoretic notions of cost. We distinguish between goal-directed alternatives that realise a fixed communicative intent and goal-agnostic alternatives defined only by contextual plausibility, allowing us to derive speaker- and listener-oriented interpretations of different cost measures. We present a procedure to generate both types of alternative sets using language models. Analysing production choices in open-ended dialogue under both deterministic and probabilistic cost minimisation, we find that surprisal minimisation relative to goal-directed alternatives provides the strongest predictive account under both analyses. By contrast, uniform information density and length-based costs exhibit weaker and less consistent predictive power across conditions. More broadly, our study suggests that alternative-conditioned optimisation with LM-generated alternatives provides a principled framework for studying speaker and listener pressures in naturalistic language production.