The Pragmatic Mind of Machines: Tracing the Emergence of Pragmatic Competence in Large Language Models

TL;DR

This paper investigates how large language models develop pragmatic understanding during training, introducing a new dataset and evaluating models at different stages to reveal emergent pragmatic competence aligned with human communication.

Contribution

The study introduces ALTPRAG, a novel dataset for assessing pragmatic inference in LLMs, and systematically evaluates how pragmatic skills emerge and improve through training stages.

Findings

Pragmatic competence improves with model size and training data scale.

Supervised fine-tuning and reinforcement learning further enhance pragmatic understanding.

Base models already show sensitivity to pragmatic cues, which grows with training.

Abstract

Current large language models (LLMs) have demonstrated emerging capabilities in social intelligence tasks, including implicature resolution and theory-of-mind reasoning, both of which require substantial pragmatic understanding. However, how LLMs acquire this pragmatic competence throughout the training process remains poorly understood. In this work, we introduce ALTPRAG, a dataset grounded in the pragmatic concept of alternatives, to evaluate whether LLMs at different training stages can accurately infer nuanced speaker intentions. Each instance pairs two equally plausible yet pragmatically divergent continuations and requires the model to (i) infer the speaker's intended meaning and (ii) explain when and why a speaker would choose one utterance over its alternative, thus directly probing pragmatic competence through contrastive reasoning. We systematically evaluate 22 LLMs across 3 key training stages: after pre-training, supervised fine-tuning (SFT), and preference optimization, to examine the development of pragmatic competence. Our results show that even base models exhibit notable sensitivity to pragmatic cues, which improves consistently with increases in model and data scale. Additionally, SFT and RLHF contribute further gains, particularly in cognitive-pragmatic scenarios. These findings highlight pragmatic competence as an emergent and compositional property of LLM training and offer new insights for aligning models with human communicative norms.