Measuring Intent Comprehension in LLMs

TL;DR

This paper introduces a formal framework to evaluate whether large language models can reliably understand user intent by producing consistent outputs for semantically equivalent prompts, addressing a key challenge in high-stakes applications.

Contribution

The paper proposes a variance decomposition framework for assessing intent comprehension in LLMs and applies it across multiple models to measure robustness and understanding.

Findings

Larger models tend to better attribute output variance to user intent.

Model understanding of intent improves with size but gains are often modest.

The framework highlights the importance of semantic diagnostics over accuracy-only benchmarks.

Abstract

People judge interactions with large language models (LLMs) as successful when outputs match what they want, not what they type. Yet LLMs are trained to predict the next token solely from text input, not underlying intent. Because written language is an imperfect proxy for intent, and correlations between phrasing and desired outcomes can break down in training data, models that rely too heavily on surface cues may respond inconsistently to semantically equivalent prompts. This makes it essential to evaluate whether LLMs can reliably infer user intent-especially in high-stakes settings where robustness and generalization are critical. We introduce a formal framework for assessing intent comprehension in LLMs: whether a model demonstrates robust understanding of user intent by producing consistent outputs across semantically equivalent prompts while differentiating between prompts with distinct intents. Our evaluation approach is based on a variance decomposition of model responses into three components: variability due to user intent, user articulation, and model uncertainty. Models that understand what users want, and are not overly sensitive to textual cues, should attribute most output variance to intent differences, rather than articulation style. Applying this framework across diverse domains, we find that, within the five LLaMA and Gemma models we evaluate, larger models typically assign a greater share of variance to intent, indicating stronger comprehension of intent, although gains are uneven and often modest with increasing model size. These results motivate moving beyond accuracy-only benchmarks toward semantic diagnostics that directly assess whether models understand what users intend.