CEI: A Benchmark for Evaluating Pragmatic Reasoning in Language Models

TL;DR

The paper introduces the CEI Benchmark, a dataset of 300 scenarios designed to evaluate large language models' ability to perform pragmatic reasoning in diverse, real-world communication contexts involving ambiguity and social dynamics.

Contribution

It provides a new, comprehensive benchmark with detailed annotation methodology for assessing pragmatic inference in language models.

Findings

LLMs struggle with pragmatic inference, especially in ambiguous scenarios.

Inter-annotator agreement is low, highlighting the inherent difficulty of pragmatic reasoning.

The dataset covers multiple pragmatic subtypes across various social settings.

Abstract

Pragmatic reasoning, inferring intended meaning beyond literal semantics, underpins everyday communication yet remains difficult for large language models. We present the Contextual Emotional Inference (CEI) Benchmark: 300 human-validated scenarios for evaluating how well LLMs disambiguate pragmatically complex utterances. Each scenario pairs a situational context and speaker-listener roles (with explicit power relations) against an ambiguous utterance. The dataset covers five pragmatic subtypes (sarcasm/irony, mixed signals, strategic politeness, passive aggression, deflection/misdirection) drawn from workplace, family, social, and service settings, with three power configurations (peer, higher-to-lower, lower-to-higher). Three trained annotators independently labeled every scenario. Inter-annotator agreement (Fleiss' kappa = 0.06-0.25 by subtype) is low but expected: pragmatic inference admits multiple valid readings, and the disagreement itself is informative. We describe our annotation methodology, including a 4-level quality control pipeline that combines automated statistical checks with expert adjudication. CEI is released under CC-BY-4.0.