PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities

TL;DR

This paper introduces PUB, a benchmark dataset for evaluating large language models' pragmatic understanding across various phenomena, revealing current models' limitations compared to human performance.

Contribution

The paper presents a new benchmark dataset with 14 tasks across four pragmatics phenomena, enabling systematic evaluation of LLMs' pragmatic reasoning abilities.

Findings

Fine-tuning improves smaller models' pragmatics

Larger models perform similarly with or without chat adaptation

Models show variability and gaps compared to human performance

Abstract

LLMs have demonstrated remarkable capability for understanding semantics, but they often struggle with understanding pragmatics. To demonstrate this fact, we release a Pragmatics Understanding Benchmark (PUB) dataset consisting of fourteen tasks in four pragmatics phenomena, namely, Implicature, Presupposition, Reference, and Deixis. We curated high-quality test sets for each task, consisting of Multiple Choice Question Answers (MCQA). PUB includes a total of 28k data points, 6.1k of which have been created by us, and the rest are adapted from existing datasets. We evaluated nine models varying in the number of parameters and type of training. Our study indicates that fine-tuning for instruction-following and chat significantly enhances the pragmatics capabilities of smaller language models. However, for larger models, the base versions perform comparably with their chat-adapted counterparts. Additionally, there is a noticeable performance gap between human capabilities and model capabilities. Furthermore, unlike the consistent performance of humans across various tasks, the models demonstrate variability in their proficiency, with performance levels fluctuating due to different hints and the complexities of tasks within the same dataset. Overall, the benchmark aims to provide a comprehensive evaluation of LLM's ability to handle real-world language tasks that require pragmatic reasoning.