DiPlomat: A Dialogue Dataset for Situated Pragmatic Reasoning

TL;DR

DiPlomat introduces a new dataset and tasks for evaluating machine pragmatic reasoning in situated conversations, revealing current models' limitations in understanding implicit meanings and context.

Contribution

The paper presents a novel dataset, DiPlomat, and two benchmark tasks for pragmatic reasoning, highlighting the challenges faced by state-of-the-art models in this domain.

Findings

Large language models perform poorly on pragmatic reasoning tasks.

Context comprehension is crucial for effective human-machine interaction.

Current models struggle with applying pragmatic reasoning to implicit meanings.

Abstract

Pragmatic reasoning plays a pivotal role in deciphering implicit meanings that frequently arise in real-life conversations and is essential for the development of communicative social agents. In this paper, we introduce a novel challenge, DiPlomat, aiming at benchmarking machines' capabilities on pragmatic reasoning and situated conversational understanding. Compared with previous works that treat different figurative expressions (e.g. metaphor, sarcasm) as individual tasks, DiPlomat provides a cohesive framework towards general pragmatic understanding. Our dataset is created through the utilization of Amazon Mechanical Turk ( AMT ), resulting in a total of 4, 177 multi-turn dialogues. In conjunction with the dataset, we propose two tasks, Pragmatic Identification and Reasoning (PIR) and Conversational Question Answering (CQA). Experimental results with state-of-the-art (SOTA) neural architectures reveal several significant findings: 1) large language models ( LLMs) exhibit poor performance in tackling this subjective domain; 2) comprehensive comprehension of context emerges as a critical factor for establishing benign human-machine interactions; 3) current models defect in the application of pragmatic reasoning. As a result, we call on more attention to improve the ability of context understanding, reasoning, and implied meaning modeling.