Understand the Implication: Learning to Think for Pragmatic Understanding

TL;DR

This paper introduces a new dataset and training method that incorporate explicit reasoning to improve large language models' pragmatic understanding, significantly enhancing their ability to interpret implicit meanings beyond literal text.

Contribution

The paper presents a novel dataset with explicit reasoning annotations and demonstrates that thought-based training improves pragmatic understanding in LLMs, including transfer to unseen tasks.

Findings

Improved accuracy by 11.12% in pragmatic tasks.

Thought-based training enhances understanding of implicit meaning.

Transfer learning shows a 16.10% improvement on unseen pragmatics tasks.

Abstract

Pragmatics, the ability to infer meaning beyond literal interpretation, is crucial for social cognition and communication. While LLMs have been benchmarked for their pragmatic understanding, improving their performance remains underexplored. Existing methods rely on annotated labels but overlook the reasoning process humans naturally use to interpret implicit meaning. To bridge this gap, we introduce a novel pragmatic dataset, ImpliedMeaningPreference, that includes explicit reasoning (thoughts) for both correct and incorrect interpretations. Through preference-tuning and supervised fine-tuning, we demonstrate that thought-based learning significantly enhances LLMs' pragmatic understanding, improving accuracy by 11.12% across model families. We further discuss a transfer-learning study where we evaluate the performance of thought-based training for the other tasks of pragmatics (presupposition, deixis) that are not seen during the training time and observe an improvement of 16.10% compared to label-trained models.