Explanation Generation for Contradiction Reconciliation with LLMs

TL;DR

This paper introduces the task of generating explanations that reconcile contradictory statements using large language models, highlighting their current limitations and proposing methods for automatic evaluation.

Contribution

It defines the new task of reconciliatory explanation generation, repurposes existing NLI datasets for this purpose, and evaluates 18 LLMs on this task.

Findings

Most LLMs achieve limited success in reconciliation explanations.

Model size increase yields diminishing returns in reasoning ability.

The proposed metrics enable scalable automatic evaluation.

Abstract

Existing NLP work commonly treats contradictions as errors to be resolved by choosing which statements to accept or discard. Yet a key aspect of human reasoning in social interactions and professional domains is the ability to hypothesize explanations that reconcile contradictions. For example, "Cassie hates coffee" and "She buys coffee everyday" may appear contradictory, yet both are compatible if Cassie has the unenviable daily chore of buying coffee for all her coworkers. Despite the growing reasoning capabilities of large language models (LLMs), their ability to hypothesize such reconciliatory explanations remains largely unexplored. To address this gap, we introduce the task of reconciliatory explanation generation, where models must generate explanations that effectively render contradictory statements compatible. We propose a novel method of repurposing existing natural language inference (NLI) datasets, and introduce quality metrics that enable scalable automatic evaluation. Experiments with 18 LLMs show that most models achieve limited success in this task, and that the benefit of extending test-time compute by "thinking" plateaus as model size increases. Our results highlight an under-explored dimension of LLM reasoning and the need to address this limitation in enhancing LLMs' downstream applications such as chatbots and scientific aids.