Consensus or Conflict? Fine-Grained Evaluation of Conflicting Answers in Question-Answering

TL;DR

This paper introduces NATCONFQA, a new benchmark for conflict-aware multi-answer question answering, highlighting the challenges LLMs face in identifying and resolving conflicting answers in realistic scenarios.

Contribution

The paper presents a novel, cost-effective methodology to create a realistic benchmark with conflict labels and evaluates LLMs' performance on it, revealing their limitations.

Findings

LLMs struggle with conflict detection in MAQA.

Existing models often fail to identify conflicting answer pairs.

NATCONFQA provides a more realistic evaluation of LLMs in conflict scenarios.

Abstract

Large Language Models (LLMs) have demonstrated strong performance in question answering (QA) tasks. However, Multi-Answer Question Answering (MAQA), where a question may have several valid answers, remains challenging. Traditional QA settings often assume consistency across evidences, but MAQA can involve conflicting answers. Constructing datasets that reflect such conflicts is costly and labor-intensive, while existing benchmarks often rely on synthetic data, restrict the task to yes/no questions, or apply unverified automated annotation. To advance research in this area, we extend the conflict-aware MAQA setting to require models not only to identify all valid answers, but also to detect specific conflicting answer pairs, if any. To support this task, we introduce a novel cost-effective methodology for leveraging fact-checking datasets to construct NATCONFQA, a new benchmark for realistic, conflict-aware MAQA, enriched with detailed conflict labels, for all answer pairs. We evaluate eight high-end LLMs on NATCONFQA, revealing their fragility in handling various types of conflicts and the flawed strategies they employ to resolve them.