MAD: A Benchmark for Multi-Turn Audio Dialogue Fact-Checking

TL;DR

MAD introduces a novel multi-turn audio dialogue dataset for fact-checking, capturing complex spoken misinformation dynamics and providing benchmarks that reveal current model limitations in conversational audio verification.

Contribution

This paper presents MAD, the first dataset for multi-turn spoken dialogue fact-checking with detailed annotations, enabling research on conversational and acoustic misinformation detection.

Findings

Pretrained models achieve 72-74% accuracy at sentence level.

Models reach 71-72% accuracy at dialogue level.

MAD exposes significant challenges in multimodal conversational fact-checking.

Abstract

Despite the growing popularity of audio platforms, fact-checking spoken content remains significantly underdeveloped. Misinformation in speech often unfolds across multi-turn dialogues, shaped by speaker interactions, disfluencies, overlapping speech, and emotional tone-factors that complicate both claim detection and verification. Existing datasets fall short by focusing on isolated sentences or text transcripts, without modeling the conversational and acoustic complexity of spoken misinformation. We introduce MAD (Multi-turn Audio Dialogues), the first fact-checking dataset aligned with multi-turn spoken dialogues and corresponding audio. MAD captures how misinformation is introduced, contested, and reinforced through natural conversation. Each dialogue includes annotations for speaker turns, dialogue scenarios, information spread styles, sentence-level check-worthiness, and both sentence- and dialogue-level veracity. The dataset supports two core tasks: check-worthy claim detection and claim verification. Benchmarking shows that even strong pretrained models reach only 72-74% accuracy at the sentence level and 71-72% at the dialogue level in claim verification, underscoring MAD's difficulty. MAD offers a high-quality benchmark for advancing multimodal and conversational fact-checking, while also surfacing open challenges related to reasoning over speech and dialogue dynamics.