DEBATE: A Dataset for Disentangling Textual Ambiguity in Mandarin Through Speech

TL;DR

DEBATE is a pioneering Chinese speech-text dataset designed to explore how speech cues can resolve textual ambiguities, highlighting significant gaps between machine and human understanding of spoken intent.

Contribution

This paper introduces DEBATE, the first dataset pairing ambiguous Chinese utterances with speech cues, enabling research on speech-based disambiguation and speaker intent.

Findings

Large performance gaps between models and humans in understanding spoken ambiguity.

DEBATE dataset contains 1,001 ambiguous utterances recorded by 10 speakers.

Benchmarking shows current models struggle with speech cues for disambiguation.

Abstract

Despite extensive research on textual and visual disambiguation, disambiguation through speech (DTS) remains underexplored. This is largely due to the lack of high-quality datasets that pair spoken sentences with richly ambiguous text. To address this gap, we present DEBATE, a unique public Chinese speech-text dataset designed to study how speech cues and patterns-pronunciation, pause, stress and intonation-can help resolve textual ambiguity and reveal a speaker's true intent. DEBATE contains 1,001 carefully selected ambiguous utterances, each recorded by 10 native speakers, capturing diverse linguistic ambiguities and their disambiguation through speech. We detail the data collection pipeline and provide rigorous quality analysis. Additionally, we benchmark three state-of-the-art large speech and audio-language models, illustrating clear and huge performance gaps between machine and human understanding of spoken intent. DEBATE represents the first effort of its kind and offers a foundation for building similar DTS datasets across languages and cultures. The dataset and associated code are available at: https://github.com/SmileHnu/DEBATE.