AV-Dialog: Spoken Dialogue Models with Audio-Visual Input

TL;DR

AV-Dialog is a multimodal dialogue system that leverages audio and visual cues to improve robustness and coherence in noisy, multi-speaker environments, outperforming audio-only models.

Contribution

First multimodal dialogue framework combining audio and visual cues for speaker tracking, turn-taking, and response generation in noisy settings.

Findings

Outperforms audio-only models under interference

Reduces transcription errors in noisy environments

Enhances human-rated dialogue quality

Abstract

Dialogue models falter in noisy, multi-speaker environments, often producing irrelevant responses and awkward turn-taking. We present AV-Dialog, the first multimodal dialog framework that uses both audio and visual cues to track the target speaker, predict turn-taking, and generate coherent responses. By combining acoustic tokenization with multi-task, multi-stage training on monadic, synthetic, and real audio-visual dialogue datasets, AV-Dialog achieves robust streaming transcription, semantically grounded turn-boundary detection and accurate responses, resulting in a natural conversational flow. Experiments show that AV-Dialog outperforms audio-only models under interference, reducing transcription errors, improving turn-taking prediction, and enhancing human-rated dialogue quality. These results highlight the power of seeing as well as hearing for speaker-aware interaction, paving the way for {spoken} dialogue agents that perform {robustly} in real-world, noisy environments.