VividVoice: A Unified Framework for Scene-Aware Visually-Driven Speech Synthesis

TL;DR

VividVoice introduces a comprehensive framework for scene-aware, visually-driven speech synthesis that effectively aligns visual scenes with speech characteristics, overcoming data and modality challenges to produce more immersive audio experiences.

Contribution

The paper presents VividVoice, a novel unified generative framework with a large-scale dataset and a new alignment module for improved scene-aware speech synthesis.

Findings

Outperforms baseline models in audio fidelity and clarity

Achieves fine-grained visual-to-audio alignment

Demonstrates strong multimodal consistency

Abstract

We introduce and define a novel task-Scene-Aware Visually-Driven Speech Synthesis, aimed at addressing the limitations of existing speech generation models in creating immersive auditory experiences that align with the real physical world. To tackle the two core challenges of data scarcity and modality decoupling, we propose VividVoice, a unified generative framework. First, we constructed a large-scale, high-quality hybrid multimodal dataset, Vivid-210K, which, through an innovative programmatic pipeline, establishes a strong correlation between visual scenes, speaker identity, and audio for the first time. Second, we designed a core alignment module, D-MSVA, which leverages a decoupled memory bank architecture and a cross-modal hybrid supervision strategy to achieve fine-grained alignment from visual scenes to timbre and environmental acoustic features. Both subjective and objective experimental results provide strong evidence that VividVoice significantly outperforms existing baseline models in terms of audio fidelity, content clarity, and multimodal consistency. Our demo is available at https://chengyuann.github.io/VividVoice/.