Does Hearing Help Seeing? Investigating Audio-Video Joint Denoising for Video Generation

TL;DR

This paper investigates whether joint audio-video denoising improves video quality in generative models, finding that it enhances video realism and motion understanding beyond synchronization, especially in complex scenes.

Contribution

Introduces a parameter-efficient AVFullDiT architecture leveraging pre-trained modules for joint denoising, providing systematic evidence of cross-modal benefits for video generation.

Findings

Joint denoising improves video quality beyond synchrony.

Audio acts as a privileged signal for causal understanding.

Enhanced performance on scenes with large and contact motions.

Abstract

Recent audio-video generative systems suggest that coupling modalities benefits not only audio-video synchrony but also the video modality itself. We pose a fundamental question: Does audio-video joint denoising training improve video generation, even when we only care about video quality? To study this, we introduce a parameter-efficient Audio-Video Full DiT (AVFullDiT) architecture that leverages pre-trained text-to-video (T2V) and text-to-audio (T2A) modules for joint denoising. We train (i) a T2AV model with AVFullDiT and (ii) a T2V-only counterpart under identical settings. Our results provide the first systematic evidence that audio-video joint denoising can deliver more than synchrony. We observe consistent improvements on challenging subsets featuring large and object contact motions. We hypothesize that predicting audio acts as a privileged signal, encouraging the model to internalize causal relationships between visual events and their acoustic consequences (e.g., collision $\times$ impact sound), which in turn regularizes video dynamics. Our findings suggest that cross-modal co-training is a promising approach to developing stronger, more physically grounded world models. Code and dataset will be made publicly available.