Audio-Visual World Models: Towards Multisensory Imagination in Sight and Sound

TL;DR

This paper introduces a formal framework and a new dataset for multimodal audio-visual world models, enabling agents to simulate and predict environmental dynamics involving sight and sound.

Contribution

It presents the first formal definition of audio-visual world models, a novel dataset AVW-4k, and a new model AV-CDiT with a specialized architecture for multimodal integration.

Findings

AV-CDiT achieves high-fidelity multimodal predictions.

The model significantly improves performance in audio-visual navigation tasks.

The framework enables synchronized simulation of visual and auditory dynamics.

Abstract

World models simulate environmental dynamics to enable agents to plan and reason about future states. While existing approaches have primarily focused on visual observations, real-world perception inherently involves multiple sensory modalities. Audio provides crucial spatial and temporal cues such as sound source localization and acoustic scene properties, yet its integration into world models remains largely unexplored. No prior work has formally defined what constitutes an audio-visual world model or how to jointly capture binaural spatial audio and visual dynamics under precise action control. This work presents the first formal framework for Audio-Visual World Models (AVWM), formulating multimodal environment simulation as a partially observable Markov decision process with synchronized audio-visual observations. To address the lack of suitable training data, we construct AVW-4k, a dataset comprising 30 hours of binaural audio-visual trajectories with action annotations across 76 indoor environments. We propose AV-CDiT, an Audio-Visual Conditional Diffusion Transformer with a novel modality expert architecture that balances visual and auditory learning, optimized through a three-stage training strategy for effective multimodal integration. Extensive experiments demonstrate that AV-CDiT achieves high-fidelity multimodal prediction across visual and auditory modalities. Furthermore, we validate its practical utility in continuous audio-visual navigation tasks, where AVWM significantly enhances the agent's performance.