JMSC: Joint Spatial–Temporal Modeling with Semantic Completion for Audio–Visual Learning

TL;DR

This paper introduces JMSC, a new framework for audio-visual learning that improves understanding of dynamic scenes by combining spatial and temporal information with semantic completion.

Contribution

The novel JMSC framework uses cross-modal latent reconstruction and joint modeling of spatial and temporal features under audio guidance.

Findings

JMSC achieves state-of-the-art performance on multiple audio-visual tasks.

The method maintains high computational efficiency while improving semantic understanding.

Cross-modal reconstruction enhances the model's ability to capture complementary audio-visual semantics.

Abstract

Audio–visual learning seeks to achieve holistic scene understanding by integrating auditory and visual cues. Early research focused on fully fine-tuning pre-trained models, incurring high computational costs. Consequently, recent studies have adopted parameter-efficient tuning methods to adapt large-scale vision models to the audio–visual domain. Despite the competitive performance of existing methods, several challenges persist. Firstly, effectively leveraging the complementary semantics between the audio and visual modalities remains difficult, as these two modalities capture fundamentally different aspects of a video. Secondly, comprehending dynamic video context is challenging because both spatial attributes (such as scale) and temporal characteristics (such as motion) of objects co-evolve over time, making semantic comprehension more complex. To address these challenges, we propose a novel framework, named Joint Spatial–Temporal Modeling with Semantic Completion (JMSC). JMSC introduces cross-modal latent reconstruction, which moves beyond shallow correlation by encouraging the model to reconstruct one modality’s complete semantic summary from a masked version of its counterpart. Furthermore, JMSC learns a unified representation of video spatial attributes and temporal changes by jointly modeling them under audio guidance, enabling accurate localization and consistent tracking in dynamic video scenes. Experimental results demonstrate that JMSC achieves state-of-the-art performance across multiple downstream tasks while maintaining high computational efficiency.

Figures