Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learning

TL;DR

This paper introduces TG-DP, a dual-path framework that decouples reconstruction and alignment in audio-visual learning, guided by a teacher model, leading to improved zero-shot retrieval and robust representations.

Contribution

TG-DP's novel separation of objectives and teacher guidance enhances cross-modal alignment and reduces semantic noise in large-scale audio-visual pretraining.

Findings

Achieves state-of-the-art zero-shot retrieval performance on AudioSet.

Improves R@1 from 35.2% to 37.4% for video-to-audio retrieval.

Maintains semantic robustness with top linear-probe results on AS20K and VGGSound.

Abstract

Recent advances in audio-visual representation learning have shown the value of combining contrastive alignment with masked reconstruction. However, jointly optimizing these objectives in a single forward pass forces the contrastive branch to rely on randomly visible patches designed for reconstruction rather than cross-modal alignment, introducing semantic noise and optimization interference. We propose TG-DP, a Teacher-Guided Dual-Path framework that decouples reconstruction and alignment into separate optimization paths. By disentangling the masking regimes of the two branches, TG-DP enables the contrastive pathway to use a visibility pattern better suited to cross-modal alignment. A teacher model further provides auxiliary guidance for organizing visible tokens in this branch, helping reduce interference and stabilize cross-modal representation learning. TG-DP achieves state-of-the-art performance in zero-shot retrieval. On AudioSet, it improves R@1 from 35.2\% to 37.4\% for video-to-audio retrieval and from 27.9\% to 37.1\% for audio-to-video retrieval. The learned representations also remain semantically robust, achieving state-of-the-art linear-probe performance on AS20K and VGGSound. Taken together, our results suggest that decoupling multimodal objectives and introducing teacher-guided structure into the contrastive pathway provide an effective framework for improving large-scale audio-visual pretraining. Code is available at https://github.com/wanglg20/TG-DP.