LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment

TL;DR

LanguageBind introduces a novel framework that extends video-language pretraining to N modalities by using language as a semantic anchor, enabling multi-modal alignment and improving performance across diverse benchmarks.

Contribution

The paper proposes a new method to extend video-language pretraining to multiple modalities using language as a semantic bridge, along with a large aligned dataset VIDAL-10M.

Findings

Achieved superior performance on 15 diverse benchmarks.

Effectively aligned multiple modalities to language in shared feature space.

Demonstrated the effectiveness of indirect alignment and modality complementarity.

Abstract

The video-language (VL) pretraining has achieved remarkable improvement in multiple downstream tasks. However, the current VL pretraining framework is hard to extend to multiple modalities (N modalities, N>=3) beyond vision and language. We thus propose LanguageBind, taking the language as the bind across different modalities because the language modality is well-explored and contains rich semantics. Specifically, we freeze the language encoder acquired by VL pretraining, then train encoders for other modalities with contrastive learning. As a result, all modalities are mapped to a shared feature space, implementing multi-modal semantic alignment. While LanguageBind ensures that we can extend VL modalities to N modalities, we also need a high-quality dataset with alignment data pairs centered on language. We thus propose VIDAL-10M with Video, Infrared, Depth, Audio and their corresponding Language, naming as VIDAL-10M. In our VIDAL-10M, all videos are from short video platforms with complete semantics rather than truncated segments from long videos, and all the video, depth, infrared, and audio modalities are aligned to their textual descriptions. LanguageBind has achieved superior performance on a wide range of 15 benchmarks covering video, audio, depth, and infrared. Moreover, multiple experiments have provided evidence for the effectiveness of LanguageBind in achieving indirect alignment and complementarity among diverse modalities. Code address: https://github.com/PKU-YuanGroup/LanguageBind