CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebook

TL;DR

CodeBind introduces a novel multimodal alignment framework using a compositional codebook approach, effectively handling modality discrepancies and data scarcity to improve classification and retrieval performance.

Contribution

It proposes a decoupled representation learning method with shared and modality-specific codebooks, enabling alignment without fully paired data and capturing modality-unique features.

Findings

Achieves state-of-the-art results across nine modalities.

Effectively handles data scarcity and modality discrepancies.

Improves semantic consistency in multimodal tasks.

Abstract

Multimodal representation alignment is pivotal for large language models and robotics. Traditional methods are often hindered by cross-modal information discrepancies and data scarcity, leading to suboptimal alignment spaces that overlook modality-unique features. We propose CodeBind, a framework that optimizes multimodal representation spaces through a modality-shared-specific codebook design. By incrementally aligning target and bridging modalities, CodeBind bypasses the need for fully paired data. Unlike traditional hard alignment, CodeBind decomposes features into shared components for semantic consistency and specific components for modality-unique details. This design utilizes a compositional vector quantization scheme, where a shared codebook bridges modality gaps and modality-specific codebooks mitigate representation bias by preventing dominant modalities from overshadowing others. Validated across nine modalities (text, image, video, audio, depth, thermal, tactile, 3D point cloud, EEG), CodeBind achieves state-of-the-art performance in multimodal classification and retrieval tasks.