Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations

TL;DR

CoDAAR introduces a novel discrete multimodal learning framework that aligns semantic representations across modalities, enabling robust cross-modal and cross-domain generalization while preserving modality-specific structures.

Contribution

It proposes a unified discrete space with index-level alignment, combining temporal and semantic alignment mechanisms to improve multimodal generalization.

Findings

Achieves state-of-the-art results on multiple cross-modal benchmarks.

Effectively preserves modality-specific structures while enabling generalization.

Demonstrates robustness across diverse multimodal tasks.

Abstract

Multimodal learning seeks to integrate information across diverse sensory sources, yet current approaches struggle to balance cross-modal generalizability with modality-specific structure. Continuous (implicit) methods preserve fine-grained priors but render generalization challenging, while discrete (explicit) approaches enforce shared prototypes at the expense of modality specificity. We introduce CoDAAR (Cross-modal Discrete Alignment And Reconstruction), a novel framework that resolves this long-standing trade-off by establishing semantic consensus across modality-specific codebooks through index-level alignment. This design uniquely allows CoDAAR to preserve modality-unique structures while achieving generalizable cross-modal representations within a unified discrete space. CoDAAR combines two complementary mechanisms: Discrete Temporal Alignment (DTA), which enables fine-grained temporal quantization, and Cascading Semantic Alignment (CSA), which promotes progressive cross-modal semantic agreement. Together, they establish a competition-free unified representation space. Trained with self-supervised reconstruction objectives on paired multimodal sequences, CoDAAR demonstrates robust cross-modal and cross-domain generalization. Across Cross-Modal Generalization benchmarks, including event classification, localization, video segmentation, and cross-dataset transfer, CoDAAR achieves state-of-the-art performance, establishing a new paradigm for discrete and generalizable multimodal representation learning.