CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning

TL;DR

This paper introduces CLCR, a novel multimodal learning framework that organizes features into a semantic hierarchy and employs cross-level interactions to improve shared and private information extraction, enhancing performance across diverse tasks.

Contribution

The paper proposes a hierarchical semantic organization and cross-level interaction mechanism for multimodal data, addressing semantic misalignment and error propagation in existing fusion methods.

Findings

Achieves state-of-the-art results on six multimodal benchmarks.

Effectively separates shared and private features across modalities.

Demonstrates strong generalization across various tasks.

Abstract

Multimodal learning aims to capture both shared and private information from multiple modalities. However, existing methods that project all modalities into a single latent space for fusion often overlook the asynchronous, multi-level semantic structure of multimodal data. This oversight induces semantic misalignment and error propagation, thereby degrading representation quality. To address this issue, we propose Cross-Level Co-Representation (CLCR), which explicitly organizes each modality's features into a three-level semantic hierarchy and specifies level-wise constraints for cross-modal interactions. First, a semantic hierarchy encoder aligns shallow, mid, and deep features across modalities, establishing a common basis for interaction. And then, at each level, an Intra-Level Co-Exchange Domain (IntraCED) factorizes features into shared and private subspaces and restricts cross-modal attention to the shared subspace via a learnable token budget. This design ensures that only shared semantics are exchanged and prevents leakage from private channels. To integrate information across levels, the Inter-Level Co-Aggregation Domain (InterCAD) synchronizes semantic scales using learned anchors, selectively fuses the shared representations, and gates private cues to form a compact task representation. We further introduce regularization terms to enforce separation of shared and private features and to minimize cross-level interference. Experiments on six benchmarks spanning emotion recognition, event localization, sentiment analysis, and action recognition show that CLCR achieves strong performance and generalizes well across tasks.