Decoupling Common and Unique Representations for Multimodal Self-supervised Learning

TL;DR

This paper introduces DeCUR, a novel method for multimodal self-supervised learning that effectively separates common and unique representations, improving integration and handling missing modalities across various sensor data scenarios.

Contribution

DeCUR is a new approach that decouples inter- and intra-modal embeddings, enhancing multimodal learning by capturing complementary information and addressing modality-specific features.

Findings

Consistent performance improvement across three multimodal scenarios.

Effective in both multimodal and modality-missing settings.

Applicable to various architectures.

Abstract

The increasing availability of multi-sensor data sparks wide interest in multimodal self-supervised learning. However, most existing approaches learn only common representations across modalities while ignoring intra-modal training and modality-unique representations. We propose Decoupling Common and Unique Representations (DeCUR), a simple yet effective method for multimodal self-supervised learning. By distinguishing inter- and intra-modal embeddings through multimodal redundancy reduction, DeCUR can integrate complementary information across different modalities. We evaluate DeCUR in three common multimodal scenarios (radar-optical, RGB-elevation, and RGB-depth), and demonstrate its consistent improvement regardless of architectures and for both multimodal and modality-missing settings. With thorough experiments and comprehensive analysis, we hope this work can provide valuable insights and raise more interest in researching the hidden relationships of multimodal representations.