Spectral Disentanglement and Enhancement: A Dual-domain Contrastive Framework for Representation Learning

TL;DR

This paper introduces SDE, a dual-domain contrastive framework that uses spectral analysis to improve multimodal representation learning by enhancing signal separation and robustness.

Contribution

It proposes a novel spectral disentanglement and enhancement method that adaptively partitions feature dimensions and integrates spectral regularization into contrastive learning.

Findings

Outperforms state-of-the-art methods on large-scale benchmarks.

Improves robustness and generalization of learned representations.

Effectively integrates spectral regularization into contrastive frameworks.

Abstract

Large-scale multimodal contrastive learning has recently achieved impressive success in learning rich and transferable representations, yet it remains fundamentally limited by the uniform treatment of feature dimensions and the neglect of the intrinsic spectral structure of the learned features. Empirical evidence indicates that high-dimensional embeddings tend to collapse into narrow cones, concentrating task-relevant semantics in a small subspace, while the majority of dimensions remain occupied by noise and spurious correlations. Such spectral imbalance and entanglement undermine model generalization. We propose Spectral Disentanglement and Enhancement (SDE), a novel framework that bridges the gap between the geometry of the embedded spaces and their spectral properties. Our approach leverages singular value decomposition to adaptively partition feature dimensions into strong signals that capture task-critical semantics, weak signals that reflect ancillary correlations, and noise representing irrelevant perturbations. A curriculum-based spectral enhancement strategy is then applied, selectively amplifying informative components with theoretical guarantees on training stability. Building upon the enhanced features, we further introduce a dual-domain contrastive loss that jointly optimizes alignment in both the feature and spectral spaces, effectively integrating spectral regularization into the training process and encouraging richer, more robust representations. Extensive experiments on large-scale multimodal benchmarks demonstrate that SDE consistently improves representation robustness and generalization, outperforming state-of-the-art methods. SDE integrates seamlessly with existing contrastive pipelines, offering an effective solution for multimodal representation learning.