Group Contrastive Learning for Weakly Paired Multimodal Data

TL;DR

GROOVE introduces a semi-supervised multi-modal learning method with a novel group-level contrastive loss, effectively handling weakly paired data and improving cross-modal representation tasks in high-content perturbation datasets.

Contribution

The paper proposes GroupCLIP, a new contrastive loss for weakly paired multimodal data, integrated with an autoencoder framework, and introduces a comprehensive evaluation framework for such methods.

Findings

GROOVE performs on par or better than existing methods in real datasets.

GroupCLIP is identified as the key component for performance improvements.

The evaluation framework reveals no single aligner dominates across all settings.

Abstract

We present GROOVE, a semi-supervised multi-modal representation learning approach for high-content perturbation data where samples across modalities are weakly paired through shared perturbation labels but lack direct correspondence. Our primary contribution is GroupCLIP, a novel group-level contrastive loss that bridges the gap between CLIP for paired cross-modal data and SupCon for uni-modal supervised contrastive learning, addressing a fundamental gap in contrastive learning for weakly-paired settings. We integrate GroupCLIP with an on-the-fly backtranslating autoencoder framework to encourage cross-modally entangled representations while maintaining group-level coherence within a shared latent space. Critically, we introduce a comprehensive combinatorial evaluation framework that systematically assesses representation learners across multiple optimal transport aligners, addressing key limitations in existing evaluation strategies. This framework includes novel simulations that systematically vary shared versus modality-specific perturbation effects enabling principled assessment of method robustness. Our combinatorial benchmarking reveals that there is not yet an aligner that uniformly dominates across settings or modality pairs. Across simulations and two real single-cell genetic perturbation datasets, GROOVE performs on par with or outperforms existing approaches for downstream cross-modal matching and imputation tasks. Our ablation studies demonstrate that GroupCLIP is the key component driving performance gains. These results highlight the importance of leveraging group-level constraints for effective multi-modal representation learning in scenarios where only weak pairing is available.