Divergence-Based Similarity Function for Multi-View Contrastive Learning

TL;DR

This paper introduces a divergence-based similarity function for multi-view contrastive learning that models joint structure across views, improving performance and efficiency without requiring hyperparameter tuning.

Contribution

The paper proposes a novel divergence-based similarity function that captures joint multi-view structure, enhancing contrastive learning beyond pairwise relationships.

Findings

Consistently improves performance across multiple tasks

Operates effectively without temperature hyperparameter tuning

Achieves greater efficiency than existing multi-view methods

Abstract

Recent success in contrastive learning has sparked growing interest in more effectively leveraging multiple augmented views of data. While prior methods incorporate multiple views at the loss or feature level, they primarily capture pairwise relationships and fail to model the joint structure across all views. In this work, we propose a divergence-based similarity function (DSF) that explicitly captures the joint structure by representing each set of augmented views as a distribution and measuring similarity as the divergence between distributions. Extensive experiments demonstrate that DSF consistently improves performance across diverse tasks, including kNN classification, linear evaluation, transfer learning, and distribution shift, while also achieving greater efficiency than other multi-view methods. Furthermore, we establish a connection between DSF and cosine similarity, and demonstrate that, unlike cosine similarity, DSF operates effectively without the need for tuning a temperature hyperparameter.