Deep Bregman Divergence for Contrastive Learning of Visual Representations

TL;DR

This paper introduces deep Bregman divergences to enhance contrastive learning by capturing distribution-level divergences, leading to improved visual representations in self-supervised and semi-supervised tasks.

Contribution

It proposes a novel contrastive learning framework using deep Bregman divergences to better capture distribution differences, outperforming existing methods.

Findings

Outperforms baseline and previous methods on classification and detection tasks

Enhances representation quality by capturing distribution divergences

Learned features transfer well across datasets and tasks

Abstract

Deep Bregman divergence measures divergence of data points using neural networks which is beyond Euclidean distance and capable of capturing divergence over distributions. In this paper, we propose deep Bregman divergences for contrastive learning of visual representation where we aim to enhance contrastive loss used in self-supervised learning by training additional networks based on functional Bregman divergence. In contrast to the conventional contrastive learning methods which are solely based on divergences between single points, our framework can capture the divergence between distributions which improves the quality of learned representation. We show the combination of conventional contrastive loss and our proposed divergence loss outperforms baseline and most of the previous methods for self-supervised and semi-supervised learning on multiple classifications and object detection tasks and datasets. Moreover, the learned representations generalize well when transferred to the other datasets and tasks. The source code and our models are available in supplementary and will be released with paper.