On Discriminative Probabilistic Modeling for Self-Supervised Representation Learning

TL;DR

This paper introduces a probabilistic modeling framework for self-supervised learning, utilizing multiple importance sampling to improve contrastive loss, and proposes a new non-parametric method that outperforms existing baselines in image-language pretraining.

Contribution

It develops a novel probabilistic approach with a new contrastive objective and efficient algorithm, addressing limitations of current InfoNCE-based methods in self-supervised learning.

Findings

Superior performance on CC3M and CC12M datasets

Effective Monte Carlo integration via MIS

New contrastive objective improves learning quality

Abstract

We study the discriminative probabilistic modeling on a continuous domain for the data prediction task of (multimodal) self-supervised representation learning. To address the challenge of computing the integral in the partition function for each anchor data, we leverage the multiple importance sampling (MIS) technique for robust Monte Carlo integration, which can recover InfoNCE-based contrastive loss as a special case. Within this probabilistic modeling framework, we conduct generalization error analysis to reveal the limitation of current InfoNCE-based contrastive loss for self-supervised representation learning and derive insights for developing better approaches by reducing the error of Monte Carlo integration. To this end, we propose a novel non-parametric method for approximating the sum of conditional probability densities required by MIS through convex optimization, yielding a new contrastive objective for self-supervised representation learning. Moreover, we design an efficient algorithm for solving the proposed objective. We empirically compare our algorithm to representative baselines on the contrastive image-language pretraining task. Experimental results on the CC3M and CC12M datasets demonstrate the superior overall performance of our algorithm. Our code is available at https://github.com/bokun-wang/NUCLR.