Probabilistic Self-supervised Learning via Scoring Rules Minimization

TL;DR

ProSMIN introduces a probabilistic self-supervised learning framework that uses scoring rules minimization to improve representation quality, robustness, and calibration across various tasks and datasets.

Contribution

It presents a novel loss function based on proper scoring rules and a theoretical convergence analysis for probabilistic self-supervised learning.

Findings

Achieves superior accuracy on ImageNet variants

Improves calibration and robustness in downstream tasks

Demonstrates scalability on large-scale datasets

Abstract

In this paper, we propose a novel probabilistic self-supervised learning via Scoring Rule Minimization (ProSMIN), which leverages the power of probabilistic models to enhance representation quality and mitigate collapsing representations. Our proposed approach involves two neural networks; the online network and the target network, which collaborate and learn the diverse distribution of representations from each other through knowledge distillation. By presenting the input samples in two augmented formats, the online network is trained to predict the target network representation of the same sample under a different augmented view. The two networks are trained via our new loss function based on proper scoring rules. We provide a theoretical justification for ProSMIN's convergence, demonstrating the strict propriety of its modified scoring rule. This insight validates the method's optimization process and contributes to its robustness and effectiveness in improving representation quality. We evaluate our probabilistic model on various downstream tasks, such as in-distribution generalization, out-of-distribution detection, dataset corruption, low-shot learning, and transfer learning. Our method achieves superior accuracy and calibration, surpassing the self-supervised baseline in a wide range of experiments on large-scale datasets like ImageNet-O and ImageNet-C, ProSMIN demonstrates its scalability and real-world applicability.