Benchmark for Uncertainty & Robustness in Self-Supervised Learning

TL;DR

This paper establishes a benchmark for evaluating uncertainty estimation and robustness in self-supervised learning methods across vision and language tasks, addressing a critical gap in assessing model reliability in high-stakes applications.

Contribution

It introduces a comprehensive benchmark evaluating SSL methods on generalization and uncertainty under distributional shifts, with experimental results and open-source code for reproducibility.

Findings

SSL methods vary in robustness and uncertainty estimation

Benchmark provides standardized evaluation across datasets

Results highlight need for improved SSL reliability measures

Abstract

Self-Supervised Learning (SSL) is crucial for real-world applications, especially in data-hungry domains such as healthcare and self-driving cars. In addition to a lack of labeled data, these applications also suffer from distributional shifts. Therefore, an SSL method should provide robust generalization and uncertainty estimation in the test dataset to be considered a reliable model in such high-stakes domains. However, existing approaches often focus on generalization, without evaluating the model's uncertainty. The ability to compare SSL techniques for improving these estimates is therefore critical for research on the reliability of self-supervision models. In this paper, we explore variants of SSL methods, including Jigsaw Puzzles, Context, Rotation, Geometric Transformations Prediction for vision, as well as BERT and GPT for language tasks. We train SSL in auxiliary learning for vision and pre-training for language model, then evaluate the generalization (in-out classification accuracy) and uncertainty (expected calibration error) across different distribution covariate shift datasets, including MNIST-C, CIFAR-10-C, CIFAR-10.1, and MNLI. Our goal is to create a benchmark with outputs from experiments, providing a starting point for new SSL methods in Reliable Machine Learning. All source code to reproduce results is available at https://github.com/hamanhbui/reliable_ssl_baselines.