Knowing When Not to Predict: Self Supervised Learning and Abstention for Safer DR Screening

TL;DR

This paper investigates how self-supervised learning pretraining duration affects model confidence and abstention capabilities in medical image screening, emphasizing reliability over mere accuracy.

Contribution

It provides a comprehensive analysis of SSL pretraining length on confidence calibration and abstention, highlighting its impact on model reliability in safety-critical tasks.

Findings

SSL pretraining improves selective prediction over training from scratch.

Longer pretraining does not always enhance reliability once accuracy saturates.

Abstention-aware evaluation is crucial for assessing model safety in medical screening.

Abstract

Self-supervised learning (SSL) is now a standard way to pretrain medical image models, but performance is still mostly judged by downstream accuracy. For safety-critical screening tasks such as diabetic retinopathy grading, this is not enough: a model must also know when its predictions are unreliable and defer uncertain cases for clinical review. In this work, we examine how the length of SSL pretraining influences calibrated confidence and confidence-based abstention. We evaluate multiple SSL checkpoints under a fixed fine-tuning protocol and assess calibrated confidence, coverage, selective accuracy, and selective macro-F1. Across datasets and data regimes, SSL pretraining improves selective prediction compared to training from scratch. Unlike prior SSL studies that primarily evaluate downstream accuracy or AUROC, we analyze how SSL pretraining duration influences confidence behavior under calibrated confidence-based abstention. However, once accuracy saturates, selective performance can still change markedly across checkpoints, and longer pretraining does not consistently improve reliability. These results underscore the importance of abstention-aware evaluation and suggest that pretraining length should be treated as an important reliability-related design choice rather than only a computational detail. Code is available at GitHub.