Pretext Matters: An Empirical Study of SSL Methods in Medical Imaging

TL;DR

This study empirically compares different self-supervised learning methods in medical imaging, revealing how the choice of SSL objective should align with the spatial structure and noise characteristics of specific imaging modalities.

Contribution

It systematically evaluates the effectiveness of joint embedding architectures versus predictive architectures in medical imaging, providing guidelines for matching SSL objectives to modality-specific properties.

Findings

JEAs excel with spatially localized signals like histopathology.

JEPAs perform better with globally structured signals like ultrasounds.

SSL method choice impacts clinical relevance of learned features.

Abstract

Though self-supervised learning (SSL) has demonstrated incredible ability to learn robust representations from unlabeled data, the choice of optimal SSL strategy can lead to vastly different performance outcomes in specialized domains. Joint embedding architectures (JEAs) and joint embedding predictive architectures (JEPAs) have shown robustness to noise and strong semantic feature learning compared to pixel reconstruction-based SSL methods, leading to widespread adoption in medical imaging. However, no prior work has systematically investigated which SSL objective is better aligned with the spatial organization of clinically relevant signal. In this work, we empirically investigate how the choice of SSL method impacts the learned representations in medical imaging. We select two representative imaging modalities characterized by unique noise profiles: ultrasound and histopathology. When informative signal is spatially localized, as in histopathology, JEAs are more effective due to their view-invariance objective. In contrast, when diagnostically relevant information is globally structured, such as the macroscopic anatomy present in liver ultrasounds, JEPAs are optimal. These differences are especially evident in the clinical relevance of the learned features, as independently validated by board-certified radiologists and pathologists. Together, our results provide a framework for matching SSL objectives to the structural and noise properties of medical imaging modalities.