A Self-Supervised Paradigm for Data-Efficient Medical Foundation Model Pre-training: V-information Optimization Framework

TL;DR

This paper introduces a theoretical V-information framework for self-supervised medical model pre-training, enabling more data-efficient learning by selecting diverse and challenging samples, leading to significant performance gains with less data.

Contribution

It provides the first theoretical foundation for sample selection in medical foundation models using V-information, and develops OptiDEL, a method that improves data efficiency and model performance.

Findings

OptiDEL outperforms state-of-the-art methods across eight datasets.

Models trained on 5% data with OptiDEL achieve 6.2% higher mIoU.

OptiDEL achieves 4.7% average mIoU improvement using 20x less data.

Abstract

Self-supervised pre-training medical foundation models on large-scale datasets demonstrate exceptional performance. Recent research challenges this common paradigm by introducing data-effective learning approaches, demonstrating that merely increasing pre-training data volume does not necessarily improve model performance. However, current methods still have unclear standards and the underlying theoretical foundation remains unknown. In this paper, as the first attempt to address this limitation, we introduce V-information into self-supervised pre-training of foundation models to provide a theoretical foundation for sample selection. Our derivation confirms that by optimizing V-information, sample selection can be framed as an optimization problem where choosing diverse and challenging samples enhances model performance even under limited training data. Under this guidance, we develop an optimized data-effective learning method (OptiDEL) to optimize V-information in real-world medical domains by generating more diverse and harder samples. We compare the OptiDEL method with state-of-the-art approaches finding that OptiDEL consistently outperforms existing approaches across eight different datasets, with foundation models trained on only 5% of the pre-training data achieving up to 6.2% higher mIoU than those trained on the full dataset. Remarkably, OptiDEL demonstrates an average improvement of 4.7% mIoU over competing methods while using 20x less training data.