From pre-training to downstream performance: Does domain-specific pre-training make sense?

TL;DR

This study systematically compares various pre-training strategies for medical imaging models, highlighting that modality-matched pre-training data significantly enhances downstream performance and that self-supervised learning's effectiveness varies.

Contribution

It provides a comprehensive analysis of pre-training approaches in medical imaging, emphasizing the importance of data modality matching for improved model performance.

Findings

Pre-training on data matching the target modality improves performance.

Self-supervised learning can outperform supervised methods depending on context.

Matching pre-training data with target modalities is crucial for downstream success.

Abstract

Deep learning techniques have revolutionised medical imaging, improving diagnostic accuracy and enabling both more accurate and earlier disease detection. However, the relationship between pre-training strategies and downstream performance in medical imaging models requires further exploration. Here, we systematically compare convolutional neural networks and transformers, examining various pre-training approaches, including supervised and self-supervised learning, as well as different initialisations and data modalities. Models are evaluated on natural images, chest X-rays, chest CT and retina OCT images, considering the effects of matching pre-training data with target modalities. Our findings indicate that only pre-training on data closely matching the target modality significantly improves downstream performance. While self-supervised learning can outperform supervised methods, its effectiveness varies with context. The study underscores the importance of pre-training strategies to enhance the reliability and effectiveness of deep learning models in medical imaging. By addressing these key factors, our research aims to contribute to the development of more accurate and dependable diagnostic tools, ultimately improving patient outcomes in clinical settings.