Cats or CAT scans: transfer learning from natural or medical image source datasets?

TL;DR

This paper reviews transfer learning strategies in medical image analysis, comparing the effectiveness of using natural image datasets like cats versus medical datasets like head CTs as source data.

Contribution

It provides a comprehensive survey of studies comparing different source datasets for transfer learning in medical imaging, highlighting the need for further research.

Findings

Diverse source datasets can improve model robustness.

The optimal source dataset depends on the specific task.

More research is needed to understand transfer learning effectiveness.

Abstract

Transfer learning is a widely used strategy in medical image analysis. Instead of only training a network with a limited amount of data from the target task of interest, we can first train the network with other, potentially larger source datasets, creating a more robust model. The source datasets do not have to be related to the target task. For a classification task in lung CT images, we could use both head CT images, or images of cats, as the source. While head CT images appear more similar to lung CT images, the number and diversity of cat images might lead to a better model overall. In this survey we review a number of papers that have performed similar comparisons. Although the answer to which strategy is best seems to be "it depends", we discuss a number of research directions we need to take as a community, to gain more understanding of this topic.