Towards Optimal Convolutional Transfer Learning Architectures for Breast Lesion Classification and ACL Tear Detection

TL;DR

This study investigates optimal CNN architectures for medical image classification, comparing pre-training datasets, and finds that certain architectures perform best, but RadImageNet pre-training does not always improve results.

Contribution

It extends prior work by systematically evaluating CNN architectures and pre-training effects for breast lesion and ACL tear detection, providing practical guidelines for model selection.

Findings

ResNet50 with partial backbone unfreezing yields top performance.

Best models achieve AUCs of 0.9969 for ACL tears and 0.9641 for breast lesions.

RadImageNet pre-training does not consistently outperform ImageNet pre-training.

Abstract

Modern computer vision models have proven to be highly useful for medical imaging classification and segmentation tasks, but the scarcity of medical imaging data often limits the efficacy of models trained from scratch. Transfer learning has emerged as a pivotal solution to this, enabling the fine-tuning of high-performance models on small data. Mei et al. (2022) found that pre-training CNNs on a large dataset of radiologist-labeled images (RadImageNet) enhanced model performance on downstream tasks compared to ImageNet pretraining. The present work extends Mei et al. (2022) by conducting a comprehensive investigation to determine optimal CNN architectures for breast lesion malignancy detection and ACL tear detection, as well as performing statistical analysis to compare the effect of RadImageNet and ImageNet pre-training on downstream model performance. Our findings suggest that 1-dimensional convolutional classifiers with skip connections, ResNet50 pre-trained backbones, and partial backbone unfreezing yields optimal downstream medical classification performance. Our best models achieve AUCs of 0.9969 for ACL tear detection and 0.9641 for breast nodule malignancy detection, competitive with the results reported by Mei et al. (2022) and surpassing other previous works. We do not find evidence confirming RadImageNet pre-training to provide superior downstream performance for ACL tear and breast lesion classification tasks.