Uncovering the effects of model initialization on deep model generalization: A study with adult and pediatric Chest X-ray images

TL;DR

This study investigates how different model initialization techniques affect the generalization of deep learning models on adult and pediatric chest X-ray datasets, highlighting the benefits of ImageNet pretraining and ensemble methods for robustness.

Contribution

It introduces a comprehensive analysis of initialization techniques in medical imaging, proposing novel ensemble methods and demonstrating the superiority of ImageNet pretraining for model generalization.

Findings

ImageNet-pretrained models outperform randomly initialized models in generalization.

Weight-level ensembles significantly improve recall over individual models.

Pretraining ensures consistent performance across internal and external datasets.

Abstract

Model initialization techniques are vital for improving the performance and reliability of deep learning models in medical computer vision applications. While much literature exists on non-medical images, the impacts on medical images, particularly chest X-rays (CXRs) are less understood. Addressing this gap, our study explores three deep model initialization techniques: Cold-start, Warm-start, and Shrink and Perturb start, focusing on adult and pediatric populations. We specifically focus on scenarios with periodically arriving data for training, thereby embracing the real-world scenarios of ongoing data influx and the need for model updates. We evaluate these models for generalizability against external adult and pediatric CXR datasets. We also propose novel ensemble methods: F-score-weighted Sequential Least-Squares Quadratic Programming (F-SLSQP) and Attention-Guided Ensembles with Learnable Fuzzy Softmax to aggregate weight parameters from multiple models to capitalize on their collective knowledge and complementary representations. We perform statistical significance tests with 95% confidence intervals and p-values to analyze model performance. Our evaluations indicate models initialized with ImageNet-pre-trained weights demonstrate superior generalizability over randomly initialized counterparts, contradicting some findings for non-medical images. Notably, ImageNet-pretrained models exhibit consistent performance during internal and external testing across different training scenarios. Weight-level ensembles of these models show significantly higher recall (p<0.05) during testing compared to individual models. Thus, our study accentuates the benefits of ImageNet-pretrained weight initialization, especially when used with weight-level ensembles, for creating robust and generalizable deep learning solutions.