MedMerge: Merging Models for Effective Transfer Learning to Medical Imaging Tasks

TL;DR

MedMerge introduces a novel model merging technique that combines models from different initializations to enhance transfer learning performance in medical imaging, achieving up to 7% F1 score improvement.

Contribution

This work presents a new method for merging models with different initializations, leveraging kernel-level weights to improve transfer learning in medical imaging tasks.

Findings

Achieved up to 7% improvement in F1 score.

Effective merging of models from different initializations.

Applicable across various medical imaging tasks.

Abstract

Transfer learning has become a powerful tool to initialize deep learning models to achieve faster convergence and higher performance. This is especially useful in the medical imaging analysis domain, where data scarcity limits possible performance gains for deep learning models. Some advancements have been made in boosting the transfer learning performance gain by merging models starting from the same initialization. However, in the medical imaging analysis domain, there is an opportunity to merge models starting from different initializations, thus combining the features learned from different tasks. In this work, we propose MedMerge, a method whereby the weights of different models can be merged, and their features can be effectively utilized to boost performance on a new task. With MedMerge, we learn kernel-level weights that can later be used to merge the models into a single model, even when starting from different initializations. Testing on various medical imaging analysis tasks, we show that our merged model can achieve significant performance gains, with up to 7% improvement on the F1 score. The code implementation of this work is available at github.com/BioMedIA-MBZUAI/MedMerge.