DynaMMo: Dynamic Model Merging for Efficient Class Incremental Learning for Medical Images

TL;DR

DynaMMo introduces a computationally efficient method for continual learning in medical imaging by merging models at different training stages, significantly reducing computational costs while maintaining high accuracy.

Contribution

The paper proposes DynaMMo, a novel dynamic model merging approach that reduces computational overhead in continual learning for medical images, outperforming existing methods.

Findings

Achieves around 10-fold reduction in GFLOPS.

Maintains a small accuracy drop of 2.76%.

Effective on three public datasets.

Abstract

Continual learning, the ability to acquire knowledge from new data while retaining previously learned information, is a fundamental challenge in machine learning. Various approaches, including memory replay, knowledge distillation, model regularization, and dynamic network expansion, have been proposed to address this issue. Thus far, dynamic network expansion methods have achieved state-of-the-art performance at the cost of incurring significant computational overhead. This is due to the need for additional model buffers, which makes it less feasible in resource-constrained settings, particularly in the medical domain. To overcome this challenge, we propose Dynamic Model Merging, DynaMMo, a method that merges multiple networks at different stages of model training to achieve better computational efficiency. Specifically, we employ lightweight learnable modules for each task and combine them into a unified model to minimize computational overhead. DynaMMo achieves this without compromising performance, offering a cost-effective solution for continual learning in medical applications. We evaluate DynaMMo on three publicly available datasets, demonstrating its effectiveness compared to existing approaches. DynaMMo offers around 10-fold reduction in GFLOPS with a small drop of 2.76 in average accuracy when compared to state-of-the-art dynamic-based approaches. The code implementation of this work will be available upon the acceptance of this work at https://github.com/BioMedIA-MBZUAI/DynaMMo.