The Mean is the Mirage: Entropy-Adaptive Model Merging under Heterogeneous Domain Shifts in Medical Imaging

TL;DR

This paper introduces an entropy-adaptive, online model merging technique for medical imaging that effectively handles heterogeneous domain shifts, improving performance without requiring labeled data or multiple models at test time.

Contribution

The paper proposes a novel entropy-adaptive merging method that dynamically combines models during test time, addressing domain shifts in medical imaging without labels or multiple models.

Findings

Consistent performance improvements over state-of-the-art baselines.

Effective handling of heterogeneous domain shifts in medical imaging.

Maintains single-model inference at test time.

Abstract

Model merging under unseen test-time distribution shifts often renders naive strategies, such as mean averaging unreliable. This challenge is especially acute in medical imaging, where models are fine-tuned locally at clinics on private data, producing domain-specific models that differ by scanner, protocol, and population. When deployed at an unseen clinical site, test cases arrive in unlabeled, non-i.i.d. batches, and the model must adapt immediately without labels. In this work, we introduce an entropy-adaptive, fully online model-merging method that yields a batch-specific merged model via only forward passes, effectively leveraging target information. We further demonstrate why mean merging is prone to failure and misaligned under heterogeneous domain shifts. Next, we mitigate encoder classifier mismatch by decoupling the encoder and classification head, merging with separate merging coefficients. We extensively evaluate our method with state-of-the-art baselines using two backbones across nine medical and natural-domain generalization image classification datasets, showing consistent gains across standard evaluation and challenging scenarios. These performance gains are achieved while retaining single-model inference at test-time, thereby demonstrating the effectiveness of our method.