Knowledge Transfer Scaling Laws for 3D Medical Imaging

TL;DR

This paper investigates how different 3D medical imaging domains transfer knowledge during pretraining, revealing variable scaling behaviors and proposing an optimized data allocation strategy that improves downstream clinical tasks.

Contribution

It uncovers asymmetric transfer dynamics among medical imaging domains and formulates a scaling-law based data allocation method for better pretraining.

Findings

Transfer-aware data allocation outperforms proportional sampling by up to 58%.

Highly transferable domains act as hubs, benefiting many others.

Transfer-aware mixtures improve downstream disease classification and segmentation.

Abstract

Vision foundation models are increasingly moving beyond 2D to volumetric domains such as 3D medical imaging, where unified pretraining across different imaging modalities (i.e. CT, MRI, and PET) could provide foundational models for diverse clinical tasks. However, training such models requires mixing heterogeneous imaging domains, and current mixture strategies remain largely heuristic. In this work, we observe that different medical imaging domains scale at variable rates during pretraining, and knowledge transfer between domains is strongly asymmetric: training on one domain can substantially improve another, but the reverse may be much weaker. Interestingly, both MAE reconstruction loss and cross-domain transfer follow predictable power-law trends with domain-specific behaviors. Motivated by these findings, we formulate data allocation as a scaling-law optimization problem. The derived allocations reveal an interpretable hub-and-island structure: highly transferable domains emerge as hubs that benefit many others and deserve strategic allocation, while isolated domains act as islands requiring direct investment. Empirically, transfer-aware allocation outperforms data-proportional sampling by up to 58% and generalizes well to unseen budgets with r=0.989. Downstream validation on disease classification and organ/lesion segmentation further confirms that the derived transfer-aware mixtures provide stronger pretrained representations for clinical 3D medical imaging tasks.