The Geometry of Transfer: Unlocking Medical Vision Manifolds for Training-Free Model Ranking

TL;DR

This paper introduces a topology-based transferability estimation framework for medical image segmentation models, enabling efficient, training-free model ranking by capturing manifold complexity and anatomical boundary information.

Contribution

It proposes a novel topology-driven approach that surpasses traditional statistical metrics for model selection in dense prediction tasks.

Findings

Achieves around 31% relative improvement over state-of-the-art baselines.

Effectively captures manifold complexity and boundary information.

Provides a training-free, robust proxy for model selection.

Abstract

The advent of large-scale self-supervised learning (SSL) has produced a vast zoo of medical foundation models. However, selecting optimal medical foundation models for specific segmentation tasks remains a computational bottleneck. Existing Transferability Estimation (TE) metrics, primarily designed for classification, rely on global statistical assumptions and fail to capture the topological complexity essential for dense prediction. We propose a novel Topology-Driven Transferability Estimation framework that evaluates manifold tractability rather than statistical overlap. Our approach introduces three components: (1) Global Representation Topology Divergence (GRTD), utilizing Minimum Spanning Trees to quantify feature-label structural isomorphism; (2) Local Boundary-Aware Topological Consistency (LBTC), which assesses manifold separability specifically at critical anatomical boundaries; and (3) Task-Adaptive Fusion, which dynamically integrates global and local metrics based on the semantic cardinality of the target task. Validated on the large-scale OpenMind benchmark across diverse anatomical targets and SSL foundation models, our approach significantly outperforms state-of-the-art baselines by around \textbf{31\%} relative improvement in the weighted Kendall, providing a robust, training-free proxy for efficient model selection without the cost of fine-tuning. The code will be made publicly available upon acceptance.