TubeMLLM: A Foundation Model for Topology Knowledge Exploration in Vessel-like Anatomy

TL;DR

TubeMLLM is a novel foundation model that enhances topology-aware perception of vessel-like anatomy in medical images by integrating topological priors and multimodal learning, achieving state-of-the-art results across diverse datasets.

Contribution

The paper introduces TubeMLLM, a unified multimodal foundation model with explicit topological priors and an adaptive loss, advancing topology consistency and zero-shot generalization in medical vessel analysis.

Findings

Achieves state-of-the-art out-of-distribution performance

Significantly reduces topological errors in color fundus images

Excels in zero-shot transfer to unseen X-ray angiography

Abstract

Modeling medical vessel-like anatomy is challenging due to its intricate topology and sensitivity to dataset shifts. Consequently, task-specific models often suffer from topological inconsistencies, including artificial disconnections and spurious merges. Motivated by the promise of multimodal large language models (MLLMs) for zero-shot generalization, we propose TubeMLLM, a unified foundation model that couples structured understanding with controllable generation for medical vessel-like anatomy. By integrating topological priors through explicit natural language prompting and aligning them with visual representations in a shared-attention architecture, TubeMLLM significantly enhances topology-aware perception. Furthermore, we construct TubeMData, a pionner multimodal benchmark comprising comprehensive topology-centric tasks, and introduce an adaptive loss weighting strategy to emphasize topology-critical regions during training. Extensive experiments on fifteen diverse datasets demonstrate our superiority. Quantitatively, TubeMLLM achieves state-of-the-art out-of-distribution performance, substantially reducing global topological discrepancies on color fundus photography (decreasing the $\beta_{0}$ number error from 37.42 to 8.58 compared to baselines). Notably, TubeMLLM exhibits exceptional zero-shot cross-modality transferring ability on unseen X-ray angiography, achieving a Dice score of 67.50% while significantly reducing the $\beta_{0}$ error to 1.21. TubeMLLM also maintains robustness against degradations such as blur, noise, and low resolution. Furthermore, in topology-aware understanding tasks, the model achieves 97.38% accuracy in evaluating mask topological quality, significantly outperforming standard vision-language baselines.