Multimodal Fusion at Three Tiers: Physics-Driven Data Generation and Vision-Language Guidance for Brain Tumor Segmentation

TL;DR

This paper introduces a three-tier multimodal fusion architecture for brain tumor segmentation that integrates physical modeling, Transformer-based feature fusion, and semantic guidance from GPT-4V, achieving state-of-the-art accuracy on multiple datasets.

Contribution

The novel three-tier fusion framework combines physical data augmentation, multi-modal feature fusion, and semantic guidance, advancing brain tumor segmentation accuracy and boundary localization.

Findings

Achieved Dice scores of 0.8665, 0.9014, and 0.8912 on BraTS 2020, 2021, and 2023 datasets.

Reduced Hausdorff Distance by an average of 6.57 mm compared to baseline.

Validated effectiveness across multiple datasets and modalities.

Abstract

Accurate brain tumor segmentation is crucial for neuro-oncology diagnosis and treatment planning. Deep learning methods have made significant progress, but automatic segmentation still faces challenges, including tumor morphological heterogeneity and complex three-dimensional spatial relationships. This paper proposes a three-tier fusion architecture that achieves precise brain tumor segmentation. The method processes information progressively at the pixel, feature, and semantic levels. At the pixel level, physical modeling extends magnetic resonance imaging (MRI) to multimodal data, including simulated ultrasound and synthetic computed tomography (CT). At the feature level, the method performs Transformer-based cross-modal feature fusion through multi-teacher collaborative distillation, integrating three expert teachers (MRI, US, CT). At the semantic level, clinical textual knowledge generated by GPT-4V is transformed into spatial guidance signals using CLIP contrastive learning and Feature-wise Linear Modulation (FiLM). These three tiers together form a complete processing chain from data augmentation to feature extraction to semantic guidance. We validated the method on the Brain Tumor Segmentation (BraTS) 2020, 2021, and 2023 datasets. The model achieves average Dice coefficients of 0.8665, 0.9014, and 0.8912 on the three datasets, respectively, and reduces the 95% Hausdorff Distance (HD95) by an average of 6.57 millimeters compared with the baseline. This method provides a new paradigm for precise tumor segmentation and boundary localization.