Sub-Region-Aware Modality Fusion and Adaptive Prompting for Multi-Modal Brain Tumor Segmentation

TL;DR

This paper presents a novel framework that enhances multi-modal brain tumor segmentation by using sub-region-aware attention and adaptive prompting, significantly improving accuracy on challenging tumor sub-regions.

Contribution

It introduces sub-region-aware modality attention and adaptive prompt engineering to better fuse multi-modal data and adapt foundation models for medical image segmentation.

Findings

Outperforms baseline methods on BraTS 2020 dataset

Significantly improves segmentation of necrotic core

Demonstrates effective multi-modal fusion and adaptation

Abstract

The successful adaptation of foundation models to multi-modal medical imaging is a critical yet unresolved challenge. Existing models often struggle to effectively fuse information from multiple sources and adapt to the heterogeneous nature of pathological tissues. To address this, we introduce a novel framework for adapting foundation models to multi-modal medical imaging, featuring two key technical innovations: sub-region-aware modality attention and adaptive prompt engineering. The attention mechanism enables the model to learn the optimal combination of modalities for each tumor sub-region, while the adaptive prompting strategy leverages the inherent capabilities of foundation models to refine segmentation accuracy. We validate our framework on the BraTS 2020 brain tumor segmentation dataset, demonstrating that our approach significantly outperforms baseline methods, particularly in the challenging necrotic core sub-region. Our work provides a principled and effective approach to multi-modal fusion and prompting, paving the way for more accurate and robust foundation model-based solutions in medical imaging.