Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities

TL;DR

This paper introduces UniME, a two-stage architecture combining a unified ViT encoder and modality-specific CNN encoders, to improve brain tumor segmentation with missing MRI modalities, demonstrating superior performance on BraTS datasets.

Contribution

The novel two-stage UniME framework effectively handles missing modalities by decoupling representation learning from segmentation, outperforming prior methods on BraTS benchmarks.

Findings

UniME achieves higher segmentation accuracy with missing modalities.

The unified representation is robust across different missing modality scenarios.

Experimental results on BraTS 2023 and 2024 validate the method's effectiveness.

Abstract

Multimodal MRI offers complementary information for brain tumor segmentation, but clinical scans often lack one or more modalities, which degrades segmentation performance. In this paper, we propose UniME (Uni-Encoder Meets Multi-Encoders), a two-stage heterogeneous method for brain tumor segmentation with missing modalities that reconciles the trade-offs among fine-grained structure capture, cross-modal complementarity modeling, and exploitation of available modalities. The idea is to decouple representation learning from segmentation via a two-stage heterogeneous architecture. Stage 1 pretrains a single ViT Uni-Encoder with masked image modeling to establish a unified representation robust to missing modalities. Stage 2 adds modality-specific CNN Multi-Encoders to extract high-resolution, multi-scale, fine-grained features. We fuse these features with the global representation to produce precise segmentations. Experiments on BraTS 2023 and BraTS 2024 show that UniME outperforms previous methods under incomplete multi-modal scenarios. The code is available at https://github.com/Hooorace-S/UniME