A Simple and Robust Framework for Cross-Modality Medical Image Segmentation applied to Vision Transformers

TL;DR

This paper introduces a simple, robust framework for cross-modality medical image segmentation using a single conditional model with adaptive normalization, significantly improving performance across different imaging modalities.

Contribution

It proposes a novel framework that enables effective cross-modality segmentation with a single model, eliminating the need for registered images or synthetic data generation.

Findings

Outperforms existing methods on the Multi-Modality Whole Heart Segmentation Challenge.

The Conditional Vision Transformer (C-ViT) encoder improves segmentation accuracy by up to 6.87% Dice.

The framework works with non-registered, mixed modality data.

Abstract

When it comes to clinical images, automatic segmentation has a wide variety of applications and a considerable diversity of input domains, such as different types of Magnetic Resonance Images (MRIs) and Computerized Tomography (CT) scans. This heterogeneity is a challenge for cross-modality algorithms that should equally perform independently of the input image type fed to them. Often, segmentation models are trained using a single modality, preventing generalization to other types of input data without resorting to transfer learning techniques. Furthermore, the multi-modal or cross-modality architectures proposed in the literature frequently require registered images, which are not easy to collect in clinical environments, or need additional processing steps, such as synthetic image generation. In this work, we propose a simple framework to achieve fair image segmentation of multiple modalities using a single conditional model that adapts its normalization layers based on the input type, trained with non-registered interleaved mixed data. We show that our framework outperforms other cross-modality segmentation methods, when applied to the same 3D UNet baseline model, on the Multi-Modality Whole Heart Segmentation Challenge. Furthermore, we define the Conditional Vision Transformer (C-ViT) encoder, based on the proposed cross-modality framework, and we show that it brings significant improvements to the resulting segmentation, up to 6.87\% of Dice accuracy, with respect to its baseline reference. The code to reproduce our experiments and the trained model weights are available at https://github.com/matteo-bastico/MI-Seg.