Diff4MMLiTS: Advanced Multimodal Liver Tumor Segmentation via Diffusion-Based Image Synthesis and Alignment

TL;DR

Diff4MMLiTS introduces a novel four-stage pipeline for multimodal liver tumor segmentation that synthesizes aligned multimodal CT images with tumors, overcoming registration challenges in real-world clinical data.

Contribution

The paper proposes a diffusion-based image synthesis method that enables effective multimodal liver tumor segmentation without requiring strictly registered data.

Findings

Outperforms existing multimodal segmentation methods

Demonstrates robustness on public and internal datasets

Eliminates the need for aligned multimodal images

Abstract

Multimodal learning has been demonstrated to enhance performance across various clinical tasks, owing to the diverse perspectives offered by different modalities of data. However, existing multimodal segmentation methods rely on well-registered multimodal data, which is unrealistic for real-world clinical images, particularly for indistinct and diffuse regions such as liver tumors. In this paper, we introduce Diff4MMLiTS, a four-stage multimodal liver tumor segmentation pipeline: pre-registration of the target organs in multimodal CTs; dilation of the annotated modality's mask and followed by its use in inpainting to obtain multimodal normal CTs without tumors; synthesis of strictly aligned multimodal CTs with tumors using the latent diffusion model based on multimodal CT features and randomly generated tumor masks; and finally, training the segmentation model, thus eliminating the need for strictly aligned multimodal data. Extensive experiments on public and internal datasets demonstrate the superiority of Diff4MMLiTS over other state-of-the-art multimodal segmentation methods.