Joint Sequence Learning and Cross-Modality Convolution for 3D Biomedical Segmentation

TL;DR

This paper introduces a novel deep learning framework for 3D biomedical segmentation that effectively integrates multiple MRI modalities using cross-modality convolution and models sequential slices with convolutional LSTM, achieving superior results.

Contribution

It proposes a joint learning architecture combining cross-modality convolution and convolutional LSTM for improved multi-modal 3D biomedical segmentation.

Findings

Outperforms state-of-the-art methods on BRATS-2015 dataset

Effectively leverages multi-modal MRI data

Handles label imbalance with re-weighting and two-phase training

Abstract

Deep learning models such as convolutional neural net- work have been widely used in 3D biomedical segmentation and achieve state-of-the-art performance. However, most of them often adapt a single modality or stack multiple modalities as different input channels. To better leverage the multi- modalities, we propose a deep encoder-decoder structure with cross-modality convolution layers to incorporate different modalities of MRI data. In addition, we exploit convolutional LSTM to model a sequence of 2D slices, and jointly learn the multi-modalities and convolutional LSTM in an end-to-end manner. To avoid converging to the certain labels, we adopt a re-weighting scheme and two-phase training to handle the label imbalance. Experimental results on BRATS-2015 show that our method outperforms state-of-the-art biomedical segmentation approaches.