RS-Net: Regression-Segmentation 3D CNN for Synthesis of Full Resolution Missing Brain MRI in the Presence of Tumours

TL;DR

RS-Net is a 3D CNN that jointly synthesizes missing MRI scans and segments tumours, improving accuracy and providing uncertainty estimates, aiding clinical diagnosis and analysis.

Contribution

This paper introduces a novel end-to-end 3D CNN that simultaneously performs MRI synthesis and tumour segmentation, focusing on tumour regions for improved accuracy.

Findings

Outperforms state-of-the-art in PSNR and other metrics

Maintains segmentation accuracy when replacing real MRI with synthesized images

Provides voxel-wise uncertainty estimates for synthesized volumes

Abstract

Accurate synthesis of a full 3D MR image containing tumours from available MRI (e.g. to replace an image that is currently unavailable or corrupted) would provide a clinician as well as downstream inference methods with important complementary information for disease analysis. In this paper, we present an end-to-end 3D convolution neural network that takes a set of acquired MR image sequences (e.g. T1, T2, T1ce) as input and concurrently performs (1) regression of the missing full resolution 3D MRI (e.g. FLAIR) and (2) segmentation of the tumour into subtypes (e.g. enhancement, core). The hypothesis is that this would focus the network to perform accurate synthesis in the area of the tumour. Experiments on the BraTS 2015 and 2017 datasets [1] show that: (1) the proposed method gives better performance than state-of-the-art methods in terms of established global evaluation metrics (e.g. PSNR), (2) replacing real MR volumes with the synthesized MRI does not lead to significant degradation in tumour and sub-structure segmentation accuracy. The system further provides uncertainty estimates based on Monte Carlo (MC) dropout [11] for the synthesized volume at each voxel, permitting quantification of the system's confidence in the output at each location.