Spatiotemporal Modeling Encounters 3D Medical Image Analysis: Slice-Shift UNet with Multi-View Fusion

TL;DR

This paper introduces Slice Shift UNet, a 2D-based model that efficiently captures 3D features in medical imaging by multi-view learning and slice shifting, achieving comparable accuracy to 3D CNNs with lower computational costs.

Contribution

The paper presents a novel 2D CNN architecture, Slice Shift UNet, that encodes 3D information efficiently for medical image segmentation, inspired by video recognition techniques.

Findings

Achieves state-of-the-art performance on AMOS and BTCV datasets.

More efficient than traditional 3D CNNs with similar accuracy.

Effective multi-view feature learning with slice shifting mechanism.

Abstract

As a fundamental part of computational healthcare, Computer Tomography (CT) and Magnetic Resonance Imaging (MRI) provide volumetric data, making the development of algorithms for 3D image analysis a necessity. Despite being computationally cheap, 2D Convolutional Neural Networks can only extract spatial information. In contrast, 3D CNNs can extract three-dimensional features, but they have higher computational costs and latency, which is a limitation for clinical practice that requires fast and efficient models. Inspired by the field of video action recognition we propose a new 2D-based model dubbed Slice SHift UNet (SSH-UNet) which encodes three-dimensional features at 2D CNN's complexity. More precisely multi-view features are collaboratively learned by performing 2D convolutions along the three orthogonal planes of a volume and imposing a weights-sharing mechanism. The third dimension, which is neglected by the 2D convolution, is reincorporated by shifting a portion of the feature maps along the slices' axis. The effectiveness of our approach is validated in Multi-Modality Abdominal Multi-Organ Segmentation (AMOS) and Multi-Atlas Labeling Beyond the Cranial Vault (BTCV) datasets, showing that SSH-UNet is more efficient while on par in performance with state-of-the-art architectures.