nnFormer: Interleaved Transformer for Volumetric Segmentation

TL;DR

nnFormer introduces a novel 3D transformer architecture for volumetric medical image segmentation, combining interleaved convolution and self-attention, leading to significant performance improvements over previous methods.

Contribution

The paper presents nnFormer, a new 3D transformer model that integrates local and global volume-based self-attention with skip attention in a U-Net architecture for improved segmentation.

Findings

Outperforms previous transformer-based methods on three datasets

Achieves lower HD95 and comparable DSC to nnUNet

Highly complementary to nnUNet in model ensembling

Abstract

Transformer, the model of choice for natural language processing, has drawn scant attention from the medical imaging community. Given the ability to exploit long-term dependencies, transformers are promising to help atypical convolutional neural networks to overcome their inherent shortcomings of spatial inductive bias. However, most of recently proposed transformer-based segmentation approaches simply treated transformers as assisted modules to help encode global context into convolutional representations. To address this issue, we introduce nnFormer, a 3D transformer for volumetric medical image segmentation. nnFormer not only exploits the combination of interleaved convolution and self-attention operations, but also introduces local and global volume-based self-attention mechanism to learn volume representations. Moreover, nnFormer proposes to use skip attention to replace the traditional concatenation/summation operations in skip connections in U-Net like architecture. Experiments show that nnFormer significantly outperforms previous transformer-based counterparts by large margins on three public datasets. Compared to nnUNet, nnFormer produces significantly lower HD95 and comparable DSC results. Furthermore, we show that nnFormer and nnUNet are highly complementary to each other in model ensembling.