Pyramid Medical Transformer for Medical Image Segmentation

TL;DR

This paper introduces Pyramid Medical Transformer (PMTrans), a novel multi-scale attention model that combines CNN features and adaptive partitioning to improve medical image segmentation efficiency and accuracy.

Contribution

The paper proposes a pyramidal architecture with multi-resolution attention and adaptive partitioning, addressing computational costs and relation modeling limitations of existing transformers in medical imaging.

Findings

PMTrans outperforms recent CNN and transformer models on multiple datasets.

The multi-scale approach effectively captures long-range dependencies.

Adaptive partitioning improves relation modeling and computational efficiency.

Abstract

Deep neural networks have been a prevailing technique in the field of medical image processing. However, the most popular convolutional neural networks (CNNs) based methods for medical image segmentation are imperfect because they model long-range dependencies by stacking layers or enlarging filters. Transformers and the self-attention mechanism are recently proposed to effectively learn long-range dependencies by modeling all pairs of word-to-word attention regardless of their positions. The idea has also been extended to the computer vision field by creating and treating image patches as embeddings. Considering the computation complexity for whole image self-attention, current transformer-based models settle for a rigid partitioning scheme that potentially loses informative relations. Besides, current medical transformers model global context on full resolution images, leading to unnecessary computation costs. To address these issues, we developed a novel method to integrate multi-scale attention and CNN feature extraction using a pyramidal network architecture, namely Pyramid Medical Transformer (PMTrans). The PMTrans captured multi-range relations by working on multi-resolution images. An adaptive partitioning scheme was implemented to retain informative relations and to access different receptive fields efficiently. Experimental results on three medical image datasets (gland segmentation, MoNuSeg, and HECKTOR datasets) showed that PMTrans outperformed the latest CNN-based and transformer-based models for medical image segmentation.