Transformer Utilization in Medical Image Segmentation Networks

TL;DR

This paper investigates the effectiveness of Transformer components in medical image segmentation by replacing them with linear operators, revealing insights into their role and design considerations.

Contribution

It introduces Transformer Ablations to quantify Transformer effectiveness and analyzes the impact of design choices in medical image segmentation networks.

Findings

Transformer-learnt representations are replaceable.

Transformer capacity alone is insufficient for effectiveness.

Explicit feature hierarchies outperform self-attention modules.

Abstract

Owing to success in the data-rich domain of natural images, Transformers have recently become popular in medical image segmentation. However, the pairing of Transformers with convolutional blocks in varying architectural permutations leaves their relative effectiveness to open interpretation. We introduce Transformer Ablations that replace the Transformer blocks with plain linear operators to quantify this effectiveness. With experiments on 8 models on 2 medical image segmentation tasks, we explore -- 1) the replaceable nature of Transformer-learnt representations, 2) Transformer capacity alone cannot prevent representational replaceability and works in tandem with effective design, 3) The mere existence of explicit feature hierarchies in transformer blocks is more beneficial than accompanying self-attention modules, 4) Major spatial downsampling before Transformer modules should be used with caution.