A Data-scalable Transformer for Medical Image Segmentation: Architecture, Model Efficiency, and Benchmark

TL;DR

MedFormer is a scalable Transformer architecture for 3D medical image segmentation that effectively learns from limited data, generalizes across diverse tasks, and outperforms CNNs and existing Transformers on multiple datasets.

Contribution

The paper introduces MedFormer, a novel data-scalable Transformer with hierarchical modeling and multi-scale feature fusion, designed specifically for medical image segmentation without requiring pre-training.

Findings

Outperforms CNNs and vision Transformers on seven public datasets

Effective across multiple modalities like CT and MRI

Handles diverse targets including organs, tissues, and tumors

Abstract

Transformers have demonstrated remarkable performance in natural language processing and computer vision. However, existing vision Transformers struggle to learn from limited medical data and are unable to generalize on diverse medical image tasks. To tackle these challenges, we present MedFormer, a data-scalable Transformer designed for generalizable 3D medical image segmentation. Our approach incorporates three key elements: a desirable inductive bias, hierarchical modeling with linear-complexity attention, and multi-scale feature fusion that integrates spatial and semantic information globally. MedFormer can learn across tiny- to large-scale data without pre-training. Comprehensive experiments demonstrate MedFormer's potential as a versatile segmentation backbone, outperforming CNNs and vision Transformers on seven public datasets covering multiple modalities (e.g., CT and MRI) and various medical targets (e.g., healthy organs, diseased tissues, and tumors). We provide public access to our models and evaluation pipeline, offering solid baselines and unbiased comparisons to advance a wide range of downstream clinical applications.