TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation

TL;DR

TransFuse introduces a parallel architecture combining Transformers and CNNs for medical image segmentation, achieving state-of-the-art results with fewer parameters and faster inference by effectively capturing global and local features.

Contribution

The paper presents TransFuse, a novel parallel network architecture with a BiFusion module for efficient multi-level feature fusion, improving global context modeling and detail preservation.

Findings

Achieves state-of-the-art results on multiple medical segmentation tasks.

Reduces model parameters and increases inference speed.

Effectively captures both global dependencies and local details.

Abstract

Medical image segmentation - the prerequisite of numerous clinical needs - has been significantly prospered by recent advances in convolutional neural networks (CNNs). However, it exhibits general limitations on modeling explicit long-range relation, and existing cures, resorting to building deep encoders along with aggressive downsampling operations, leads to redundant deepened networks and loss of localized details. Hence, the segmentation task awaits a better solution to improve the efficiency of modeling global contexts while maintaining a strong grasp of low-level details. In this paper, we propose a novel parallel-in-branch architecture, TransFuse, to address this challenge. TransFuse combines Transformers and CNNs in a parallel style, where both global dependency and low-level spatial details can be efficiently captured in a much shallower manner. Besides, a novel fusion technique - BiFusion module is created to efficiently fuse the multi-level features from both branches. Extensive experiments demonstrate that TransFuse achieves the newest state-of-the-art results on both 2D and 3D medical image sets including polyp, skin lesion, hip, and prostate segmentation, with significant parameter decrease and inference speed improvement.