DB-SAM: Delving into High Quality Universal Medical Image Segmentation

TL;DR

DB-SAM introduces a dual-branch framework combining ViT and convolutional networks with cross-attention for improved universal medical image segmentation, effectively bridging the domain gap between natural and medical images.

Contribution

The paper proposes a novel dual-branch adapted SAM framework with cross-attention mechanisms to enhance medical image segmentation performance across diverse datasets.

Findings

Achieves 8.8% absolute gain on 21 3D medical segmentation tasks

Effectively bridges domain gap between natural and medical images

Demonstrates superior performance over recent medical SAM adapters

Abstract

Recently, the Segment Anything Model (SAM) has demonstrated promising segmentation capabilities in a variety of downstream segmentation tasks. However in the context of universal medical image segmentation there exists a notable performance discrepancy when directly applying SAM due to the domain gap between natural and 2D/3D medical data. In this work, we propose a dual-branch adapted SAM framework, named DB-SAM, that strives to effectively bridge this domain gap. Our dual-branch adapted SAM contains two branches in parallel: a ViT branch and a convolution branch. The ViT branch incorporates a learnable channel attention block after each frozen attention block, which captures domain-specific local features. On the other hand, the convolution branch employs a light-weight convolutional block to extract domain-specific shallow features from the input medical image. To perform cross-branch feature fusion, we design a bilateral cross-attention block and a ViT convolution fusion block, which dynamically combine diverse information of two branches for mask decoder. Extensive experiments on large-scale medical image dataset with various 3D and 2D medical segmentation tasks reveal the merits of our proposed contributions. On 21 3D medical image segmentation tasks, our proposed DB-SAM achieves an absolute gain of 8.8%, compared to a recent medical SAM adapter in the literature. The code and model are available at https://github.com/AlfredQin/DB-SAM.