SAMora: Enhancing SAM through Hierarchical Self-Supervised Pre-Training for Medical Images

TL;DR

SAMora enhances the Segment Anything Model for medical image segmentation by leveraging hierarchical self-supervised learning and a novel fusion module, significantly improving performance especially with limited labeled data.

Contribution

The paper introduces SAMora, a framework that captures hierarchical medical knowledge using self-supervised objectives and a hierarchical fusion module, compatible with various SAM variants.

Findings

Outperforms existing SAM variants on multiple datasets.

Achieves state-of-the-art results in few-shot and full supervision.

Reduces fine-tuning epochs by 90%.

Abstract

The Segment Anything Model (SAM) has demonstrated significant potential in medical image segmentation. Yet, its performance is limited when only a small amount of labeled data is available, while there is abundant valuable yet often overlooked hierarchical information in medical data. To address this limitation, we draw inspiration from self-supervised learning and propose SAMora, an innovative framework that captures hierarchical medical knowledge by applying complementary self-supervised learning objectives at the image, patch, and pixel levels. To fully exploit the complementarity of hierarchical knowledge within LoRAs, we introduce HL-Attn, a hierarchical fusion module that integrates multi-scale features while maintaining their distinct characteristics. SAMora is compatible with various SAM variants, including SAM2, SAMed, and H-SAM. Experimental results on the Synapse, LA, and PROMISE12 datasets demonstrate that SAMora outperforms existing SAM variants. It achieves state-of-the-art performance in both few-shot and fully supervised settings while reducing fine-tuning epochs by 90%. The code is available at https://github.com/ShChen233/SAMora.