MedSAGa: Few-shot Memory Efficient Medical Image Segmentation using Gradient Low-Rank Projection in SAM

TL;DR

MedSAGa introduces a memory-efficient few-shot medical image segmentation method using Gradient Low-Rank Projection in SAM, enabling effective segmentation with reduced computational resources in resource-limited environments.

Contribution

This work presents MedSAGa, a novel approach combining Gradient Low-Rank Projection with SAM for memory-efficient, few-shot medical image segmentation, outperforming existing models in resource-constrained settings.

Findings

Achieves 66% more memory efficiency than SOTA models.

Maintains comparable segmentation performance with significantly less memory.

Demonstrates effectiveness across multiple medical imaging datasets.

Abstract

The application of large-scale models in medical image segmentation demands substantial quantities of meticulously annotated data curated by experts along with high computational resources, both of which are challenges in resource-poor settings. In this study, we present the Medical Segment Anything Model with Galore MedSAGa where we adopt the Segment Anything Model (SAM) to achieve memory-efficient, few-shot medical image segmentation by applying Gradient Low-Rank Projection GaLore to the parameters of the image encoder of SAM. Meanwhile, the weights of the prompt encoder and mask decoder undergo full parameter fine-tuning using standard optimizers. We further assess MedSAGa's few-shot learning capabilities, reporting on its memory efficiency and segmentation performance across multiple standard medical image segmentation datasets. We compare it with several baseline models, including LoRA fine-tuned SAM (SAMed) and DAE-Former. Experiments across multiple datasets and these baseline models with different number of images for fine tuning demonstrated that the GPU memory consumption of MedSAGa is significantly less than that of the baseline models, achieving an average memory efficiency of 66% more than current state-of-the-art (SOTA) models for medical image segmentation. The combination of substantially lower memory requirements and comparable to SOTA results in few-shot learning for medical image segmentation positions MedSAGa as an optimal solution for deployment in resource-constrained settings.