Parameter Efficient Fine-Tuning of Segment Anything Model for Biomedical Imaging

TL;DR

This paper investigates parameter-efficient fine-tuning methods for the Segment Anything Model to improve biomedical image segmentation, reducing computational costs while maintaining high performance.

Contribution

It provides the first comprehensive study of PEFT for SAM in biomedical imaging, highlighting layer placement importance and offering a resource-efficient finetuning recipe.

Findings

Layer placement impacts PEFT efficiency more than layer type.

A practical recipe for resource-efficient finetuning is proposed.

Code for PEFT SAM is publicly available.

Abstract

Segmentation is an important analysis task for biomedical images, enabling the study of individual organelles, cells or organs. Deep learning has massively improved segmentation methods, but challenges remain in generalization to new conditions, requiring costly data annotation. Vision foundation models, such as Segment Anything Model (SAM), address this issue through improved generalization. However, these models still require finetuning on annotated data, although with less annotations, to achieve optimal results for new conditions. As a downside, they require more computational resources. This makes parameter-efficient finetuning (PEFT) relevant. We contribute the first comprehensive study of PEFT for SAM applied to biomedical images. We find that the placement of PEFT layers is more important for efficiency than the type of layer for vision transformers and we provide a recipe for resource-efficient finetuning. Our code is publicly available at https://github.com/computational-cell-analytics/peft-sam.