Task-Aware Low-Rank Adaptation of Segment Anything Model

TL;DR

This paper introduces TA-LoRA, a novel low-rank adaptation method that enables the Segment Anything Model to effectively perform multi-task image segmentation by incorporating task-specific and shared information.

Contribution

We propose TA-LoRA, a task-aware low-rank adaptation technique that modifies SAM for multi-task learning by injecting low-rank tensors into encoder layers.

Findings

TA-LoRA improves multi-task segmentation performance on benchmark datasets.

Modified SAM (mSAM) with TA-LoRA outperforms baseline models.

The method effectively captures task-specific features while maintaining generalization.

Abstract

The Segment Anything Model (SAM), with its remarkable zero-shot capability, has been proven to be a powerful foundation model for image segmentation tasks, which is an important task in computer vision. However, the transfer of its rich semantic information to multiple different downstream tasks remains unexplored. In this paper, we propose the Task-Aware Low-Rank Adaptation (TA-LoRA) method, which enables SAM to work as a foundation model for multi-task learning. Specifically, TA-LoRA injects an update parameter tensor into each layer of the encoder in SAM and leverages a low-rank tensor decomposition method to incorporate both task-shared and task-specific information. Furthermore, we introduce modified SAM (mSAM) for multi-task learning where we remove the prompt encoder of SAM and use task-specific no mask embeddings and mask decoder for each task. Extensive experiments conducted on benchmark datasets substantiate the efficacy of TA-LoRA in enhancing the performance of mSAM across multiple downstream tasks.