Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model

TL;DR

This paper introduces a novel parameter-efficient fine-tuning method with cross-block orchestration for the Segment Anything Model, significantly improving segmentation performance in new scenarios with minimal additional parameters.

Contribution

It proposes a cross-block communication mechanism and intra-block enhancement to enable effective adaptation of SAM with limited parameter adjustments.

Findings

Consistently improves segmentation accuracy on diverse benchmarks.

Achieves significant performance gains with only around 1K extra parameters.

Demonstrates effectiveness in adapting to various downstream scenarios.

Abstract

Parameter-efficient fine-tuning (PEFT) is an effective methodology to unleash the potential of large foundation models in novel scenarios with limited training data. In the computer vision community, PEFT has shown effectiveness in image classification, but little research has studied its ability for image segmentation. Fine-tuning segmentation models usually require a heavier adjustment of parameters to align the proper projection directions in the parameter space for new scenarios. This raises a challenge to existing PEFT algorithms, as they often inject a limited number of individual parameters into each block, which prevents substantial adjustment of the projection direction of the parameter space due to the limitation of Hidden Markov Chain along blocks. In this paper, we equip PEFT with a cross-block orchestration mechanism to enable the adaptation of the Segment Anything Model (SAM) to various downstream scenarios. We introduce a novel inter-block communication module, which integrates a learnable relation matrix to facilitate communication among different coefficient sets of each PEFT block's parameter space. Moreover, we propose an intra-block enhancement module, which introduces a linear projection head whose weights are generated from a hyper-complex layer, further enhancing the impact of the adjustment of projection directions on the entire parameter space. Extensive experiments on diverse benchmarks demonstrate that our proposed approach consistently improves the segmentation performance significantly on novel scenarios with only around 1K additional parameters.