MoPEFT: A Mixture-of-PEFTs for the Segment Anything Model

TL;DR

This paper introduces MoPEFT, a dynamic mixture-of-PEFTs framework for fine-tuning the Segment Anything Model, which adaptively selects the best PEFT methods for different tasks, improving performance on the MESS benchmark.

Contribution

The paper proposes a novel MoPEFT framework that combines multiple PEFT techniques and learns to activate the most suitable ones for specific tasks, enhancing fine-tuning effectiveness.

Findings

MoPEFT outperforms other fine-tuning methods on the MESS benchmark.

Dynamic selection of PEFT techniques improves model adaptation.

The framework demonstrates consistent performance gains across tasks.

Abstract

The emergence of foundation models, such as the Segment Anything Model (SAM), has sparked interest in Parameter-Efficient Fine-Tuning (PEFT) methods that tailor these large models to application domains outside their training data. However, different PEFT techniques modify the representation of a model differently, making it a non-trivial task to select the most appropriate method for the domain of interest. We propose a new framework, Mixture-of-PEFTs methods (MoPEFT), that is inspired by traditional Mixture-of-Experts (MoE) methodologies and is utilized for fine-tuning SAM. Our MoPEFT framework incorporates three different PEFT techniques as submodules and dynamically learns to activate the ones that are best suited for a given data-task setup. We test our method on the Segment Anything Model and show that MoPEFT consistently outperforms other fine-tuning methods on the MESS benchmark.