1%>100%: High-Efficiency Visual Adapter with Complex Linear Projection Optimization

TL;DR

This paper introduces CoLin, a low-parameter, high-efficiency adapter for vision models that significantly reduces fine-tuning costs while outperforming traditional methods across various vision tasks.

Contribution

The paper proposes a novel low-rank complex adapter with 1% parameters and a tailored loss to address convergence issues, advancing efficient vision model adaptation.

Findings

CoLin outperforms full fine-tuning and delta-tuning methods.

Achieves high efficiency with only 1% parameters.

Demonstrates effectiveness across multiple vision tasks.

Abstract

Deploying vision foundation models typically relies on efficient adaptation strategies, whereas conventional full fine-tuning suffers from prohibitive costs and low efficiency. While delta-tuning has proven effective in boosting the performance and efficiency of LLMs during adaptation, its advantages cannot be directly transferred to the fine-tuning pipeline of vision foundation models. To push the boundaries of adaptation efficiency for vision tasks, we propose an adapter with Complex Linear Projection Optimization (CoLin). For architecture, we design a novel low-rank complex adapter that introduces only about 1% parameters to the backbone. For efficiency, we theoretically prove that low-rank composite matrices suffer from severe convergence issues during training, and address this challenge with a tailored loss. Extensive experiments on object detection, segmentation, image classification, and rotated object detection (remote sensing scenario) demonstrate that CoLin outperforms both full fine-tuning and classical delta-tuning approaches with merely 1% parameters for the first time, providing a novel and efficient solution for deployment of vision foundation models. We release the code on https://github.com/DongshuoYin/CoLin.