Selective Feature Adapter for Dense Vision Transformers

TL;DR

This paper introduces Selective Feature Adapter (SFA), a novel method for fine-tuning dense vision transformers efficiently by selectively adapting features, achieving state-of-the-art performance with fewer trainable parameters across various vision tasks.

Contribution

The paper proposes a dual adapter approach with external and internal adapters, automatically discovering task-important parameters to optimize parameter efficiency and performance.

Findings

SFA achieves state-of-the-art results under parameter budget constraints.

SFA outperforms other adapter methods on dense vision tasks.

Dual adapters provide the best trade-off between performance and parameter efficiency.

Abstract

Fine-tuning pre-trained transformer models, e.g., Swin Transformer, are successful in numerous downstream for dense prediction vision tasks. However, one major issue is the cost/storage of their huge amount of parameters, which becomes increasingly challenging to handle with the growing amount of vision tasks. In this paper, we propose an effective approach to alleviate the issue, namely selective feature adapter (SFA). It achieves state-of-the-art (SoTA) performance under any given budget of trainable parameters, and demonstrates comparable or better performance than fully fine-tuned models across various dense tasks. Specifically, SFA consists of external adapters and internal adapters which are sequentially operated over a transformer model. For external adapters, we properly select the places and amount of additional multilayer perception (MLP). For internal adapters, we transform a few task-important parameters inside the transformer, which are automatically discovered through a simple yet effective lottery ticket algorithm. Our experiments show that the dual adapter module, a.k.a SFA, is essential to achieve the best trade-off on dense vision tasks, such as segmentation, detection and depth-estimation, outperforming other adapters with a single module.