TR-PTS: Task-Relevant Parameter and Token Selection for Efficient Tuning

TL;DR

TR-PTS introduces a task-driven approach that selectively fine-tunes parameters and tokens in large models, significantly improving efficiency and accuracy in vision tasks compared to traditional methods.

Contribution

It proposes a novel framework combining parameter and token selection based on task relevance, utilizing Fisher Information and dynamic token merging for improved fine-tuning.

Findings

Achieves state-of-the-art results on FGVC and VTAB-1k benchmarks.

Surpasses full fine-tuning performance by 3.40% and 10.35%.

Reduces computational costs while maintaining high accuracy.

Abstract

Large pre-trained models achieve remarkable performance in vision tasks but are impractical for fine-tuning due to high computational and storage costs. Parameter-Efficient Fine-Tuning (PEFT) methods mitigate this issue by updating only a subset of parameters; however, most existing approaches are task-agnostic, failing to fully exploit task-specific adaptations, which leads to suboptimal efficiency and performance. To address this limitation, we propose Task-Relevant Parameter and Token Selection (TR-PTS), a task-driven framework that enhances both computational efficiency and accuracy. Specifically, we introduce Task-Relevant Parameter Selection, which utilizes the Fisher Information Matrix (FIM) to identify and fine-tune only the most informative parameters in a layer-wise manner, while keeping the remaining parameters frozen. Simultaneously, Task-Relevant Token Selection dynamically preserves the most informative tokens and merges redundant ones, reducing computational overhead. By jointly optimizing parameters and tokens, TR-PTS enables the model to concentrate on task-discriminative information. We evaluate TR-PTS on benchmark, including FGVC and VTAB-1k, where it achieves state-of-the-art performance, surpassing full fine-tuning by 3.40% and 10.35%, respectively. The code are available at https://github.com/synbol/TR-PTS.