Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning

TL;DR

MetaPEFT introduces a meta-learning approach to adaptively tune hyperparameters in parameter-efficient fine-tuning, significantly improving performance on remote sensing images with minimal additional parameters.

Contribution

The paper proposes MetaPEFT, a novel method that dynamically adjusts PEFT hyperparameters during fine-tuning, addressing sensitivity issues and enhancing tail-class accuracy.

Findings

MetaPEFT outperforms existing PEFT methods on multiple datasets.

It requires fewer trainable parameters while achieving state-of-the-art results.

MetaPEFT improves cross-spectral adaptation in remote sensing tasks.

Abstract

Training large foundation models from scratch for domain-specific applications is almost impossible due to data limits and long-tailed distributions -- taking remote sensing (RS) as an example. Fine-tuning natural image pre-trained models on RS images is a straightforward solution. To reduce computational costs and improve performance on tail classes, existing methods apply parameter-efficient fine-tuning (PEFT) techniques, such as LoRA and AdaptFormer. However, we observe that fixed hyperparameters -- such as intra-layer positions, layer depth, and scaling factors, can considerably hinder PEFT performance, as fine-tuning on RS images proves highly sensitive to these settings. To address this, we propose MetaPEFT, a method incorporating adaptive scalers that dynamically adjust module influence during fine-tuning. MetaPEFT dynamically adjusts three key factors of PEFT on RS images: module insertion, layer selection, and module-wise learning rates, which collectively control the influence of PEFT modules across the network. We conduct extensive experiments on three transfer-learning scenarios and five datasets in both RS and natural image domains. The results show that MetaPEFT achieves state-of-the-art performance in cross-spectral adaptation, requiring only a small amount of trainable parameters and improving tail-class accuracy significantly.