SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning

TL;DR

SHERL is a novel memory-efficient transfer learning method that enhances model adaptation by consolidating intermediate features and utilizing minimal pre-trained layers, achieving high accuracy with reduced memory usage.

Contribution

It introduces SHERL, a new two-stage strategy that combines anti-redundancy and minimal layer utilization to improve resource-limited transfer learning performance.

Findings

Outperforms existing PETL methods on vision-and-language and language tasks.

Reduces memory usage during fine-tuning without sacrificing accuracy.

Effectively balances parameter efficiency and memory constraints.

Abstract

Parameter-efficient transfer learning (PETL) has emerged as a flourishing research field for adapting large pre-trained models to downstream tasks, greatly reducing trainable parameters while grappling with memory challenges during fine-tuning. To address it, memory-efficient series (METL) avoid backpropagating gradients through the large backbone. However, they compromise by exclusively relying on frozen intermediate outputs and limiting the exhaustive exploration of prior knowledge from pre-trained models. Moreover, the dependency and redundancy between cross-layer features are frequently overlooked, thereby submerging more discriminative representations and causing an inherent performance gap (vs. conventional PETL methods). Hence, we propose an innovative METL strategy called SHERL for resource-limited scenarios to decouple the entire adaptation into two successive and complementary processes. In the early route, intermediate outputs are consolidated via an anti-redundancy operation, enhancing their compatibility for subsequent interactions; thereby in the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead and regulate these fairly flexible features into more adaptive and powerful representations for new domains. Extensive ablations on vision-and-language and language-only tasks show that SHERL combines the strengths of both parameter and memory-efficient techniques, performing on-par or better across diverse architectures with lower memory during fine-tuning. Our code is publicly available at: https://github.com/Paranioar/SHERL.