Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs

TL;DR

This paper introduces LoRA Recycle, a novel framework enabling visual foundation models to perform few-shot learning without fine-tuning by reusing pre-tuned LoRAs through meta-learning and surrogate data, achieving rapid adaptation.

Contribution

It is the first to reuse diverse pre-tuned LoRAs for tuning-free few-shot adaptation in VFMs, utilizing meta-learning with surrogate data and a double-efficient training mechanism.

Findings

Outperforms existing methods on multiple few-shot classification benchmarks.

Enables VFMs to adapt in a single forward pass without fine-tuning.

Accelerates meta-training while maintaining or improving performance.

Abstract

Large Language Models (LLMs) such as ChatGPT demonstrate strong few-shot adaptability without requiring fine-tuning, positioning them ideal for data-limited and real-time applications. However, this adaptability has not yet been replicated in current Visual Foundation Models (VFMs), which require explicit fine-tuning with sufficient tuning data. Besides, the pretraining-finetuning paradigm has led to the surge of numerous task-specific modular components, such as Low-Rank Adaptation (LoRA). For the first time, we explore the potential of reusing diverse pre-tuned LoRAs without accessing their original training data, to achieve tuning-free few-shot adaptation in VFMs. Our framework, LoRA Recycle, distills a meta-LoRA from diverse pre-tuned LoRAs with a meta-learning objective, using surrogate data generated inversely from pre-tuned LoRAs themselves. The VFM, once equipped with the meta-LoRA, is empowered to solve new few-shot tasks in a single forward pass, akin to the in-context learning of LLMs. Additionally, we incorporate a double-efficient mechanism tailored to our framework, significantly accelerating the meta-training process while maintaining or even improving performance. Extensive experiments across various few-shot classification benchmarks across both in- and cross-domain scenarios demonstrate the superiority of our framework.