EigenLoRAx: Recycling Adapters to Find Principal Subspaces for Resource-Efficient Adaptation and Inference

TL;DR

EigenLoRAx is a novel method that recycles pretrained adapters to efficiently adapt large models to new tasks by creating a principal subspace, reducing parameters and computational costs for training and inference.

Contribution

The paper introduces EigenLoRAx, a parameter-efficient finetuning approach that leverages existing adapters to form a shared subspace, enabling rapid adaptation with fewer resources.

Findings

Achieves strong performance across diverse domains and tasks.

Requires significantly fewer parameters and memory.

Facilitates scalable deployment in resource-constrained environments.

Abstract

The rapid growth of large models has raised concerns about their environmental impact and equity in accessibility due to significant computational costs. Low-Rank Adapters (LoRA) offer a lightweight solution for finetuning large models, resulting in an abundance of publicly available adapters tailored to diverse domains. We ask: Can these pretrained adapters be leveraged to further streamline adaptation to new tasks while addressing these challenges? We introduce EigenLoRAx, a parameter-efficient finetuning method that recycles existing adapters to create a principal subspace aligned with their shared domain knowledge which can be further augmented with orthogonal basis vectors in low-resource scenarios. This enables rapid adaptation to new tasks by learning only lightweight coefficients on the principal components of the subspace-eliminating the need to finetune entire adapters. EigenLoRAx requires significantly fewer parameters and memory, improving efficiency for both training and inference. Our method demonstrates strong performance across diverse domains and tasks, offering a scalable for edge-based applications, personalization, and equitable deployment of large models in resource-constrained environments.