ROSA: Random Subspace Adaptation for Efficient Fine-Tuning

TL;DR

ROSA is a novel parameter-efficient fine-tuning method that adapts large models by selecting subspaces, outperforming previous methods like LoRA without adding inference latency, especially effective in NLP tasks.

Contribution

ROSA introduces a flexible subspace adaptation approach that surpasses LoRA in performance and expressiveness while maintaining zero inference overhead.

Findings

ROSA outperforms LoRA on almost all GLUE tasks.

ROSA achieves better results on NLP generation tasks.

ROSA is more expressive than LoRA without extra memory during inference.

Abstract

Model training requires significantly more memory, compared with inference. Parameter efficient fine-tuning (PEFT) methods provide a means of adapting large models to downstream tasks using less memory. However, existing methods such as adapters, prompt tuning or low-rank adaptation (LoRA) either introduce latency overhead at inference time or achieve subpar downstream performance compared with full fine-tuning. In this work we propose Random Subspace Adaptation (ROSA), a method that outperforms previous PEFT methods by a significant margin, while maintaining a zero latency overhead during inference time. In contrast to previous methods, ROSA is able to adapt subspaces of arbitrarily large dimension, better approximating full-finetuning. We demonstrate both theoretically and experimentally that this makes ROSA strictly more expressive than LoRA, without consuming additional memory during runtime. As PEFT methods are especially useful in the natural language processing domain, where models operate on scales that make full fine-tuning very expensive, we evaluate ROSA in two common NLP scenarios: natural language generation (NLG) and natural language understanding (NLU) with GPT-2 and RoBERTa, respectively. We show that on almost every GLUE task ROSA outperforms LoRA by a significant margin, while also outperforming LoRA on NLG tasks. Our code is available at https://github.com/rosa-paper/rosa