SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules

TL;DR

SuperLoRA is a flexible, unified framework that extends low-rank adaptation techniques, enabling efficient fine-tuning of large models with superior performance, especially when parameter budgets are extremely limited.

Contribution

It introduces a generalized SuperLoRA framework that unifies and extends various LoRA variants through novel hyper-parameter settings and techniques.

Findings

Superior transfer learning performance in few-parameter regimes

High flexibility compared to existing LoRA variants

Effective in fine-tuning large language and diffusion models

Abstract

Low-rank adaptation (LoRA) and its variants are widely employed in fine-tuning large models, including large language models for natural language processing and diffusion models for computer vision. This paper proposes a generalized framework called SuperLoRA that unifies and extends different LoRA variants, which can be realized under different hyper-parameter settings. Introducing grouping, folding, shuffling, projecting, and tensor factoring, SuperLoRA offers high flexibility compared with other LoRA variants and demonstrates superior performance for transfer learning tasks especially in the extremely few-parameter regimes.