Nonlinearity as Rank: Generative Low-Rank Adapter with Radial Basis Functions

TL;DR

GenLoRA introduces a nonlinear basis vector generation method using RBFs to improve parameter efficiency and performance in low-rank adaptation for neural networks.

Contribution

It replaces explicit low-rank basis vectors with lightweight RBF-based nonlinear generation, enabling higher effective ranks with fewer parameters.

Findings

GenLoRA achieves higher effective ranks under smaller parameter budgets.

It outperforms standard LoRA in multiple datasets and architectures.

GenLoRA demonstrates superior fine-tuning performance.

Abstract

Low-rank adaptation (LoRA) approximates the update of a pretrained weight matrix using the product of two low-rank matrices. However, standard LoRA follows an explicit-rank paradigm, where increasing model capacity requires adding more rows or columns (i.e., basis vectors) to the low-rank matrices, leading to substantial parameter growth. In this paper, we find that these basis vectors exhibit significant parameter redundancy and can be compactly represented by lightweight nonlinear functions. Therefore, we propose Generative Low-Rank Adapter (GenLoRA), which replaces explicit basis vector storage with nonlinear basis vector generation. Specifically, GenLoRA maintains a latent vector for each low-rank matrix and employs a set of lightweight radial basis functions (RBFs) to synthesize the basis vectors. Each RBF requires far fewer parameters than an explicit basis vector, enabling higher parameter efficiency in GenLoRA. Extensive experiments across multiple datasets and architectures show that GenLoRA attains higher effective LoRA ranks under smaller parameter budgets, resulting in superior fine-tuning performance. The code is available at https://anonymous.4open.science/r/GenLoRA.