AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping

TL;DR

AuroRA introduces a nonlinear mapping in LoRA to overcome its low-rank bottleneck, enabling efficient fine-tuning with fewer parameters while maintaining or surpassing full fine-tuning performance across NLP and CV tasks.

Contribution

The paper proposes AuroRA, a nonlinear extension of LoRA that enhances expressiveness and reduces parameter overhead through an Adaptive Nonlinear Layer, with theoretical guarantees and extensive empirical validation.

Findings

Matches or surpasses full fine-tuning performance with 6.18% to 25% of LoRA parameters.

Outperforms other PEFT methods by up to 10.88% in NLP and CV tasks.

Demonstrates robust performance across various rank configurations.

Abstract

Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method validated across NLP and CV domains. However, LoRA faces an inherent low-rank bottleneck: narrowing its performance gap with full finetuning requires increasing the rank of its parameter matrix, resulting in significant parameter overhead. Recent linear LoRA variants have attempted to enhance expressiveness by introducing additional linear mappings; however, their composition remains inherently linear and fails to fundamentally improve LoRA's representational capacity. To address this limitation, we propose AuroRA, which incorporates an Adaptive Nonlinear Layer (ANL) between two linear projectors to capture fixed and learnable nonlinearities. This combination forms an MLP-like structure with a compressed rank, enabling flexible and precise approximation of diverse target functions while theoretically guaranteeing lower approximation errors and bounded gradients. Extensive experiments on 22 datasets and 6 pretrained models demonstrate that AuroRA: (I) not only matches or surpasses full fine-tuning performance with only 6.18% ~ 25% of LoRA's parameters but also (II) outperforms competitive PEFT methods by up to 10.88% in both NLP and CV tasks, and (III) exhibits robust performance across various rank configurations.