AFA-LoRA: Enabling Non-Linear Adaptations in LoRA with Activation Function Annealing

TL;DR

AFA-LoRA introduces an annealed activation function to LoRA, enabling non-linear expressivity during training while maintaining mergeability, thus bridging the performance gap with full-parameter training across various tasks.

Contribution

It proposes a novel activation annealing strategy that enhances LoRA's expressivity without losing its mergeability, improving adaptation performance.

Findings

Reduces the performance gap between LoRA and full-parameter training.

Effective across supervised fine-tuning, reinforcement learning, and decoding tasks.

Maintains mergeability while adding non-linear capabilities.

Abstract

Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method. However, its linear adaptation process limits its expressive power. This means there is a gap between the expressive power of linear training and non-linear training. To bridge this gap, we propose AFA-LoRA, a novel training strategy that brings non-linear expressivity to LoRA while maintaining its seamless mergeability. Our key innovation is an annealed activation function that transitions from a non-linear to a linear transformation during training, allowing the adapter to initially adopt stronger representational capabilities before converging to a mergeable linear form. We implement our method on supervised fine-tuning, reinforcement learning, and speculative decoding. The results show that AFA-LoRA reduces the performance gap between LoRA and full-parameter training. This work enables a more powerful and practical paradigm of parameter-efficient adaptation.