Why LoRA Resists Label Noise: A Theoretical Framework for Noise-Robust Parameter-Efficient Fine-Tuning

TL;DR

This paper provides a theoretical explanation for LoRA's resistance to label noise, showing its limited capacity to memorize noise and proposing a curriculum training method that leverages rank discrepancy for improved noise detection and robust fine-tuning.

Contribution

It introduces a theoretical framework explaining LoRA's noise resistance, derives optimal rank balancing, and proposes RACT for noise detection, advancing parameter-efficient fine-tuning methods.

Findings

LoRA cannot memorize all label noise beyond a certain sample size.

Optimal rank balancing reduces noise-induced variance.

RACT achieves high noise detection accuracy while maintaining classification performance.

Abstract

Parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) have become the dominant paradigm for adapting large pretrained models. We present a theoretical framework explaining an underexplored property: LoRA's inherent resistance to label noise. Our analysis reveals three key insights. First, we prove that rank-$r$ LoRA cannot memorize all possible label assignments once the sample size exceeds $O(r(d+k-r))$, limiting its capacity to fit arbitrary noise. Second, we derive an optimal rank balancing approximation bias and noise-induced variance, showing it decreases with noise rate. Third, we establish temporal separation: clean patterns are learned early while noise memorization occurs later. We propose RACT (Rank-Aware Curriculum Training), leveraging rank discrepancy for noise detection. Experiments validate our predictions, with RACT achieving 91.1% F1 for noise detection on AG News while maintaining 91.46% accuracy, competitive with baselines that lack noise detection capability.