Curvature-Guided LoRA: Steering in the pretrained NTK subspace

TL;DR

This paper introduces Curvature-Guided LoRA, a novel parameter-efficient fine-tuning method that leverages curvature information for improved performance and convergence in large pretrained models.

Contribution

It proposes a curvature-aware, second-order formulation for LoRA, leading to a new method that better aligns predictions with full fine-tuning without heavy computation.

Findings

CG-LoRA outperforms existing LoRA variants on NLP benchmarks.

The method achieves faster convergence during fine-tuning.

It effectively utilizes local curvature information for better adaptation.

Abstract

Parameter-efficient fine-tuning methods such as LoRA enable efficient adaptation of large pretrained models but often fall short of full fine-tuning performance. Existing approaches focus on aligning parameter updates, which only indirectly control model predictions. In this work, we introduce the prediction alignment problem, aiming to match the predictor obtained via PEFT to that of full fine-tuning at the level of outputs. We show that this objective naturally leads to a curvature-aware, second-order formulation, where optimal low-rank updates correspond to a Newton-like, curvature-whitened gradient. Based on this insight, we propose Curvature-Guided LoRA (CG-LoRA), which selects and scales adaptation directions using local curvature information. Our method is computationally efficient and avoids explicit second-order matrix construction. Preliminary experiments on standard natural language understanding benchmarks demonstrate improved performance and faster convergence compared to existing LoRA variants.