Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting

TL;DR

Spectral Surgery is a training-free method that refines LoRA adapters by reweighting their singular values based on gradient sensitivity, leading to consistent performance improvements across multiple tasks and models.

Contribution

This paper introduces Spectral Surgery, a novel post-hoc refinement technique for LoRA adapters that uses SVD and gradient-based sensitivity to improve performance without additional training.

Findings

Achieves up to +4.4 points on CommonsenseQA

Improves pass@1 by +2.4 on HumanEval

Requires only about 1,000 scalar coefficient adjustments

Abstract

Low-Rank Adaptation (LoRA) improves downstream performance by restricting task updates to a low-rank parameter subspace, yet how this limited capacity is allocated within a trained adapter remains unclear. Through a geometric and empirical study across multiple tasks and backbones, we find that trained LoRA updates often exhibit an inefficient spectrum: task effects concentrate in a small subset of singular directions, while many remaining components are neutral or detrimental, motivating post-hoc refinement within the learned subspace. We propose Spectral Surgery, a training-free refinement that decomposes a LoRA update with SVD, estimates per-component sensitivity using gradients on a small calibration set, and reweights singular values under a magnitude constraint while keeping the learned directions fixed. Across Llama-3.1-8B and Qwen3-8B on four benchmarks, Spectral Surgery yields consistent gains (up to +4.4 points on CommonsenseQA and +2.4 pass@1 on HumanEval) by adjusting only $\approx 1{,}000$ scalar coefficients. These results demonstrate that SVD-structured, low-cost parameter editing can serve as a practical route to improving trained LoRA adapters in a purely post-hoc manner.