SpectralLoRA: Is Low-Frequency Structure Sufficient for LoRA Adaptation? A Spectral Analysis of Weight Updates

TL;DR

This paper analyzes the spectral structure of LoRA weight updates, revealing that low-frequency components dominate and enabling significant compression with minimal performance loss across models and tasks.

Contribution

It provides the first systematic spectral analysis of LoRA updates, demonstrating the dominance of low-frequency components and proposing spectral sparsity as a new PEFT design principle.

Findings

LoRA updates are dominated by 33% low-frequency components capturing 90% spectral energy.

Retaining only 10% of frequency coefficients reduces storage 10x with minimal accuracy loss.

RoBERTa is more spectrally compressible than BERT across tasks.

Abstract

We present a systematic empirical study of the spectral structure of LoRA weight updates. Through 2D Discrete Cosine Transform (DCT) analysis of trained adaptation matrices across BERT-base and RoBERTa-base on four GLUE benchmarks (SST-2, MNLI, CoLA, QQP), we establish that LoRA updates are universally dominated by low-frequency components: on average, just 33% of DCT coefficients capture 90% of total spectral energy. Retaining only 10% of frequency coefficients reduces adapter storage by 10x while sacrificing only 1.95 percentage points on SST-2. Notably, frequency masking at k=50% improves over full LoRA on 3 of 8 model-task pairs, suggesting high-frequency components act as adaptation noise. We further discover that RoBERTa-base is systematically more spectrally compressible than BERT-base across all tasks, and that task complexity governs spectral sensitivity: NLI tasks require more frequency budget than sentiment classification. A subsequent SVD-DCT correlation analysis (Pearson r=0.906, p<1e-9) connects the empirical 33% constant to the spectral dynamics of SGD (Olsen et al., 2025), suggesting a theoretical grounding for this finding. These findings motivate a new design principle for PEFT: spectral sparsity in adaptation.