Formalising the Logit Shift Induced by LoRA: A Technical Note

TL;DR

This paper provides a first-order formalisation of the logit shift caused by LoRA, decomposing its effects into layerwise contributions and inter-layer interactions using a Fréchet approximation.

Contribution

It introduces a novel first-order mathematical framework to understand and quantify the effects of LoRA on model logits and margins.

Findings

Decomposition of LoRA effects into layerwise contributions.

Identification of inter-layer coupling as a higher-order term.

Formalisation using a Fréchet approximation.

Abstract

This technical note provides a first-order formalisation of the logit shift and fact-margin change induced by Low-Rank Adaptation (LoRA). Using a first-order Fr\'echet approximation around the base model trajectory, we show that the multi-layer LoRA effect can be decomposed into a linear summation of layerwise contributions and a higher-order remainder term representing inter-layer coupling.