LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights

TL;DR

CRAFT introduces a parameter-efficient fine-tuning method for transformers that uses frozen Tucker decomposition on attention weights, enabling effective adaptation with minimal trainable parameters.

Contribution

It applies full Tucker decomposition directly on pre-trained weights and trains only small adaptation matrices, reducing parameter count while maintaining performance.

Findings

Achieves competitive results on GLUE with fewer parameters.

Uses only 41K adaptation parameters, independent of model size.

Demonstrates effectiveness of cross-layer tensor decomposition for PEFT.

Abstract

We introduce CRAFT (Cross-layer Rank Adaptation via Frozen Tucker), a parameter-efficient fine-tuning (PEFT) method that applies Tucker tensor decomposition to pre-trained attention weight matrices stacked across transformer layers and trains only small square adaptation matrices on the resulting frozen Tucker factors. Existing tensor-based PEFT methods decompose gradient updates: LoTR applies Tucker decomposition with shared factor matrices, while SuperLoRA groups and reshapes $\Delta W$ across layers before applying Tucker decomposition. Separately, methods like PiSSA apply SVD to pre-trained weights but operate independently per layer. CRAFT bridges these two lines of work: it performs full Tucker decomposition via Higher-Order SVD (HOSVD) directly on pre-trained weights organized as cross-layer 3D tensors, freezes all resulting factors, and adapts the model through lightweight trainable transformations applied to each factor matrix. Experiments on the GLUE benchmark using RoBERTa-base and RoBERTa-large demonstrate that CRAFT achieves competitive performance with existing methods while requiring only 41K Tucker adaptation parameters--a count independent of model dimension and depth at fixed Tucker ranks.