LoRTA: Low Rank Tensor Adaptation of Large Language Models

TL;DR

LoRTA introduces a higher-order tensor decomposition method for efficient fine-tuning of large language models, reducing parameters while maintaining performance across various benchmarks.

Contribution

This work proposes a novel CP tensor decomposition approach for parameter-efficient fine-tuning, surpassing existing low-rank matrix and tensor methods in flexibility and compactness.

Findings

Reduces trainable parameters significantly.

Maintains performance comparable to full fine-tuning.

Effective across NLP and protein folding tasks.

Abstract

Low Rank Adaptation (LoRA) is a popular Parameter Efficient Fine Tuning (PEFT) method that effectively adapts large pre-trained models for downstream tasks. LoRA parameterizes model updates using low-rank matrices at each layer, significantly reducing the number of trainable parameters and, consequently, resource requirements during fine-tuning. However, the lower bound on the number of trainable parameters remains high due to the use of the low-rank matrix model. Recent works have addressed this limitation by proposing low rank tensor parameterizations for model updates. However, they only exploit redundancy across layers, or tensorize individual matrices using ad-hoc schemes that introduce additional hyperparameters. In this work, we propose a higher-order Candecomp/Parafac (CP) decomposition, enabling a more compact and flexible representation compared to existing matrix and tensor based PEFT methods. Our experiments on Natural Language Understanding, Instruction Tuning, Preference Optimization and Protein Folding benchmarks demonstrate that our method can achieve a reduction in the number of parameters while maintaining comparable performance.