Joint Tensor-Train Parameterization for Efficient and Expressive Low-Rank Adaptation

TL;DR

This paper introduces TensorGuide, a novel tensor-train-guided framework for low-rank adaptation that enhances expressivity and efficiency in neural model fine-tuning, outperforming existing methods through joint low-rank matrix generation.

Contribution

TensorGuide presents a unified TT-based approach to generate correlated low-rank matrices, improving expressivity and parameter efficiency without increasing trainable parameters.

Findings

TensorGuide outperforms standard LoRA and TT-LoRA in accuracy.

The method improves generalization and scalability.

Theoretical analysis confirms enhanced optimization dynamics.

Abstract

Low-Rank Adaptation (LoRA) is widely recognized for its parameter-efficient fine-tuning of large-scale neural models. However, standard LoRA independently optimizes low-rank matrices, which inherently limits its expressivity and generalization capabilities. While classical tensor-train (TT) decomposition can be separately employed on individual LoRA matrices, this work demonstrates that the classical TT-based approach neither significantly improves parameter efficiency nor achieves substantial performance gains. This paper proposes TensorGuide, a novel tensor-train-guided adaptation framework to overcome these limitations. TensorGuide generates two correlated low-rank LoRA matrices through a unified TT structure driven by controlled Gaussian noise. The resulting joint TT representation inherently provides structured, low-rank adaptations, significantly enhancing expressivity, generalization, and parameter efficiency without increasing the number of trainable parameters. Theoretically, we justify these improvements through neural tangent kernel analyses, demonstrating superior optimization dynamics and enhanced generalization. Extensive experiments on quantum dot classification and GPT-2 fine-tuning benchmarks demonstrate that TensorGuide-based LoRA consistently outperforms standard LoRA and TT-LoRA, achieving improved accuracy and scalability with fewer parameters.