TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models

TL;DR

TLoRA+ is a novel low-rank parameter-efficient fine-tuning method for large language models that improves performance while maintaining efficiency, validated through experiments on the GLUE benchmark.

Contribution

It introduces TLoRA+, an enhanced PEFT approach that incorporates a new optimizer into weight matrices, boosting performance without significant computational overhead.

Findings

Consistently outperforms existing PEFT methods on GLUE benchmark.

Maintains low inference latency comparable to LoRA.

Demonstrates robustness across diverse model architectures.

Abstract

Fine-tuning large language models (LLMs) aims to adapt pre-trained models to specific tasks using relatively small and domain-specific datasets. Among Parameter-Efficient Fine-Tuning (PEFT) methods, Low-Rank Adaptation (LoRA) stands out by matching the performance of full fine-tuning while avoiding additional inference latency. In this paper, we propose a novel PEFT method that incorporates the TLoRA+ optimizer into the weight matrices of pre-trained models. The proposed approach not only preserves the efficiency of low-rank adaptation but also further enhances performance without significantly increasing computational cost. We conduct experiments on the GLUE benchmark across diverse model architectures. Numerical experiments consistently demonstrate the effectiveness and robustness of our proposed method.