RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates

TL;DR

RoCoFT introduces a parameter-efficient fine-tuning approach for large language models that updates only specific rows and columns of weight matrices, achieving comparable or better accuracy with less memory and computation.

Contribution

The paper presents RoCoFT, a novel fine-tuning method that selectively updates row and column parameters, backed by neural tangent kernel analysis and extensive empirical validation.

Findings

Achieves comparable or better accuracy than state-of-the-art PEFT methods.

Reduces memory and computational requirements during fine-tuning.

Kernel analysis shows close approximation to full-parameter models.

Abstract

We propose RoCoFT, a parameter-efficient fine-tuning method for large-scale language models (LMs) based on updating only a few rows and columns of the weight matrices in transformers. Through extensive experiments with medium-size LMs like BERT and RoBERTa, and larger LMs like Bloom-7B, Llama2-7B, and Llama2-13B, we show that our method gives comparable or better accuracies than state-of-art PEFT methods while also being more memory and computation-efficient. We also study the reason behind the effectiveness of our method with tools from neural tangent kernel theory. We empirically demonstrate that our kernel, constructed using a restricted set of row and column parameters, are numerically close to the full-parameter kernel and gives comparable classification performance. Ablation studies are conducted to investigate the impact of different algorithmic choices, including the selection strategy for rows and columns as well as the optimal rank for effective implementation of our method.