HUT: A More Computation Efficient Fine-Tuning Method With Hadamard Updated Transformation

TL;DR

HUT introduces a Hadamard-based parameter-efficient fine-tuning method that preserves parameter correlation, enhances expressiveness, and reduces computational costs for large pre-trained language models.

Contribution

The paper proposes the Hadamard Updated Transformation (HUT), a novel PEFT approach that constructs direct transformations from original to updated parameters, improving efficiency and expressiveness.

Findings

HUT achieves comparable or better performance than existing PEFT methods.

HUT significantly reduces computational complexity in fine-tuning.

Theoretical and experimental validation on RoBERTa and GPT-2 demonstrate effectiveness.

Abstract

Fine-tuning pre-trained language models for downstream tasks has achieved impressive results in NLP. However, fine-tuning all parameters becomes impractical due to the rapidly increasing size of model parameters. To address this, Parameter Efficient Fine-Tuning (PEFT) methods update only a subset of parameters. Most PEFT methods, such as LoRA, use incremental updates, which involve adding learned weight matrix increments to the original parameters. Although effective, these methods face limitations in capturing complex parameter dynamics and do not maintain a strong correlation between the original and updated parameters. To overcome these challenges, we propose the direct Updated Transformation (UT) paradigm, which constructs a transformation directly from the original to the updated parameters. This approach ensures that the correlation between the original and updated parameters is preserved, leveraging the semantic features learned during pre-training. Building on this paradigm, we present the Hadamard Updated Transformation (HUT) method. HUT efficiently updates the original weight matrix using the Hadamard transformation with two low-rank matrices, offering a more expressive and flexible update mechanism. This allows HUT to capture richer parameter features through functional transformations, reducing computational complexity while maintaining or improving model quality. Theoretical analysis and extensive experiments on RoBERTa and GPT-2 validate the effectiveness of HUT. Results show that HUT performs on par with or better than other PEFT methods in terms of model quality, while significantly reducing computational complexity.