Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation

TL;DR

This paper introduces a Householder reflection-based adaptation method that unifies low-rank and orthogonal adaptation techniques, improving large model fine-tuning efficiency and performance.

Contribution

It proposes a novel Householder reflection adaptation (HRA) method that bridges low-rank and orthogonal adaptation, with regularization for enhanced model capacity.

Findings

HRA outperforms state-of-the-art methods in large language model adaptation.

HRA achieves superior performance with fewer trainable parameters.

The method is integrated into the PEFT package and available for public use.

Abstract

While following different technical routes, both low-rank and orthogonal adaptation techniques can efficiently adapt large-scale pre-training models in specific tasks or domains based on a small piece of trainable parameters. In this study, we bridge the gap between these two techniques, proposing a simple but effective adaptation method based on Householder reflections. Given a pre-trained model, our method fine-tunes its layers by multiplying each frozen weight matrix with an orthogonal matrix constructed by a chain of learnable Householder reflections (HRs). This HR-based orthogonal fine-tuning is equivalent to an adaptive low-rank adaptation. Moreover, we show that the orthogonality of the reflection planes corresponding to the HRs impacts the model capacity and regularity. The analysis motivates us to regularize the orthogonality of the HRs, leading to different implementations of the proposed Householder reflection adaptation (HRA) method. Compared with state-of-the-art methods, HRA achieves superior performance with fewer learnable parameters when adapting large language models and conditional image generators. The code of the experiments is available at \url{https://github.com/DaShenZi721/HRA}, and the method has been merged into the \href{https://github.com/huggingface/peft}{PEFT} package.