HOFT: Householder Orthogonal Fine-tuning

TL;DR

This paper introduces HOFT and SHOFT, novel orthogonal fine-tuning methods for foundation models that reduce time and memory costs while maintaining or improving performance across various NLP and reasoning tasks.

Contribution

The paper proposes HOFT and SHOFT, new orthogonal fine-tuning techniques that are more efficient and have favorable theoretical properties compared to existing methods.

Findings

HOFT and SHOFT achieve comparable or better results than state-of-the-art methods.

Both methods effectively adapt models in diverse downstream tasks.

Theoretical analysis supports the advantages of orthogonal fine-tuning.

Abstract

Adaptation of foundation models using low-rank methods is a widespread approach. Another way to adapt these models is to employ orthogonal fine-tuning methods, which are less time and memory efficient despite their good generalization properties. In this work, we propose Householder Orthogonal Fine-tuning (HOFT), a novel orthogonal fine-tuning method that aims to alleviate time and space complexity. Moreover, some theoretical properties of the orthogonal fine-tuning paradigm are explored. From this exploration, Scaled Householder Orthogonal Fine-tuning (SHOFT) is proposed. Both HOFT and SHOFT are evaluated in downstream tasks, namely commonsense reasoning, machine translation, subject-driven generation and mathematical reasoning. Compared with state-of-the-art adaptation methods, HOFT and SHOFT show comparable or better results.