Group and Shuffle: Efficient Structured Orthogonal Parametrization

TL;DR

This paper introduces a new structured orthogonal parametrization that enhances the efficiency of orthogonal fine-tuning in neural networks, applicable across various domains including text-to-image models and language tasks.

Contribution

It unifies and generalizes structured matrix classes to improve parameter and computational efficiency in orthogonal fine-tuning methods.

Findings

Improved efficiency in fine-tuning large models.

Successful application to text-to-image diffusion models.

Effective adaptation for orthogonal convolutions and 1-Lipschitz networks.

Abstract

The increasing size of neural networks has led to a growing demand for methods of efficient fine-tuning. Recently, an orthogonal fine-tuning paradigm was introduced that uses orthogonal matrices for adapting the weights of a pretrained model. In this paper, we introduce a new class of structured matrices, which unifies and generalizes structured classes from previous works. We examine properties of this class and build a structured orthogonal parametrization upon it. We then use this parametrization to modify the orthogonal fine-tuning framework, improving parameter and computational efficiency. We empirically validate our method on different domains, including adapting of text-to-image diffusion models and downstream task fine-tuning in language modeling. Additionally, we adapt our construction for orthogonal convolutions and conduct experiments with 1-Lipschitz neural networks.