LoCO: Low-rank Compositional Rotation Fine-tuning

TL;DR

LoCO introduces a parameter-efficient fine-tuning method using orthogonal transformations via low-rank skew-symmetric matrices, enabling effective adaptation across NLP and vision models with practical computation.

Contribution

It proposes a novel PEFT approach that constructs orthogonal transformations through low-rank compositional rotations, improving structure preservation and computational efficiency.

Findings

LoCO achieves superior or competitive performance across multiple domains.

The method maintains orthogonality with low approximation error.

It enables fully parallel computation of compositional rotations.

Abstract

Parameter-efficient fine-tuning (PEFT) has emerged as an critical technique for adapting large-scale foundation models across natural language processing and computer vision. While existing methods such as low-rank adaptations achieve parameter efficiency via low-rank weight updates, they are limited in their ability to preserve the geometric structure of pretrained representations. We introduce Low-rank Compositional Orthogonal fine-tuning (LoCO), a novel PEFT method that constructs orthogonal transformations through low-rank skew-symmetric matrices and compositional rotation chains. We propose an approximation scheme that enables fully parallel computation of compositional rotations, making the approach practical for high-dimensional feature spaces. Our method maintains low computational complexity while maintaining orthogonality with controlled approximation error. We validate LoCO across diverse domains, including diffusion transformer fine-tuning, vision transformer adaptation, and language model adaptation. Our method demonstrates superior or competitive performance compared to both existing orthogonal and non-orthogonal methods.