HD-PiSSA: High-Rank Distributed Orthogonal Adaptation

TL;DR

HD-PiSSA is a novel distributed PEFT method for large language models that significantly increases update expressiveness by assigning different principal components to each device, leading to substantial performance improvements.

Contribution

It introduces a high-rank distributed PEFT approach that assigns orthogonal adapters across devices, expanding update directions and improving fine-tuning performance on complex tasks.

Findings

Over 16x higher effective update ranks with 8 GPUs.

Achieves 10.0 absolute point gains over LoRA in multi-task learning.

Demonstrates significant performance improvements across diverse benchmarks.

Abstract

Existing parameter-efficient fine-tuning (PEFT) methods for large language models (LLMs), such as LoRA and PiSSA, constrain model updates to low-rank subspaces, limiting their expressiveness and leading to suboptimal performance on complex tasks. To address this, we introduce High-rank Distributed PiSSA (HD-PiSSA), a distributed PEFT approach that initializes orthogonal adapters across different devices and aggregates their delta updates collectively on W for fine-tuning. Unlike Data Parallel LoRA or PiSSA, which maintain identical adapters across all devices, HD-PiSSA assigns different principal components of the pre-trained weights to each GPU, significantly expanding the range of update directions. This results in over 16x higher effective updated ranks than data-parallel LoRA or PiSSA when fine-tuning on 8 GPUs with the same per-device adapter rank. Empirically, we evaluate HD-PiSSA across various challenging downstream tasks, including mathematics, code generation, and multi-task learning. In the multi-task setting, HD-PiSSA achieves average gains of 10.0 absolute points (14.63%) over LoRA and 4.98 points (6.60%) over PiSSA across 12 benchmarks, demonstrating its benefits from the extra optimization flexibility.