MiSS: Revisiting the Trade-off in LoRA with an Efficient Shard-Sharing Structure

TL;DR

MiSS introduces a shard-sharing structure for LoRA that improves convergence speed and balances performance, memory, and efficiency, supported by theoretical and empirical evidence.

Contribution

Proposes Matrix Shard Sharing (MiSS) and MiSS$^e$, novel methods that enhance LoRA's convergence and efficiency while maintaining performance.

Findings

Reduces optimization complexity without performance loss

Achieves better trade-offs among performance, memory, and efficiency

Occupies a favorable position on the Pareto frontier among PEFT methods

Abstract

Low-Rank Adaptation (LoRA) is a widely adopted technique for parameter-efficient fine-tuning, but its slow convergence has spurred the development of numerous variants. Nevertheless, existing methods often fail to improve performance, memory footprint, and computational efficiency simultaneously. To address this challenge, we revisit the causes of LoRA's slow convergence. Building on these insights, we propose Matrix Shard Sharing (MiSS), which updates shards of the original weight matrix using a single shared trainable matrix $\boldsymbol{D}$, initialized to zeros. To simultaneously ensure computational efficiency, low memory footprint, and scalable serving, we introduce MiSS$^e$. Both theoretical analysis and empirical results demonstrate that our method reduces optimization complexity without compromising performance, thereby achieving a more favorable trade-off among performance, memory, and efficiency. Furthermore, we conduct a comprehensive comparative analysis of various PEFT methods, evaluating their memory usage, initialization overhead, and computational efficiency. By mapping the Pareto frontier across these dimensions, we show that MiSS occupies a favorable position, effectively capturing the advantages of prior approaches.