Low-Rank Interconnected Adaptation across Layers

TL;DR

Lily introduces a low-rank interconnected adaptation framework that enhances parameter efficiency and expressiveness in fine-tuning large models by sharing components across layers and using data-dependent routing.

Contribution

It proposes a novel PEFT method with interconnected low-rank adapters, improving adaptation performance and efficiency over traditional LoRA.

Findings

Lily outperforms existing methods across various tasks and models.

The interconnected structure allows higher-rank updates with fewer parameters.

Data-dependent routing enhances cross-domain adaptability.

Abstract

Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method that learns weight updates $\Delta W = AB$ for pretrained weights $W$ through low-rank adapters $A$ and $B$. While LoRA ensures hardware efficiency, its low-rank weight updates limit adaptation performance. In this paper, we propose low-rank interconnected adaptation across layers (Lily), a novel PEFT method that introduces an interconnected framework with locally shared $A$ and globally shared $B$ experts. This structure eliminates redundant per-layer $AB$ pairs, enabling higher-rank $\Delta W$ with equal or fewer parameters. To enhance expressiveness, we use data-dependent routers to determine $A$-$B$ interconnections, preventing $B$ experts from converging to the same behavior and improving representational power across domains. Experiments across modalities, architectures, and model sizes demonstrate Lily's superior performance and efficiency. GitHub: https://github.com/yibozhong/lily