Crowded in B-Space: Calibrating Shared Directions for LoRA Merging

TL;DR

This paper introduces Pico, a data-free calibration method for LoRA merging that improves performance by addressing shared directions in the output matrix, leading to better task-specific adaptation.

Contribution

Pico calibrates the output matrix B before merging, enhancing the effectiveness of existing LoRA merging methods across multiple domains.

Findings

Pico improves average accuracy by 3.4-8.3 points over base methods.

Pico enables merged adapters to outperform full-task LoRA training.

Addressing shared directions in B reduces merge interference.

Abstract

Merging separately trained LoRA adapters is a practical alternative to joint multi-task training, but it often hurts performance. Existing methods usually treat the LoRA update $\Delta W = BA$ as a single object and do not distinguish the two LoRA matrices. We show that the main source of LoRA merge interference comes from the output-side matrix $B$. Across tasks, $B$ repeatedly uses a small set of shared directions, while $A$ remains much more task-specific. As a result, the merged adapter overemphasizes these shared directions, and task-specific information is lost. We propose Pico (Pre-merge interference calibration in output-space), a data-free method that calibrates $B$ before merge by downscaling over-shared directions and then rescaling the merged update. Pico plugs directly into existing merging methods such as Task Arithmetic, TIES, and TSV-M. Across eight different benchmarks from math, coding, finance, and medical domains, Pico improves average accuracy by 3.4-8.3 points over the corresponding base method and achieves the best overall average performance. Pico also enables merged adapters to outperform the LoRA trained with all task data. These results show that LoRA merging works better when the two LoRA matrices are treated separately.