Unlocking the Global Synergies in Low-Rank Adapters

TL;DR

HeteroLoRA is a novel search algorithm that optimally allocates low-rank adaptation parameters in large language models, improving fine-tuning performance using zero-cost proxies.

Contribution

We introduce HeteroLoRA, a lightweight search method that enhances LoRA fine-tuning by better parameter allocation across models and connection types.

Findings

HeteroLoRA improves accuracy by 1.6% on MRPC.

It effectively allocates parameters in complex search spaces.

The method enhances model performance with the same parameter budget.

Abstract

Low-rank Adaption (LoRA) has been the de-facto parameter-efficient fine-tuning technique for large language models. We present HeteroLoRA, a light-weight search algorithm that leverages zero-cost proxies to allocate the limited LoRA trainable parameters across the model for better fine-tuned performance. In addition to the allocation for the standard LoRA-adapted models, we also demonstrate the efficacy of HeteroLoRA by performing the allocation in a more challenging search space that includes LoRA modules and LoRA-adapted shortcut connections. Experiments show that HeteroLoRA enables improvements in model performance given the same parameter budge. For example, on MRPC, we see an improvement of 1.6% in accuracy with similar training parameter budget. We will open-source our algorithm once the paper is accepted.