BeamLoRA: Beam-Constraint Low-Rank Adaptation

TL;DR

BeamLoRA introduces a beam-search inspired method to dynamically select optimal sub-solutions within LoRA modules, significantly improving fine-tuning accuracy of large language models across diverse tasks.

Contribution

It presents a novel beam-based approach to adaptively optimize LoRA ranks during fine-tuning, addressing limitations of static rank selection.

Findings

Consistently outperforms baseline methods across multiple datasets.

Enhances fine-tuning performance without increasing model size.

Effective across various tasks like math reasoning, code generation, and commonsense reasoning.

Abstract

Due to the demand for efficient fine-tuning of large language models, Low-Rank Adaptation (LoRA) has been widely adopted as one of the most effective parameter-efficient fine-tuning methods. Nevertheless, while LoRA improves efficiency, there remains room for improvement in accuracy. Herein, we adopt a novel perspective to assess the characteristics of LoRA ranks. The results reveal that different ranks within the LoRA modules not only exhibit varying levels of importance but also evolve dynamically throughout the fine-tuning process, which may limit the performance of LoRA. Based on these findings, we propose BeamLoRA, which conceptualizes each LoRA module as a beam where each rank naturally corresponds to a potential sub-solution, and the fine-tuning process becomes a search for the optimal sub-solution combination. BeamLoRA dynamically eliminates underperforming sub-solutions while expanding the parameter space for promising ones, enhancing performance with a fixed rank. Extensive experiments across three base models and 12 datasets spanning math reasoning, code generation, and commonsense reasoning demonstrate that BeamLoRA consistently enhances the performance of LoRA, surpassing the other baseline methods.