Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation

TL;DR

This paper introduces CoTo, a progressive training strategy for low-rank adaptation that gradually increases adapter activation, leading to better model generalization, robustness, and efficiency across various tasks and LoRA variants.

Contribution

CoTo is a novel progressive training method that enhances low-rank adaptation by promoting balanced optimization and broader exploration of the loss landscape.

Findings

Boosts single-task performance

Improves multi-task merging accuracy

Enhances pruning robustness

Abstract

Low-rank adaptation (LoRA) has emerged as a leading parameter-efficient fine-tuning technique for adapting large foundation models, yet it often locks adapters into suboptimal minima near their initialization. This hampers model generalization and limits downstream operators such as adapter merging and pruning. Here, we propose CoTo, a progressive training strategy that gradually increases adapters' activation probability over the course of fine-tuning. By stochastically deactivating adapters, CoTo encourages more balanced optimization and broader exploration of the loss landscape. We provide a theoretical analysis showing that CoTo promotes layer-wise dropout stability and linear mode connectivity, and we adopt a cooperative-game approach to quantify each adapter's marginal contribution. Extensive experiments demonstrate that CoTo consistently boosts single-task performance, enhances multi-task merging accuracy, improves pruning robustness, and reduces training overhead, all while remaining compatible with diverse LoRA variants. Code is available at https://github.com/zwebzone/coto.