HydraOpt: Navigating the Efficiency-Performance Trade-off of Adapter Merging

TL;DR

HydraOpt is a novel model merging method for low-rank adapters in large language models that balances storage efficiency and task performance, reducing memory use while maintaining competitive accuracy.

Contribution

HydraOpt introduces a new merging technique that exploits matrix similarities to navigate the efficiency-performance trade-off in adapter merging.

Findings

48% reduction in storage size compared to separate adapters

0.2-1.8% performance drop with HydraOpt

Outperforms existing merging methods in efficiency and performance

Abstract

Large language models (LLMs) often leverage adapters, such as low-rank-based adapters, to achieve strong performance on downstream tasks. However, storing a separate adapter for each task significantly increases memory requirements, posing a challenge for resource-constrained environments such as mobile devices. Although model merging techniques can reduce storage costs, they typically result in substantial performance degradation. In this work, we introduce HydraOpt, a new model merging technique that capitalizes on the inherent similarities between the matrices of low-rank adapters. Unlike existing methods that produce a fixed trade-off between storage size and performance, HydraOpt allows us to navigate this spectrum of efficiency and performance. Our experiments show that HydraOpt significantly reduces storage size (48% reduction) compared to storing all adapters, while achieving competitive performance (0.2-1.8% drop). Furthermore, it outperforms existing merging techniques in terms of performance at the same or slightly worse storage efficiency.