3-in-1: 2D Rotary Adaptation for Efficient Finetuning, Efficient Batching and Composability

TL;DR

This paper introduces RoAd, a novel 2D rotation-based method for parameter-efficient fine-tuning of large language models that improves adaptability, batching efficiency, and interpretability with minimal parameter overhead.

Contribution

RoAd provides a simple 2D rotation technique that addresses multiple challenges in PEFT, including multi-adapter deployment, efficiency, and interpretability, with state-of-the-art performance.

Findings

Achieves high performance on GLUE and reasoning tasks with less than 0.1% trainable parameters.

Enables efficient batching for multiple adapters with minimal overhead.

Improves interpretability through distributed interchange intervention experiments.

Abstract

Parameter-efficient finetuning (PEFT) methods effectively adapt large language models (LLMs) to diverse downstream tasks, reducing storage and GPU memory demands. Despite these advantages, several applications pose new challenges to PEFT beyond mere parameter efficiency. One notable challenge involves the efficient deployment of LLMs equipped with multiple task- or user-specific adapters, particularly when different adapters are needed for distinct requests within the same batch. Another challenge is the interpretability of LLMs, which is crucial for understanding how LLMs function. Previous studies introduced various approaches to address different challenges. In this paper, we introduce a novel method, RoAd, which employs a straightforward 2D rotation to adapt LLMs and addresses all the above challenges: (1) RoAd is remarkably parameter-efficient, delivering optimal performance on GLUE, eight commonsense reasoning tasks and four arithmetic reasoning tasks with $<0.1\%$ trainable parameters; (2) RoAd facilitates the efficient serving of requests requiring different adapters within a batch, with an overhead comparable to element-wise multiplication instead of batch matrix multiplication; (3) RoAd enhances LLM's interpretability through integration within a framework of distributed interchange intervention, demonstrated via composition experiments.