LoRMA: Low-Rank Multiplicative Adaptation for LLMs

TL;DR

LoRMA introduces a multiplicative adaptation method for large language models, enhancing efficiency and effectiveness over traditional additive approaches like LoRA by employing matrix multiplicative transformations.

Contribution

This paper proposes Low-Rank Multiplicative Adaptation (LoRMA), a novel approach shifting from additive to multiplicative updates for efficient LLM adaptation.

Findings

LoRMA outperforms LoRA on multiple NLP tasks.

Effective strategies mitigate computational complexity and rank bottleneck.

Extensive experiments validate the approach's effectiveness.

Abstract

Large Language Models have shown remarkable capabilities in the NLP domain. Their effectiveness can mainly be attributed to their ability to adapt to an array of downstream tasks. However, generally, full fine-tuning is a computationally expensive job. To mitigate this, many techniques have been developed that prime efficiency, a prominent one being Low-Rank Adaptation (LoRA). However, LoRA and its variants employ re-parametrized additive updates. In this paper, we propose Low-Rank Multiplicative Adaptation (LoRMA), which shifts the paradigm of additive updates to a richer space of matrix multiplicative transformations. We tackle challenges such as computational complexity and rank bottleneck of matrix multiplication by effectively re-ordering operations and introducing rank inflation strategies. We conduct extensive experiments to demonstrate the effectiveness of our approach in terms of various evaluation metrics.