LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation

TL;DR

LaMDA introduces a spectrally decomposed low-dimensional adaptation method for fine-tuning large language models, significantly reducing trainable parameters and GPU memory while maintaining or improving performance across various NLP tasks.

Contribution

The paper presents LaMDA, a novel fine-tuning approach that leverages spectral decomposition to reduce parameter updates and memory usage, with an enhanced version LaMDA++ incorporating adaptive rank allocation.

Findings

Achieves up to 17.7x fewer parameter updates.

Reduces peak GPU memory usage by up to 1.32x.

Matches or surpasses existing fine-tuning methods in performance.

Abstract

Low-rank adaptation (LoRA) has become the default approach to fine-tune large language models (LLMs) due to its significant reduction in trainable parameters. However, trainable parameter demand for LoRA increases with increasing model embedding dimensions, leading to high compute costs. Additionally, its backward updates require storing high-dimensional intermediate activations and optimizer states, demanding high peak GPU memory. In this paper, we introduce large model fine-tuning via spectrally decomposed low-dimensional adaptation (LaMDA), a novel approach to fine-tuning large language models, which leverages low-dimensional adaptation to achieve significant reductions in trainable parameters and peak GPU memory footprint. LaMDA freezes a first projection matrix (PMA) in the adaptation path while introducing a low-dimensional trainable square matrix, resulting in substantial reductions in trainable parameters and peak GPU memory usage. LaMDA gradually freezes a second projection matrix (PMB) during the early fine-tuning stages, reducing the compute cost associated with weight updates to enhance parameter efficiency further. We also present an enhancement, LaMDA++, incorporating a ``lite-weight" adaptive rank allocation for the LoRA path via normalized spectrum analysis of pre-trained model weights. We evaluate LaMDA/LaMDA++ across various tasks, including natural language understanding with the GLUE benchmark, text summarization, natural language generation, and complex reasoning on different LLMs. Results show that LaMDA matches or surpasses the performance of existing alternatives while requiring up to 17.7x fewer parameter updates and up to 1.32x lower peak GPU memory usage during fine-tuning. Code will be publicly available.