Outlier-weighed Layerwise Sampling for LLM Fine-tuning

TL;DR

The paper introduces Outlier-weighed Layerwise Sampling (OWS), a memory-efficient fine-tuning method for LLMs that selectively fine-tunes layers with more outliers, outperforming baseline methods in accuracy and memory usage.

Contribution

OWS is a novel fine-tuning approach that strategically samples layers based on outlier distribution, improving performance while reducing memory costs.

Findings

OWS outperforms baseline fine-tuning methods across multiple benchmarks.

OWS achieves up to 1.1% accuracy gain on Commonsense Reasoning.

OWS enables fine-tuning of 7B LLMs with only 21GB of memory.

Abstract

The rapid advancements in Large Language Models (LLMs) have revolutionized various natural language processing tasks. However, the substantial size of LLMs presents significant challenges in training or fine-tuning. While parameter-efficient approaches such as low-rank adaptation (LoRA) have gained popularity, they often compromise performance compared to full-rank fine-tuning. In this paper, we propose Outlier-weighed Layerwise Sampling (OWS), a new memory-efficient fine-tuning approach, inspired by the layerwise outlier distribution of LLMs. Unlike LoRA, which adds extra adapters to all layers, OWS strategically assigns higher sampling probabilities to layers with more outliers, selectively sampling only a few layers and fine-tuning their pre-trained weights. To further increase the number of fine-tuned layers without a proportional rise in memory costs, we incorporate gradient low-rank projection, further boosting the approach's performance. Our extensive experiments across various architectures, including LLaMa2 and Mistral, demonstrate that OWS consistently outperforms baseline approaches, including full fine-tuning. Specifically, it achieves up to a 1.1% average accuracy gain on the Commonsense Reasoning benchmark, a 3.0% improvement on MMLU, and a notable 10% boost on MT-Bench, while being more memory efficient. OWS allows us to fine-tune 7B LLMs with only 21GB of memory. Our code is available at https://github.com/pixeli99/OWS.