A Study of Optimizations for Fine-tuning Large Language Models

TL;DR

This paper provides a comprehensive analysis of various optimization techniques for fine-tuning large language models, focusing on memory efficiency and runtime performance, and offers practical recommendations for different resource scenarios.

Contribution

It systematically evaluates multiple fine-tuning optimizations and proposes effective strategies and combinations for large models under resource constraints.

Findings

Gradient Checkpointing reduces memory usage significantly.

Optimal optimization combinations balance memory and runtime effectively.

Strategies enable fine-tuning of models with hundreds of billions of parameters.

Abstract

Fine-tuning large language models is a popular choice among users trying to adapt them for specific applications. However, fine-tuning these models is a demanding task because the user has to examine several factors, such as resource budget, runtime, model size and context length among others. A specific challenge is that fine-tuning is memory intensive, imposing constraints on the required hardware memory and context length of training data that can be handled. In this work, we share a detailed study on a variety of fine-tuning optimizations across different fine-tuning scenarios. In particular, we assess Gradient Checkpointing, Low-Rank Adaptation, DeepSpeed's Zero Redundancy Optimizer and FlashAttention. With a focus on memory and runtime, we examine the impact of different optimization combinations on GPU memory usage and execution runtime during fine-tuning phase. We provide our recommendation on the best default optimization for balancing memory and runtime across diverse model sizes. We share effective strategies for fine-tuning very large models with tens or hundreds of billions of parameters and enabling large context lengths during fine-tuning. Furthermore, we propose the appropriate optimization mixtures for fine-tuning under GPU resource limitations.