Understanding the Performance and Estimating the Cost of LLM Fine-Tuning

TL;DR

This paper analyzes the performance and cost of fine-tuning Large Language Models using sparse Mixture of Experts, providing insights into efficiency, optimization, and an analytical cost estimation model for practitioners.

Contribution

It characterizes sparse MoE LLM fine-tuning performance and develops a cost estimation model based on GPU and model parameters.

Findings

Sparse MoE models achieve competitive accuracy with efficient GPU utilization.

Optimization of MoE layers significantly improves fine-tuning performance.

The analytical cost model accurately predicts training throughput and expenses.

Abstract

Due to the cost-prohibitive nature of training Large Language Models (LLMs), fine-tuning has emerged as an attractive alternative for specializing LLMs for specific tasks using limited compute resources in a cost-effective manner. In this paper, we characterize sparse Mixture of Experts (MoE) based LLM fine-tuning to understand their accuracy and runtime performance on a single GPU. Our evaluation provides unique insights into the training efficacy of sparse and dense versions of MoE models, as well as their runtime characteristics, including maximum batch size, execution time breakdown, end-to-end throughput, GPU hardware utilization, and load distribution. Our study identifies the optimization of the MoE layer as crucial for further improving the performance of LLM fine-tuning. Using our profiling results, we also develop and validate an analytical model to estimate the cost of LLM fine-tuning on the cloud. This model, based on parameters of the model and GPU architecture, estimates LLM throughput and the cost of training, aiding practitioners in industry and academia to budget the cost of fine-tuning a specific model.