Low-Memory Neural Network Training: A Technical Report

TL;DR

This paper investigates the actual memory needs for training neural networks and evaluates techniques like sparsity, low precision, microbatching, and gradient checkpointing to significantly reduce memory usage with minimal impact on model quality.

Contribution

It provides a comprehensive analysis of memory reduction techniques for training neural networks and demonstrates their effectiveness through extensive experiments on benchmark models.

Findings

Up to 60.7x memory reduction for WideResNet training with 0.4% accuracy loss.

Up to 8.7x memory reduction for Transformer training with 0.15 BLEU score drop.

Tradeoffs between memory savings, accuracy, and computation are characterized.

Abstract

Memory is increasingly often the bottleneck when training neural network models. Despite this, techniques to lower the overall memory requirements of training have been less widely studied compared to the extensive literature on reducing the memory requirements of inference. In this paper we study a fundamental question: How much memory is actually needed to train a neural network? To answer this question, we profile the overall memory usage of training on two representative deep learning benchmarks -- the WideResNet model for image classification and the DynamicConv Transformer model for machine translation -- and comprehensively evaluate four standard techniques for reducing the training memory requirements: (1) imposing sparsity on the model, (2) using low precision, (3) microbatching, and (4) gradient checkpointing. We explore how each of these techniques in isolation affects both the peak memory usage of training and the quality of the end model, and explore the memory, accuracy, and computation tradeoffs incurred when combining these techniques. Using appropriate combinations of these techniques, we show that it is possible to the reduce the memory required to train a WideResNet-28-2 on CIFAR-10 by up to 60.7x with a 0.4% loss in accuracy, and reduce the memory required to train a DynamicConv model on IWSLT'14 German to English translation by up to 8.7x with a BLEU score drop of 0.15.