Pinpointing the Memory Behaviors of DNN Training

TL;DR

This paper characterizes the memory behaviors of GPU during DNN training, revealing stable, iterative access patterns that can inform future memory-efficient training optimizations.

Contribution

It introduces a method to pinpoint memory behaviors of GPU during DNN training by instrumenting runtime memory allocators, providing new insights into memory access patterns.

Findings

Memory access patterns are stable and iterative.

Memory behaviors vary across different device memory blocks.

Insights can guide future memory optimization strategies.

Abstract

The training of deep neural networks (DNNs) is usually memory-hungry due to the limited device memory capacity of DNN accelerators. Characterizing the memory behaviors of DNN training is critical to optimize the device memory pressures. In this work, we pinpoint the memory behaviors of each device memory block of GPU during training by instrumenting the memory allocators of the runtime system. Our results show that the memory access patterns of device memory blocks are stable and follow an iterative fashion. These observations are useful for the future optimization of memory-efficient training from the perspective of raw memory access patterns.