perf4sight: A toolflow to model CNN training performance on Edge GPUs

TL;DR

perf4sight is a toolflow that accurately models CNN training latency and memory footprint on edge GPUs, enabling efficient network adaptation with minimal resource use.

Contribution

It introduces an automated methodology for predicting CNN training performance metrics on edge devices, facilitating optimal network configuration selection.

Findings

Models predict latency with 91% accuracy.

Models predict memory footprint with 95% accuracy.

Enables rapid identification of resource-efficient network topologies.

Abstract

The increased memory and processing capabilities of today's edge devices create opportunities for greater edge intelligence. In the domain of vision, the ability to adapt a Convolutional Neural Network's (CNN) structure and parameters to the input data distribution leads to systems with lower memory footprint, latency and power consumption. However, due to the limited compute resources and memory budget on edge devices, it is necessary for the system to be able to predict the latency and memory footprint of the training process in order to identify favourable training configurations of the network topology and device combination for efficient network adaptation. This work proposes perf4sight, an automated methodology for developing accurate models that predict CNN training memory footprint and latency given a target device and network. This enables rapid identification of network topologies that can be retrained on the edge device with low resource consumption. With PyTorch as the framework and NVIDIA Jetson TX2 as the target device, the developed models predict training memory footprint and latency with 95% and 91% accuracy respectively for a wide range of networks, opening the path towards efficient network adaptation on edge GPUs.