Efficient Memory Management for Deep Neural Net Inference

TL;DR

This paper presents strategies for efficient memory sharing in deep neural network inference on edge devices, reducing memory footprint and enabling better performance on resource-constrained hardware.

Contribution

It introduces novel memory management techniques that optimize buffer sharing among tensors, achieving up to 11% smaller memory usage compared to existing methods.

Findings

Up to 11% reduction in memory footprint

Improved memory sharing strategies for neural nets

Enhanced inference efficiency on edge devices

Abstract

While deep neural net inference was considered a task for servers only, latest advances in technology allow the task of inference to be moved to mobile and embedded devices, desired for various reasons ranging from latency to privacy. These devices are not only limited by their compute power and battery, but also by their inferior physical memory and cache, and thus, an efficient memory manager becomes a crucial component for deep neural net inference at the edge. We explore various strategies to smartly share memory buffers among intermediate tensors in deep neural nets. Employing these can result in up to 11% smaller memory footprint than the state of the art.