Optimizing Memory-Access Patterns for Deep Learning Accelerators

TL;DR

This paper introduces a systematic polyhedral model-based method to optimize memory-access patterns in deep learning accelerators, significantly improving performance by reducing memory accesses.

Contribution

It presents a novel approach that analyzes all operators of a DL model collectively to minimize memory accesses using the polyhedral model.

Findings

Reduces memory access overhead in neural network models

Improves performance on AWS Inferentia chip

Demonstrates substantial efficiency gains

Abstract

Deep learning (DL) workloads are moving towards accelerators for faster processing and lower cost. Modern DL accelerators are good at handling the large-scale multiply-accumulate operations that dominate DL workloads; however, it is challenging to make full use of the compute power of an accelerator since the data must be properly staged in a software-managed scratchpad memory. Failing to do so can result in significant performance loss. This paper proposes a systematic approach which leverages the polyhedral model to analyze all operators of a DL model together to minimize the number of memory accesses. Experiments show that our approach can substantially reduce the impact of memory accesses required by common neural-network models on a homegrown AWS machine-learning inference chip named Inferentia, which is available through Amazon EC2 Inf1 instances.