Optimising the Performance of Convolutional Neural Networks across Computing Systems using Transfer Learning

TL;DR

This paper introduces a transfer learning-based performance modeling approach to optimize convolutional neural network inference across different hardware platforms, significantly reducing profiling time from hours to seconds.

Contribution

It presents a novel machine learning method that replaces extensive profiling with transfer learning, enabling rapid and transferable performance estimation for neural network primitives.

Findings

Performance modeling reduces optimization time from hours to seconds.

Transfer learning enables model adaptation to different hardware with minimal data.

The approach maintains high accuracy across diverse computing platforms.

Abstract

The choice of convolutional routines (primitives) to implement neural networks has a tremendous impact on their inference performance (execution speed) on a given hardware platform. To optimise a neural network by primitive selection, the optimal primitive is identified for each layer of the network. This process requires a lengthy profiling stage, iterating over all the available primitives for each layer configuration, to measure their execution time on the target platform. Because each primitive exploits the hardware in different ways, new profiling is needed to obtain the best performance when moving to another platform. In this work, we propose to replace this prohibitively expensive profiling stage with a machine learning based approach of performance modeling. Our approach speeds up the optimisation time drastically. After training, our performance model can estimate the performance of convolutional primitives in any layer configuration. The time to optimise the execution of large neural networks via primitive selection is reduced from hours to just seconds. Our performance model is easily transferable to other target platforms. We demonstrate this by training a performance model on an Intel platform and performing transfer learning to AMD and ARM processor devices with minimal profiled samples.