A Framework to Enable Algorithmic Design Choice Exploration in DNNs

TL;DR

This paper introduces an open source framework that allows easy exploration and selection of different algorithms in deep neural networks, enabling performance optimization without additional overhead.

Contribution

The paper presents a novel framework that provides fine-grain algorithmic control and high-performance implementations for DNNs, facilitating algorithmic exploration and optimization.

Findings

Built-in implementations match PyTorch performance

Framework incurs no additional performance overhead

Enables easy algorithmic experimentation in DNNs

Abstract

Deep learning technologies, particularly deep neural networks (DNNs), have demonstrated significant success across many domains. This success has been accompanied by substantial advancements and innovations in the algorithms behind the operations required by DNNs. These enhanced algorithms hold the potential to greatly increase the performance of DNNs. However, discovering the best performing algorithm for a DNN and altering the DNN to use such algorithm is a difficult and time consuming task. To address this, we introduce an open source framework which provides easy to use fine grain algorithmic control for DNNs, enabling algorithmic exploration and selection. Along with built-in high performance implementations of common deep learning operations, the framework enables users to implement and select their own algorithms to be utilized by the DNN. The framework's built-in accelerated implementations are shown to yield outputs equivalent to and exhibit similar performance as implementations in PyTorch, a popular DNN framework. Moreover, the framework incurs no additional performance overhead, meaning that performance depends solely on the algorithms chosen by the user.