Utilizing Static Analysis and Code Generation to Accelerate Neural Networks

TL;DR

This paper introduces SONNC, a compiler that uses static analysis to optimize neural network code, significantly improving performance and reducing complexity compared to traditional methods.

Contribution

The paper presents SONNC, a novel compiler leveraging static analysis for optimized parallel neural network code generation, outperforming hand-optimized and MATLAB implementations.

Findings

SONNC outperforms hand-optimized C++ code by up to 7.8X.

SONNC outperforms MATLAB code by up to 24X.

SONNC reduces code complexity for sparse networks.

Abstract

As datasets continue to grow, neural network (NN) applications are becoming increasingly limited by both the amount of available computational power and the ease of developing high-performance applications. Researchers often must have expert systems knowledge to make their algorithms run efficiently. Although available computing power increases rapidly each year, algorithm efficiency is not able to keep pace due to the use of general purpose compilers, which are not able to fully optimize specialized application domains. Within the domain of NNs, we have the added knowledge that network architecture remains constant during training, meaning the architecture's data structure can be statically optimized by a compiler. In this paper, we present SONNC, a compiler for NNs that utilizes static analysis to generate optimized parallel code. We show that SONNC's use of static optimizations make it able to outperform hand-optimized C++ code by up to 7.8X, and MATLAB code by up to 24X. Additionally, we show that use of SONNC significantly reduces code complexity when using structurally sparse networks.