QuTiBench: Benchmarking Neural Networks on Heterogeneous Hardware

TL;DR

QuTiBench is a comprehensive benchmarking suite designed to evaluate neural networks across diverse hardware architectures and algorithmic optimizations, including quantization, to aid system designers in making informed decisions.

Contribution

It introduces a novel multi-tiered benchmark suite that supports algorithmic optimizations like quantization and evaluates heterogeneous hardware architectures for neural networks.

Findings

Supports algorithmic optimizations such as quantization

Enables comparison of diverse hardware architectures

Facilitates understanding of trade-offs in neural network deployment

Abstract

Neural Networks have become one of the most successful universal machine learning algorithms. They play a key role in enabling machine vision and speech recognition for example. Their computational complexity is enormous and comes along with equally challenging memory requirements, which limits deployment in particular within energy constrained, embedded environments. In order to address these implementation challenges, a broad spectrum of new customized and heterogeneous hardware architectures have emerged, often accompanied with co-designed algorithms to extract maximum benefit out of the hardware. Furthermore, numerous optimization techniques are being explored for neural networks to reduce compute and memory requirements while maintaining accuracy. This results in an abundance of algorithmic and architectural choices, some of which fit specific use cases better than others. For system level designers, there is currently no good way to compare the variety of hardware, algorithm and optimization options. While there are many benchmarking efforts in this field, they cover only subsections of the embedded design space. None of the existing benchmarks support essential algorithmic optimizations such as quantization, an important technique to stay on chip, or specialized heterogeneous hardware architectures. We propose a novel benchmark suite, QuTiBench, that addresses this need. QuTiBench is a novel multi-tiered benchmarking methodology that supports algorithmic optimizations such as quantization and helps system developers understand the benefits and limitations of these novel compute architectures in regard to specific neural networks and will help drive future innovation. We invite the community to contribute to QuTiBench in order to support the full spectrum of choices in implementing machine learning systems.