Flexible Deep Neural Network Processing

TL;DR

This paper introduces a flexible ensemble processing method for deep neural networks that significantly reduces inference latency with minimal accuracy loss, adaptable to different quality and runtime requirements.

Contribution

The paper presents a novel flexible ensemble processing technique for DNNs that dynamically balances accuracy and latency, applicable to various network architectures.

Findings

Significant reduction in inference latency achieved

Minimal accuracy drop with the proposed method

Effective on AlexNet and ResNet-50 architectures

Abstract

The recent success of Deep Neural Networks (DNNs) has drastically improved the state of the art for many application domains. While achieving high accuracy performance, deploying state-of-the-art DNNs is a challenge since they typically require billions of expensive arithmetic computations. In addition, DNNs are typically deployed in ensemble to boost accuracy performance, which further exacerbates the system requirements. This computational overhead is an issue for many platforms, e.g. data centers and embedded systems, with tight latency and energy budgets. In this article, we introduce flexible DNNs ensemble processing technique, which achieves large reduction in average inference latency while incurring small to negligible accuracy drop. Our technique is flexible in that it allows for dynamic adaptation between quality of results (QoR) and execution runtime. We demonstrate the effectiveness of the technique on AlexNet and ResNet-50 using the ImageNet dataset. This technique can also easily handle other types of networks.