QoS-Nets: Adaptive Approximate Neural Network Inference

TL;DR

This paper presents QoS-Nets, a method for dynamically adjusting neural network accuracy and resource use at runtime by selecting and fine-tuning multiple approximate multiplier configurations, enabling power savings with minimal accuracy loss.

Contribution

It introduces a novel search and fine-tuning approach for multiple operating points of approximate multipliers, allowing adaptive QoS in neural networks.

Findings

Achieves 15.3% to 42.8% power savings with minimal accuracy loss.

Supports multiple runtime operating points with only 2.75% increase in parameters.

Demonstrates effectiveness on MobileNetV2 with adaptive approximate multipliers.

Abstract

In order to vary the arithmetic resource consumption of neural network applications at runtime, this work proposes the flexible reuse of approximate multipliers for neural network layer computations. We introduce a search algorithm that chooses an appropriate subset of approximate multipliers of a user-defined size from a larger search space and enables retraining to maximize task performance. Unlike previous work, our approach can output more than a single, static assignment of approximate multiplier instances to layers. These different operating points allow a system to gradually adapt its Quality of Service (QoS) to changing environmental conditions by increasing or decreasing its accuracy and resource consumption. QoS-Nets achieves this by reassigning the selected approximate multiplier instances to layers at runtime. To combine multiple operating points with the use of retraining, we propose a fine-tuning scheme that shares the majority of parameters between operating points, with only a small amount of additional parameters required per operating point. In our evaluation on MobileNetV2, QoS-Nets is used to select four approximate multiplier instances for three different operating points. These operating points result in power savings for multiplications between 15.3% and 42.8% at a Top-5 accuracy loss between 0.3 and 2.33 percentage points. Through our fine-tuning scheme, all three operating points only increase the model's parameter count by only 2.75%.