Incomplete Dot Products for Dynamic Computation Scaling in Neural Network Inference

TL;DR

This paper introduces incomplete dot products (IDP), a method allowing neural networks to dynamically adjust computation during inference by selectively using a subset of channels, balancing accuracy and efficiency.

Contribution

The paper proposes IDP with a channel contribution profile, enabling a single network to scale computation dynamically without multiple models, and extends it to multiple profiles for different scaling ranges.

Findings

IDP reduces computation by up to 75% on MNIST and CIFAR-10.

Using 50% IDP on VGG-16 achieves 70% accuracy on CIFAR-10.

IDP maintains accuracy with reduced channels, enabling dynamic inference adjustments.

Abstract

We propose the use of incomplete dot products (IDP) to dynamically adjust the number of input channels used in each layer of a convolutional neural network during feedforward inference. IDP adds monotonically non-increasing coefficients, referred to as a "profile", to the channels during training. The profile orders the contribution of each channel in non-increasing order. At inference time, the number of channels used can be dynamically adjusted to trade off accuracy for lowered power consumption and reduced latency by selecting only a beginning subset of channels. This approach allows for a single network to dynamically scale over a computation range, as opposed to training and deploying multiple networks to support different levels of computation scaling. Additionally, we extend the notion to multiple profiles, each optimized for some specific range of computation scaling. We present experiments on the computation and accuracy trade-offs of IDP for popular image classification models and datasets. We demonstrate that, for MNIST and CIFAR-10, IDP reduces computation significantly, e.g., by 75%, without significantly compromising accuracy. We argue that IDP provides a convenient and effective means for devices to lower computation costs dynamically to reflect the current computation budget of the system. For example, VGG-16 with 50% IDP (using only the first 50% of channels) achieves 70% in accuracy on the CIFAR-10 dataset compared to the standard network which achieves only 35% accuracy when using the reduced channel set.