Predicting Probabilities of Error to Combine Quantization and Early Exiting: QuEE

TL;DR

This paper introduces QuEE, a dynamic network that combines quantization and early exiting by predicting the potential accuracy gains, enabling input-dependent computation reduction for efficient inference.

Contribution

It proposes a novel approach to combine quantization and early exit strategies using accuracy prediction, addressing the complexity of input-dependent computation.

Findings

Effective in reducing computation on multiple datasets

Accurately predicts potential accuracy improvements

Enhances efficiency of neural network inference

Abstract

Machine learning models can solve complex tasks but often require significant computational resources during inference. This has led to the development of various post-training computation reduction methods that tackle this issue in different ways, such as quantization which reduces the precision of weights and arithmetic operations, and dynamic networks which adapt computation to the sample at hand. In this work, we propose a more general dynamic network that can combine both quantization and early exit dynamic network: QuEE. Our algorithm can be seen as a form of soft early exiting or input-dependent compression. Rather than a binary decision between exiting or continuing, we introduce the possibility of continuing with reduced computation. This complicates the traditionally considered early exiting problem, which we solve through a principled formulation. The crucial factor of our approach is accurate prediction of the potential accuracy improvement achievable through further computation. We demonstrate the effectiveness of our method through empirical evaluation, as well as exploring the conditions for its success on 4 classification datasets.