SQUAD: Scalable Quorum Adaptive Decisions via ensemble of early exit neural networks

TL;DR

SQUAD introduces a quorum-based early-exit inference scheme combined with ensemble learning and neural architecture search, significantly improving uncertainty estimation, accuracy, and reducing latency in neural network predictions.

Contribution

The paper presents SQUAD, a novel inference scheme integrating quorum-based early exits with ensemble learning and QUEST NAS, enhancing uncertainty estimation and efficiency.

Findings

Test accuracy improved by up to 5.95% over state-of-the-art methods.

Inference latency reduced by up to 70.60% compared to static ensembles.

Achieved comparable computational cost with better robustness and accuracy.

Abstract

Early-exit neural networks have become popular for reducing inference latency by allowing intermediate predictions when sufficient confidence is achieved. However, standard approaches typically rely on single-model confidence thresholds, which are frequently unreliable due to inherent calibration issues. To address this, we introduce SQUAD (Scalable Quorum Adaptive Decisions), the first inference scheme that integrates early-exit mechanisms with distributed ensemble learning, improving uncertainty estimation while reducing the inference time. Unlike traditional methods that depend on individual confidence scores, SQUAD employs a quorum-based stopping criterion on early-exit learners by collecting intermediate predictions incrementally in order of computational complexity until a consensus is reached and halting the computation at that exit if the consensus is statistically significant. To maximize the efficacy of this voting mechanism, we also introduce QUEST (Quorum Search Technique), a Neural Architecture Search method to select early-exit learners with optimized hierarchical diversity, ensuring learners are complementary at every intermediate layer. This consensus-driven approach yields statistically robust early exits, improving the test accuracy up to 5.95% compared to state-of-the-art dynamic solutions with a comparable computational cost and reducing the inference latency up to 70.60% compared to static ensembles while maintaining a good accuracy.