Performance Control in Early Exiting to Deploy Large Models at the Same Cost of Smaller Ones

TL;DR

This paper introduces Performance Control Early Exiting (PCEE), a novel method that improves large model deployment efficiency by controlling accuracy based on average validation accuracy rather than confidence, enabling better performance-cost trade-offs.

Contribution

The paper presents PCEE, a new early exiting technique that enhances control over model performance and allows larger models to be deployed at similar costs as smaller ones.

Findings

PCEE outperforms confidence-based methods in controlling accuracy.

Larger models with PCEE achieve higher performance at the same computational cost.

PCEE enables scalable deployment of large models with efficient resource use.

Abstract

Early Exiting (EE) is a promising technique for speeding up inference by adaptively allocating compute resources to data points based on their difficulty. The approach enables predictions to exit at earlier layers for simpler samples while reserving more computation for challenging ones. In this study, we first present a novel perspective on the EE approach, showing that larger models deployed with EE can achieve higher performance than smaller models while maintaining similar computational costs. As existing EE approaches rely on confidence estimation at each exit point, we further study the impact of overconfidence on the controllability of the compute-performance trade-off. We introduce Performance Control Early Exiting (PCEE), a method that enables accuracy thresholding by basing decisions not on a data point's confidence but on the average accuracy of samples with similar confidence levels from a held-out validation set. In our experiments, we show that PCEE offers a simple yet computationally efficient approach that provides better control over performance than standard confidence-based approaches, and allows us to scale up model sizes to yield performance gain while reducing the computational cost.