Fast yet Safe: Early-Exiting with Risk Control

TL;DR

This paper explores how to safely implement early-exit neural networks by applying risk control methods, enabling faster inference without sacrificing performance across vision and language tasks.

Contribution

It introduces a risk control framework for EENNs that adaptively determines safe exit points, balancing speed and accuracy in a distribution-free, post-hoc manner.

Findings

Significant computational savings achieved

Maintains user-specified performance levels

Effective across diverse vision and language tasks

Abstract

Scaling machine learning models significantly improves their performance. However, such gains come at the cost of inference being slow and resource-intensive. Early-exit neural networks (EENNs) offer a promising solution: they accelerate inference by allowing intermediate layers to exit and produce a prediction early. Yet a fundamental issue with EENNs is how to determine when to exit without severely degrading performance. In other words, when is it 'safe' for an EENN to go 'fast'? To address this issue, we investigate how to adapt frameworks of risk control to EENNs. Risk control offers a distribution-free, post-hoc solution that tunes the EENN's exiting mechanism so that exits only occur when the output is of sufficient quality. We empirically validate our insights on a range of vision and language tasks, demonstrating that risk control can produce substantial computational savings, all the while preserving user-specified performance goals.