Improving Prediction Certainty Estimation for Reliable Early Exiting via Null Space Projection

TL;DR

This paper introduces a novel early exiting method for pre-trained language models that improves prediction certainty estimation by considering class-irrelevant information, leading to faster inference with minimal performance loss.

Contribution

It proposes the NSP score to better estimate prediction certainty and a CAP score for more reliable early exiting, outperforming state-of-the-art methods on the GLUE benchmark.

Findings

Achieves 2.19x speed-up on average across GLUE tasks.

Surpasses SOTA ConsistentEE by 28% in efficiency.

Maintains negligible performance degradation.

Abstract

Early exiting has demonstrated great potential in accelerating the inference of pre-trained language models (PLMs) by enabling easy samples to exit at shallow layers, eliminating the need for executing deeper layers. However, existing early exiting methods primarily rely on class-relevant logits to formulate their exiting signals for estimating prediction certainty, neglecting the detrimental influence of class-irrelevant information in the features on prediction certainty. This leads to an overestimation of prediction certainty, causing premature exiting of samples with incorrect early predictions. To remedy this, we define an NSP score to estimate prediction certainty by considering the proportion of class-irrelevant information in the features. On this basis, we propose a novel early exiting method based on the Certainty-Aware Probability (CAP) score, which integrates insights from both logits and the NSP score to enhance prediction certainty estimation, thus enabling more reliable exiting decisions. The experimental results on the GLUE benchmark show that our method can achieve an average speed-up ratio of 2.19x across all tasks with negligible performance degradation, surpassing the state-of-the-art (SOTA) ConsistentEE by 28%, yielding a better trade-off between task performance and inference efficiency. The code is available at https://github.com/He-Jianing/NSP.git.