A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation

TL;DR

This paper introduces HashEE, a simple hash-based early exiting method for language models that improves efficiency and performance without additional parameters, outperforming previous methods across various tasks.

Contribution

Proposes HashEE, a novel hash-based early exiting approach that eliminates the need for learn-to-exit modules, enhancing efficiency and effectiveness in language understanding and generation.

Findings

HashEE achieves higher accuracy with fewer FLOPs.

HashEE reduces inference time compared to state-of-the-art methods.

Modern neural models struggle to predict instance difficulty accurately.

Abstract

Early exiting allows instances to exit at different layers according to the estimation of difficulty. Previous works usually adopt heuristic metrics such as the entropy of internal outputs to measure instance difficulty, which suffers from generalization and threshold-tuning. In contrast, learning to exit, or learning to predict instance difficulty is a more appealing way. Though some effort has been devoted to employing such "learn-to-exit" modules, it is still unknown whether and how well the instance difficulty can be learned. As a response, we first conduct experiments on the learnability of instance difficulty, which demonstrates that modern neural models perform poorly on predicting instance difficulty. Based on this observation, we propose a simple-yet-effective Hash-based Early Exiting approach (HashEE) that replaces the learn-to-exit modules with hash functions to assign each token to a fixed exiting layer. Different from previous methods, HashEE requires no internal classifiers nor extra parameters, and therefore is more efficient. Experimental results on classification, regression, and generation tasks demonstrate that HashEE can achieve higher performance with fewer FLOPs and inference time compared with previous state-of-the-art early exiting methods.