Embedding Recycling for Language Models

TL;DR

Embedding recycling (ER) leverages cached model activations to significantly speed up training and inference across multiple language models and tasks with minimal accuracy loss.

Contribution

This paper provides the first extensive evaluation of ER techniques across diverse models and tasks, demonstrating their practical effectiveness and potential for speed improvements.

Findings

Over 90% training speedup with minimal accuracy impact

Effective ER across models from 17M to 900M parameters

Identifies key areas for future research in ER methods

Abstract

Real-world applications of neural language models often involve running many different models over the same corpus. The high computational cost of these runs has led to interest in techniques that can reuse the contextualized embeddings produced in previous runs to speed training and inference of future ones. We refer to this approach as embedding recycling (ER). While multiple ER techniques have been proposed, their practical effectiveness is still unknown because existing evaluations consider very few models and do not adequately account for overhead costs. We perform an extensive evaluation of ER across eight different models (17 to 900 million parameters) and fourteen tasks in English. We show how a simple ER technique that caches activations from an intermediate layer of a pretrained model, and learns task-specific adapters on the later layers, is broadly effective. For the best-performing baseline in our experiments (DeBERTa-v2 XL), adding a precomputed cache results in a >90% speedup during training and 87-91% speedup for inference, with negligible impact on accuracy. Our analysis reveals important areas of future work.