Alternating Updates for Efficient Transformers

TL;DR

AltUp is a simple method that increases transformer model capacity by widening token embeddings with minimal latency increase, enabling faster inference without sacrificing accuracy.

Contribution

The paper introduces Alternating Updates (AltUp), a novel technique to widen transformer representations efficiently by updating subblocks, compatible with existing models and methods.

Findings

Achieves up to 87% speedup on SuperGLUE and SQuAD benchmarks.

Maintains comparable accuracy while significantly reducing inference latency.

Effective across diverse transformer models and language tasks.

Abstract

It has been well established that increasing scale in deep transformer networks leads to improved quality and performance. However, this increase in scale often comes with prohibitive increases in compute cost and inference latency. We introduce Alternating Updates (AltUp), a simple-to-implement method to increase a model's capacity without the computational burden. AltUp enables the widening of the learned representation, i.e., the token embedding, while only incurring a negligible increase in latency. AltUp achieves this by working on a subblock of the widened representation at each layer and using a predict-and-correct mechanism to update the inactivated blocks. We present extensions of AltUp, such as its applicability to the sequence dimension, and demonstrate how AltUp can be synergistically combined with existing approaches, such as Sparse Mixture-of-Experts models, to obtain efficient models with even higher capacity. Our experiments on benchmark transformer models and language tasks demonstrate the consistent effectiveness of AltUp on a diverse set of scenarios. Notably, on SuperGLUE and SQuAD benchmarks, AltUp enables up to $87\%$ speedup relative to the dense baselines at the same accuracy.