AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling

TL;DR

AbbIE introduces a recursive, block-based encoder that improves language modeling efficiency and performance by enabling dynamic compute scaling at test time without specialized training.

Contribution

It presents a novel recursive encoder architecture that generalizes to arbitrary iteration lengths and enhances language modeling with dynamic compute scaling.

Findings

Up to 12% improvement in zero-shot in-context learning

Up to 5% reduction in language perplexity

Effective on models up to 350M parameters

Abstract

We introduce the Autoregressive Block-Based Iterative Encoder (AbbIE), a novel recursive generalization of the encoder-only Transformer architecture, which achieves better perplexity than a standard Transformer and allows for the dynamic scaling of compute resources at test time. This simple, recursive approach is a complement to scaling large language model (LLM) performance through parameter and token counts. AbbIE performs its iterations in latent space, but unlike latent reasoning models, does not require a specialized dataset or training protocol. We show that AbbIE upward generalizes (ability to generalize to arbitrary iteration lengths) at test time by only using 2 iterations during train time, far outperforming alternative iterative methods. AbbIE's ability to scale its computational expenditure based on the complexity of the task gives it an up to \textbf{12\%} improvement in zero-shot in-context learning tasks versus other iterative and standard methods and up to 5\% improvement in language perplexity. The results from this study open a new avenue to Transformer performance scaling. We perform all of our evaluations on model sizes up to 350M parameters.