ENTP: Encoder-only Next Token Prediction

TL;DR

This paper introduces Encoder-only Next Token Prediction (ENTP), a novel approach that outperforms decoder-only Transformers in tasks with unbounded compute, demonstrating advantages in expressive power and practical performance.

Contribution

The paper presents ENTP, an encoder-only architecture for next-token prediction, highlighting its theoretical and empirical benefits over traditional decoder-only models.

Findings

ENTP performs well on the Count3 task, unlike decoder-only models.

ENTP outperforms decoder-only Transformers in addition, in-context learning, and language modeling tasks.

Theoretical analysis shows ENTP's superior expressive power.

Abstract

Next-token prediction is conventionally done using decoder-only Transformers with causal attention, as this approach allows for efficient reuse of keys and values. What if we were not compute-limited, should we still use decoder-only Transformers? In this work, we introduce Encoder-only Next Token Prediction (ENTP). We explore the differences between ENTP and decoder-only Transformers in expressive power and complexity, highlighting potential advantages of ENTP in settings with unbounded compute. We introduce the $\operatorname{Count3}$ task and show, both theoretically and experimentally, that while ENTP can perform this task easily, a decoder-only Transformer cannot. Finally, we empirically demonstrate the superior performance of ENTP across representative tasks where next-token prediction based Transformers can be evaluated, including addition, in-context learning, and language modeling.