Evo: Autoregressive-Diffusion Large Language Models with Evolving Balance

TL;DR

Evo introduces a unified latent trajectory model that seamlessly combines autoregressive and diffusion-based language generation, enabling adaptive, efficient, and high-quality text synthesis across diverse tasks.

Contribution

The paper presents Evo, a novel duality latent trajectory model that unifies autoregressive and diffusion models within a continuous evolutionary framework, with a shared probability flow and end-to-end training.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Maintains fast inference speed while improving quality.

Demonstrates robust reasoning and code generation capabilities.

Abstract

We introduce \textbf{Evo}, a duality latent trajectory model that bridges autoregressive (AR) and diffusion-based language generation within a continuous evolutionary generative framework. Rather than treating AR decoding and diffusion generation as separate paradigms, Evo reconceptualizes text generation as a latent flow: each token is associated with a vector-valued embedding that evolves over a progression variable $t_i \in [0, 1]$, indicating its semantic maturity. Low $t_i$ values correspond to confident AR-like refinement, while high values invoke diffusion-style planning, allowing the model to adaptively balance AR and diffusion based on uncertainty. Theoretically, we show that both AR and diffusion models emerge as discretizations of a shared probability flow, and we derive Evo's training objective from a unified variational ELBO. The model is implemented as a time-conditioned Transformer governed by a shared vector field, trained end-to-end to jointly infer latent codes and their progression times. During decoding, Evo performs efficient, semantics-aware refinement, achieving high-quality outputs without sacrificing speed. Empirically, Evo 8B achieves state-of-the-art or highly competitive results on 15 diverse benchmarks, including reasoning (GSM8K, ARC-C), code generation (HumanEval, MBPP), and general language understanding, while maintaining fast inference speed. Our results demonstrate that Evo delivers a new paradigm for LLM design with strong generation quality, robust symbolic reasoning, and decoding efficiency.