Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective

TL;DR

This paper introduces ESPO, a sequence-level RL framework for diffusion LLMs that treats entire sequences as actions, overcoming likelihood approximation challenges and significantly improving performance on reasoning and coding tasks.

Contribution

The paper proposes a novel ELBO-based sequence-level policy optimization method for diffusion LLMs, addressing fundamental likelihood approximation issues in RL.

Findings

ESPO outperforms token-level baselines by 20-40 points on the Countdown task

Achieves consistent improvements on math and coding benchmarks

Establishes sequence-level optimization as a principled RL paradigm for dLLMs

Abstract

Reinforcement Learning (RL) has proven highly effective for autoregressive language models, but adapting these methods to diffusion large language models (dLLMs) presents fundamental challenges. The core difficulty lies in likelihood approximation: while autoregressive models naturally provide token-level conditional probabilities essential for token-level RL objectives (e.g., GRPO), dLLMs generate sequences through iterative non-autoregressive denoising steps that lack this factorization. To address this fundamental mismatch, we propose ELBO-based Sequence-level Policy Optimization (ESPO), a principled RL framework that treats entire sequence generation as a single action and uses the ELBO as a tractable sequence-level likelihood proxy. Our method incorporates per-token normalization of importance ratios and robust KL-divergence estimation to ensure stable large-scale training. Extensive experiments on mathematical reasoning, coding, and planning tasks demonstrate that ESPO significantly outperforms token-level baselines, achieving dramatic improvements of 20-40 points on the Countdown task, while maintaining consistent gains on math and coding benchmarks. Our approach establishes sequence-level optimization as a principled and empirically effective paradigm for RL in dLLMs. Our code is available at https://github.com/ML-GSAI/ESPO.