DiRL: An Efficient Post-Training Framework for Diffusion Language Models

TL;DR

DiRL introduces an efficient post-training framework for diffusion language models, improving performance on complex reasoning tasks like mathematics through optimized training and inference strategies.

Contribution

The paper presents DiRL, a novel post-training framework combining blockwise training and optimized inference, along with DiPO, an unbiased policy optimization method for dLLMs.

Findings

Achieves state-of-the-art math performance among dLLMs.

Surpasses Qwen2.5 series on multiple benchmarks.

Enables efficient online model updates.

Abstract

Diffusion Language Models (dLLMs) have emerged as promising alternatives to Auto-Regressive (AR) models. While recent efforts have validated their pre-training potential and accelerated inference speeds, the post-training landscape for dLLMs remains underdeveloped. Existing methods suffer from computational inefficiency and objective mismatches between training and inference, severely limiting performance on complex reasoning tasks such as mathematics. To address this, we introduce DiRL, an efficient post-training framework that tightly integrates FlexAttention-accelerated blockwise training with LMDeploy-optimized inference. This architecture enables a streamlined online model update loop, facilitating efficient two-stage post-training (Supervised Fine-Tuning followed by Reinforcement Learning). Building on this framework, we propose DiPO, the first unbiased Group Relative Policy Optimization (GRPO) implementation tailored for dLLMs. We validate our approach by training DiRL-8B-Instruct on high-quality math data. Our model achieves state-of-the-art math performance among dLLMs and surpasses comparable models in the Qwen2.5 series on several benchmarks.