Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zones

TL;DR

This paper identifies structured confusion zones in diffusion LLM trajectories and introduces ATPO, a step-selection method that improves reasoning accuracy and stability by focusing on high-leverage steps.

Contribution

It reveals the importance of dynamic confusion zones in dLLMs and proposes ATPO, a novel step-selection strategy that enhances reasoning performance without additional computational costs.

Findings

ATPO improves reasoning accuracy across benchmarks.

Focusing on high-leverage steps increases training stability.

Structured confusion zones predict success or failure.

Abstract

Diffusion Large Language Models (dLLMs) are rapidly emerging alongside autoregressive models as a powerful paradigm for complex reasoning, with reinforcement learning increasingly used for downstream alignment. Existing trajectory-based RL methods uniformly allocate policy gradients across denoising steps, implicitly treating all steps as equally important. We challenge this assumption by analyzing trajectories with several step-level metrics: entropy-based uncertainty, Confidence-Margin (CM) uncertainty, and Rate of Entropy Change (RoEC). These reveal structured "zones of confusion": transient spikes in uncertainty and instability that strongly predict final success or failure, while most steps remain stable. We propose Adaptive Trajectory Policy Optimization (ATPO), a lightweight step-selection strategy that dynamically reallocates gradient updates to these high-leverage steps without changing the RL objective, rewards, or compute budget. Using a hybrid RoEC+CM rule, ATPO delivers substantial gains in reasoning accuracy and training stability across benchmarks, showing that exploiting trajectory dynamics is key to advancing dLLM RL.