Break the Block: Dynamic-size Reasoning Blocks for Diffusion Large Language Models via Monotonic Entropy Descent with Reinforcement Learning

TL;DR

This paper introduces b1, a post-training framework for diffusion large language models that dynamically adjusts reasoning block sizes using a Monotonic Entropy Descent objective with reinforcement learning, improving reasoning coherence.

Contribution

It presents a novel method for learning dynamic reasoning block sizes in dLLMs, addressing the limitations of fixed-size blocks for better reasoning performance.

Findings

b1 consistently outperforms fixed-size block baselines across various benchmarks.

The method leverages entropy trends to adapt block sizes for improved reasoning coherence.

Extensive experiments validate the effectiveness of the proposed approach.

Abstract

Recent diffusion large language models (dLLMs) have demonstrated both effectiveness and efficiency in reasoning via a block-based semi-autoregressive generation paradigm. Despite their progress, the fixed-size block generations remain a critical bottleneck for effective and coherent reasoning. 1. From a global perspective, different reasoning tasks would correspond to different optimal decoding block sizes, which makes a ``one-size-fits-all'' assumption ineffective. 2. Even within a single reasoning task, the rigid block partitioning would break the logical flow and reduce reasoning coherence. Through empirical observations, we reveal that for block-wise entropy, incorrect reasoning exhibits a fluctuating and unsteady trend between blocks, whereas the correctly generated tasks follow a consistent descending trend. Therefore, this paper proposes b1, a novel post-training framework for dLLMs that learns dynamic-size reasoning blocks via a Monotonic Entropy Descent objective with reinforcement learning to enhance reasoning coherence.b1 integrates seamlessly as a plug-and-play module with existing dLLM's post-training algorithms. Extensive experiments across various reasoning benchmarks showcase b1's consistent improvement over existing fixed-size block baselines. Our code has been released at https://github.com/YanJiangJerry/Block-R1.