wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models

TL;DR

This paper introduces wd1, a novel ratio-free policy optimization method for diffusion-based large language models, improving reasoning capabilities with lower computational cost and state-of-the-art accuracy on math benchmarks.

Contribution

The paper proposes wd1, a new ratio-free RL approach reformulating the objective as weighted log-likelihood, enhancing efficiency and performance in diffusion LLMs.

Findings

wd1 outperforms previous diffusion-based methods with up to 59% accuracy improvement.

wd1++ achieves state-of-the-art results on MATH500 and GSM8K benchmarks.

The method reduces computational overhead and variance in policy optimization.

Abstract

Improving the reasoning capabilities of diffusion-based large language models (dLLMs) through reinforcement learning (RL) remains an open problem. The intractability of dLLMs likelihood function necessitates approximating the current, old, and reference policy likelihoods at each policy optimization step. This reliance introduces additional computational overhead, and can lead to large variance and estimation error in RL objective -- particularly in computing the policy ratio for importance sampling. To mitigate these issues, we introduce wd1, a novel ratio-free policy optimization approach that reformulates the RL objective as a weighted log-likelihood, requiring only a single approximation for the current parametrized policy likelihood. We formally show that our proposed method can be interpreted as energy-guided discrete diffusion training combined with negative sample unlearning, thereby confirming its theoretical soundness. In experiments on LLaDA-8B model, wd1 outperforms diffusion-based GRPO (d1) while requiring lower computational cost, achieving up to a $+59\%$ improvement in accuracy. Furthermore, we extend wd1 to denoising-stepwise weighted policy optimization (wd1++), achieving state-of-the-art math performance of $44.2\%$ on MATH500 and $84.5\%$ on GSM8K with only 20 RL training steps.