Efficient and Stable Reinforcement Learning for Diffusion Language Models

TL;DR

This paper introduces Spatio-Temporal Pruning (STP), a novel framework that enhances the efficiency and stability of reinforcement learning applied to diffusion-based large language models by reducing redundancy in the generative process.

Contribution

The paper presents STP, a new pruning framework that improves RL for dLLMs by reducing variance and redundancy, leading to more stable and efficient training.

Findings

STP reduces variance in log-likelihood estimation.

STP outperforms state-of-the-art methods in efficiency.

STP achieves higher accuracy in experiments.

Abstract

Reinforcement Learning (RL) is crucial for unlocking the complex reasoning capabilities of Diffusion-based Large Language Models (dLLMs). However, applying RL to dLLMs faces unique challenges in efficiency and stability. To address these challenges, we propose Spatio-Temporal Pruning (STP), a framework designed to simultaneously improve the efficiency and stability of RL for dLLMs. STP compresses the redundancy in the generative process through: (1) \textit{spatial pruning}, which constrains the exploration space using static priors; and (2) \textit{temporal pruning}, which bypasses redundant late-stage refinement steps. Our theoretical analysis demonstrates that STP strictly reduces the variance of the log-likelihood estimation, thereby ensuring more stable policy updates. Extensive experiments demonstrate that STP surpasses state-of-the-art baselines in both efficiency and accuracy. Our code is available at https://github.com/Lolo1222/STP.