When Diffusion Breaks Constraints: Sequential Autoregressive Generation with RL and MCTS

TL;DR

This paper identifies the limitations of diffusion models in constrained generation tasks and proposes a sequential autoregressive approach with reinforcement learning and MCTS to improve feasibility.

Contribution

It introduces a reformulation of constrained generation as discrete autoregressive sequential generation, addressing diffusion models' failure modes.

Findings

Diffusion models struggle with low-dimensional, constrained solution spaces.

Reinforcement learning improves the feasibility and success rate of constrained generation.

Monte Carlo tree search helps evaluate the value of look-ahead in shrinking feasible regions.

Abstract

Data-driven generative models excel in language and vision, but diffusion models often fail in constrained planning and design tasks, exhibiting severe constraint violations in engineering inverse design, molecular generation, multi-robot planning, and floorplan/scene synthesis even with projection or guidance. Such tasks combine hard-to-specify semantic goals with strict geometric or physical constraints (e.g., non-overlap, connectivity), yielding feasible solutions that lie on low-dimensional, small, and sometimes disconnected regions of the output space. This paper studies the failure mode through tangram generation from language, where seven fixed shapes must form a text-described silhouette while remaining connected and non-overlapping, and a simplified rectangle composition task with a learned bounding-box constraint. We find diffusion models struggle to satisfy constraints, consistent with difficulty generating samples near low-dimensional submanifolds. Motivated by locally feasible reparameterizations, we reformulate constrained generation as discrete autoregressive sequential generation. Reinforcement learning improves feasibility and task success, and Monte Carlo tree search quantifies the value of look-ahead when feasible regions shrink. Overall, the empirical, theoretical, and prior-work evidence points to a structural limitation of continuous density matching on this class of constrained-generation problems, and suggests sequential constraint-aware generation as a promising alternative.