Plan-X: Instruct Video Generation via Semantic Planning

TL;DR

Plan-X introduces a semantic planning framework that improves instruction-aligned video generation by reducing hallucinations and enhancing high-level reasoning in diffusion-based models.

Contribution

It proposes a learnable semantic planner that generates structured semantic tokens to guide video synthesis, addressing limitations of existing diffusion transformers.

Findings

Significantly reduces visual hallucinations in generated videos.

Enables fine-grained, instruction-aligned video synthesis.

Improves handling of complex scene understanding and multi-stage actions.

Abstract

Diffusion Transformers have demonstrated remarkable capabilities in visual synthesis, yet they often struggle with high-level semantic reasoning and long-horizon planning. This limitation frequently leads to visual hallucinations and mis-alignments with user instructions, especially in scenarios involving complex scene understanding, human-object interactions, multi-stage actions, and in-context motion reasoning. To address these challenges, we propose Plan-X, a framework that explicitly enforces high-level semantic planning to instruct video generation process. At its core lies a Semantic Planner, a learnable multimodal language model that reasons over the user's intent from both text prompts and visual context, and autoregressively generates a sequence of text-grounded spatio-temporal semantic tokens. These semantic tokens, complementary to high-level text prompt guidance, serve as structured "semantic sketches" over time for the video diffusion model, which has its strength at synthesizing high-fidelity visual details. Plan-X effectively integrates the strength of language models in multimodal in-context reasoning and planning, together with the strength of diffusion models in photorealistic video synthesis. Extensive experiments demonstrate that our framework substantially reduces visual hallucinations and enables fine-grained, instruction-aligned video generation consistent with multimodal context.