Motion Dreamer: Boundary Conditional Motion Reasoning for Physically Coherent Video Generation

TL;DR

Motion Dreamer is a novel framework that enables physically coherent video generation by reasoning about object motions based on boundary conditions, effectively integrating partial user inputs and improving realism.

Contribution

It introduces a two-stage approach with instance flow and motion inpainting to explicitly handle boundary conditions in motion reasoning for video synthesis.

Findings

Outperforms existing methods in motion plausibility

Achieves higher visual realism in generated videos

Effectively integrates partial user-defined motions

Abstract

Recent advances in video generation have shown promise for generating future scenarios, critical for planning and control in autonomous driving and embodied intelligence. However, real-world applications demand more than visually plausible predictions; they require reasoning about object motions based on explicitly defined boundary conditions, such as initial scene image and partial object motion. We term this capability Boundary Conditional Motion Reasoning. Current approaches either neglect explicit user-defined motion constraints, producing physically inconsistent motions, or conversely demand complete motion inputs, which are rarely available in practice. Here we introduce Motion Dreamer, a two-stage framework that explicitly separates motion reasoning from visual synthesis, addressing these limitations. Our approach introduces instance flow, a sparse-to-dense motion representation enabling effective integration of partial user-defined motions, and the motion inpainting strategy to robustly enable reasoning motions of other objects. Extensive experiments demonstrate that Motion Dreamer significantly outperforms existing methods, achieving superior motion plausibility and visual realism, thus bridging the gap towards practical boundary conditional motion reasoning. Our webpage is available: https://envision-research.github.io/MotionDreamer/.