Place-it-R1: Unlocking Environment-aware Reasoning Potential of MLLM for Video Object Insertion

TL;DR

Place-it-R1 is an innovative framework that enhances video object insertion by integrating environment-aware reasoning of multimodal large language models with diffusion models, ensuring physically plausible and high-quality edits.

Contribution

The paper introduces Place-it-R1, a novel end-to-end system that combines MLLM reasoning with diffusion models for environment-aware, physically consistent video object insertion.

Findings

Achieves physically-coherent video object insertion compared to state-of-the-art methods.

Provides user-controlled modes balancing plausibility and fidelity.

Demonstrates improved environmental understanding in video editing tasks.

Abstract

Modern video editing techniques have achieved high visual fidelity when inserting video objects. However, they focus on optimizing visual fidelity rather than physical causality, leading to edits that are physically inconsistent with their environment. In this work, we present Place-it-R$1$, an end-to-end framework for video object insertion that unlocks the environment-aware reasoning potential of Multimodal Large Language Models (MLLMs). Our framework leverages the Chain-of-Thought (CoT) reasoning of MLLMs to orchestrate video diffusion, following a Think-then-Place paradigm. To bridge cognitive reasoning and generative execution, we introduce three key innovations: First, MLLM performs physical scene understanding and interaction reasoning, generating environment-aware chain-of-thought tokens and inferring valid insertion regions to explicitly guide the diffusion toward physically plausible insertion. Then, we introduce MLLM-guided Spatial Direct Preference Optimization (DPO), where diffusion outputs are fed back to the MLLM for scoring, enabling visual naturalness. During inference, the MLLM iteratively triggers refinement cycles and elicits adaptive adjustments from the diffusion model, forming a closed-loop that progressively enhances editing quality. Furthermore, we provide two user-selectable modes: a plausibility-oriented flexible mode that permits environment modifications (\eg, generating support structures) to enhance physical plausibility, and a fidelity-oriented standard mode that preserves scene integrity for maximum fidelity, offering users explicit control over the plausibility-fidelity trade-off. Extensive experiments demonstrate Place-it-R1 achieves physically-coherent video object insertion compared with state-of-the-art solutions and commercial models.