POCI-Diff: Position Objects Consistently and Interactively with 3D-Layout Guided Diffusion

TL;DR

POCI-Diff introduces a diffusion-based framework for text-to-image generation that ensures consistent, interactive 3D layout control and editing, effectively maintaining object geometry and identity across complex multi-object scenes.

Contribution

It presents a novel unified diffusion approach that enforces 3D geometric constraints and semantic binding for improved layout adherence and object consistency in scene synthesis.

Findings

Outperforms state-of-the-art in visual fidelity and layout adherence.

Supports object insertion, removal, and transformation via regeneration.

Maintains object identity and scene coherence across edits.

Abstract

We propose a diffusion-based approach for Text-to-Image (T2I) generation with consistent and interactive 3D layout control and editing. While prior methods improve spatial adherence using 2D cues or iterative copy-warp-paste strategies, they often distort object geometry and fail to preserve consistency across edits. To address these limitations, we introduce a framework for Positioning Objects Consistently and Interactively (POCI-Diff), a novel formulation for jointly enforcing 3D geometric constraints and instance-level semantic binding within a unified diffusion process. Our method enables explicit per-object semantic control by binding individual text descriptions to specific 3D bounding boxes through Blended Latent Diffusion, allowing one-shot synthesis of complex multi-object scenes. We further propose a warping-free generative editing pipeline that supports object insertion, removal, and transformation via regeneration rather than pixel deformation. To preserve object identity and consistency across edits, we condition the diffusion process on reference images using IP-Adapter, enabling coherent object appearance throughout interactive 3D editing while maintaining global scene coherence. Experimental results demonstrate that POCI-Diff produces high-quality images consistent with the specified 3D layouts and edits, outperforming state-of-the-art methods in both visual fidelity and layout adherence while eliminating warping-induced geometric artifacts.