Controllable 3D Placement of Objects with Scene-Aware Diffusion Models

TL;DR

This paper introduces a scene-aware diffusion model approach for precise 3D object placement in images, enabling flexible control over position, orientation, and shape while preserving background integrity.

Contribution

The authors propose a novel conditioning method using visual maps and coarse masks for high-quality, controllable object placement that maintains background details.

Findings

Effective in automotive scene editing

Accurate pose and location control demonstrated

Compatible with appearance adjustments

Abstract

Image editing approaches have become more powerful and flexible with the advent of powerful text-conditioned generative models. However, placing objects in an environment with a precise location and orientation still remains a challenge, as this typically requires carefully crafted inpainting masks or prompts. In this work, we show that a carefully designed visual map, combined with coarse object masks, is sufficient for high quality object placement. We design a conditioning signal that resolves ambiguities, while being flexible enough to allow for changing of shapes or object orientations. By building on an inpainting model, we leave the background intact by design, in contrast to methods that model objects and background jointly. We demonstrate the effectiveness of our method in the automotive setting, where we compare different conditioning signals in novel object placement tasks. These tasks are designed to measure edit quality not only in terms of appearance, but also in terms of pose and location accuracy, including cases that require non-trivial shape changes. Lastly, we show that fine location control can be combined with appearance control to place existing objects in precise locations in a scene.