Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering

TL;DR

This paper introduces a method combining diffusion models with inverse rendering to insert virtual objects into images with realistic lighting, shadows, and reflections, ensuring photorealism in single images and videos.

Contribution

It presents a novel approach that guides diffusion models with inverse rendering for scene understanding and photorealistic object insertion, improving consistency and realism.

Findings

Achieves photorealistic object insertion with consistent lighting effects.

Automatically refines materials and tone-mapping for better realism.

Works effectively on both indoor and outdoor scenes.

Abstract

The correct insertion of virtual objects in images of real-world scenes requires a deep understanding of the scene's lighting, geometry and materials, as well as the image formation process. While recent large-scale diffusion models have shown strong generative and inpainting capabilities, we find that current models do not sufficiently "understand" the scene shown in a single picture to generate consistent lighting effects (shadows, bright reflections, etc.) while preserving the identity and details of the composited object. We propose using a personalized large diffusion model as guidance to a physically based inverse rendering process. Our method recovers scene lighting and tone-mapping parameters, allowing the photorealistic composition of arbitrary virtual objects in single frames or videos of indoor or outdoor scenes. Our physically based pipeline further enables automatic materials and tone-mapping refinement.