Cut-and-Paste Object Insertion by Enabling Deep Image Prior for Reshading

TL;DR

This paper introduces a novel deep image prior-based method for realistic object insertion in images, effectively correcting shading inconsistencies without requiring detailed geometric or physical models.

Contribution

The method enables realistic object insertion by reshading using a deep image prior, bypassing the need for geometric or material models of the object.

Findings

Outperforms baseline methods in qualitative and quantitative evaluations.

Achieves more convincing shading consistency in complex surface objects.

User study confirms preference over existing methods.

Abstract

We show how to insert an object from one image to another and get realistic results in the hard case, where the shading of the inserted object clashes with the shading of the scene. Rendering objects using an illumination model of the scene doesn't work, because doing so requires a geometric and material model of the object, which is hard to recover from a single image. In this paper, we introduce a method that corrects shading inconsistencies of the inserted object without requiring a geometric and physical model or an environment map. Our method uses a deep image prior (DIP), trained to produce reshaded renderings of inserted objects via consistent image decomposition inferential losses. The resulting image from DIP aims to have (a) an albedo similar to the cut-and-paste albedo, (b) a similar shading field to that of the target scene, and (c) a shading that is consistent with the cut-and-paste surface normals. The result is a simple procedure that produces convincing shading of the inserted object. We show the efficacy of our method both qualitatively and quantitatively for several objects with complex surface properties and also on a dataset of spherical lampshades for quantitative evaluation. Our method significantly outperforms an Image Harmonization (IH) baseline for all these objects. They also outperform the cut-and-paste and IH baselines in a user study with over 100 users.