Relighting Scenes with Object Insertions in Neural Radiance Fields

TL;DR

This paper introduces a NeRF-based method for inserting objects into scenes with realistic relighting and shadows, enhancing augmented reality applications by supporting novel views and physical interactions.

Contribution

A novel NeRF pipeline that enables object insertion, relighting, and shadow casting in scenes using hybrid lighting representations and efficient shadow rendering techniques.

Findings

Achieves realistic relighting effects in diverse scenes.

Supports physical interactions like shadow casting.

Outperforms previous methods in visual realism and flexibility.

Abstract

The insertion of objects into a scene and relighting are commonly utilized applications in augmented reality (AR). Previous methods focused on inserting virtual objects using CAD models or real objects from single-view images, resulting in highly limited AR application scenarios. We propose a novel NeRF-based pipeline for inserting object NeRFs into scene NeRFs, enabling novel view synthesis and realistic relighting, supporting physical interactions like casting shadows onto each other, from two sets of images depicting the object and scene. The lighting environment is in a hybrid representation of Spherical Harmonics and Spherical Gaussians, representing both high- and low-frequency lighting components very well, and supporting non-Lambertian surfaces. Specifically, we leverage the benefits of volume rendering and introduce an innovative approach for efficient shadow rendering by comparing the depth maps between the camera view and the light source view and generating vivid soft shadows. The proposed method achieves realistic relighting effects in extensive experimental evaluations.