A Real-time Method for Inserting Virtual Objects into Neural Radiance Fields

TL;DR

This paper introduces a real-time method for inserting virtual objects into neural radiance fields, enabling realistic lighting, shadows, and interaction for augmented reality applications.

Contribution

It presents a novel real-time approach that integrates lighting estimation, occlusion handling, and shadow casting within NeRFs for improved virtual object insertion.

Findings

Achieves realistic lighting and shadow effects in real-time

Outperforms state-of-the-art techniques in fidelity

Enables interactive manipulation of virtual objects

Abstract

We present the first real-time method for inserting a rigid virtual object into a neural radiance field, which produces realistic lighting and shadowing effects, as well as allows interactive manipulation of the object. By exploiting the rich information about lighting and geometry in a NeRF, our method overcomes several challenges of object insertion in augmented reality. For lighting estimation, we produce accurate, robust and 3D spatially-varying incident lighting that combines the near-field lighting from NeRF and an environment lighting to account for sources not covered by the NeRF. For occlusion, we blend the rendered virtual object with the background scene using an opacity map integrated from the NeRF. For shadows, with a precomputed field of spherical signed distance field, we query the visibility term for any point around the virtual object, and cast soft, detailed shadows onto 3D surfaces. Compared with state-of-the-art techniques, our approach can insert virtual object into scenes with superior fidelity, and has a great potential to be further applied to augmented reality systems.