Magic NeRF Lens: Interactive Fusion of Neural Radiance Fields for Virtual Facility Inspection

TL;DR

Magic NeRF Lens is an interactive VR framework that fuses neural radiance fields with traditional 3D models for efficient, immersive inspection of complex industrial facilities, enhancing safety and maintenance planning.

Contribution

The paper introduces a novel data fusion method combining NeRF with CAD models and develops two 3D magic lens effects for optimized visualization in VR.

Findings

High usability demonstrated through user studies and expert reviews.

Effective fusion of NeRF with CAD models for complex facility visualization.

Public release of the VR NeRF framework source code.

Abstract

Large industrial facilities such as particle accelerators and nuclear power plants are critical infrastructures for scientific research and industrial processes. These facilities are complex systems that not only require regular maintenance and upgrades but are often inaccessible to humans due to various safety hazards. Therefore, a virtual reality (VR) system that can quickly replicate real-world remote environments to provide users with a high level of spatial and situational awareness is crucial for facility maintenance planning. However, the exact 3D shapes of these facilities are often too complex to be accurately modeled with geometric primitives through the traditional rasterization pipeline. In this work, we develop Magic NeRF Lens, an interactive framework to support facility inspection in immersive VR using neural radiance fields (NeRF) and volumetric rendering. We introduce a novel data fusion approach that combines the complementary strengths of volumetric rendering and geometric rasterization, allowing a NeRF model to be merged with other conventional 3D data, such as a computer-aided design model. We develop two novel 3D magic lens effects to optimize NeRF rendering by exploiting the properties of human vision and context-aware visualization. We demonstrate the high usability of our framework and methods through a technical benchmark, a visual search user study, and expert reviews. In addition, the source code of our VR NeRF framework is made publicly available for future research and development.