# Single-view neural illumination estimation and editing for dynamic light field display

## TL;DR

This paper introduces a method to estimate and edit lighting from a single view, enabling realistic and interactive light field displays for VR and AR.

## Contribution

A novel neural framework for single-view illumination estimation and editing in dynamic light field synthesis.

## Key findings

- The method achieves 0.2397 W m−2 irradiance error and 7.02∘ angular deviation in illumination estimation.
- Synthesized images show a 17.0% PSNR improvement and 51.2% LPIPS reduction compared to existing methods.
- The framework enables photorealistic and perceptually consistent light field displays for immersive applications.

## Abstract

Light field rendering is widely applied to virtual reality (VR), augmented reality (AR), mixed reality (MR) and extended reality (XR). For photorealistic light field displays, it requires a dense view sampling of the scene. However, in dynamic immersive interactions, the available observations are often too sparse to synthesize the complete light field required for a high-fidelity display. Therefore, it poses a huge challenge for generating photometrically consistent views between the virtual and real world. Here, we introduce a neural illumination estimation and editing framework for adaptive light field synthesis. The proposed method can explicitly encode intrinsic parameters of illumination from one single sampling view, which is used for a hybrid-guided generative network to synthesize photometrically plausible dense views of the scene under the guidance of a complete rendering model. It deconstructs the baked-in lighting to enable consistent and high-fidelity relighting from any viewpoint. Our method estimates and edits illumination with only 0.2397 W m−2 irradiance error and 7.02∘ angular deviation, yielding synthesized images with an average 17.0% improvement in PSNR and a 51.2% reduction in LPIPS. This work presents a practical pathway towards truly interactive and adaptive digital light fields, enabling photorealistic content generation for the next generation of near-eye displays and computational imaging systems.

Single-view neural illumination estimation enables interactive lighting editing and perceptual consistency for immersive dynamic light field displays.