Photoreal Scene Reconstruction from an Egocentric Device

TL;DR

This paper introduces a novel approach for photorealistic scene reconstruction from egocentric devices by calibrating rolling shutter cameras with visual-inertial bundle adjustment and modeling sensor effects within Gaussian Splatting, leading to improved image quality.

Contribution

The study presents a new calibration method using VIBA for rolling shutter cameras and integrates a physical image formation model into Gaussian Splatting for enhanced scene reconstruction.

Findings

+1 dB PSNR improvement with VIBA

+1 dB PSNR gain from the image formation model

Effective reconstruction demonstrated on Project Aria and Meta Quest3

Abstract

In this paper, we investigate the challenges associated with using egocentric devices to photorealistic reconstruct the scene in high dynamic range. Existing methodologies typically assume using frame-rate 6DoF pose estimated from the device's visual-inertial odometry system, which may neglect crucial details necessary for pixel-accurate reconstruction. This study presents two significant findings. Firstly, in contrast to mainstream work treating RGB camera as global shutter frame-rate camera, we emphasize the importance of employing visual-inertial bundle adjustment (VIBA) to calibrate the precise timestamps and movement of the rolling shutter RGB sensing camera in a high frequency trajectory format, which ensures an accurate calibration of the physical properties of the rolling-shutter camera. Secondly, we incorporate a physical image formation model based into Gaussian Splatting, which effectively addresses the sensor characteristics, including the rolling-shutter effect of RGB cameras and the dynamic ranges measured by sensors. Our proposed formulation is applicable to the widely-used variants of Gaussian Splats representation. We conduct a comprehensive evaluation of our pipeline using the open-source Project Aria device under diverse indoor and outdoor lighting conditions, and further validate it on a Meta Quest3 device. Across all experiments, we observe a consistent visual enhancement of +1 dB in PSNR by incorporating VIBA, with an additional +1 dB achieved through our proposed image formation model. Our complete implementation, evaluation datasets, and recording profile are available at http://www.projectaria.com/photoreal-reconstruction/