User-in-the-Loop View Sampling with Error Peaking Visualization

TL;DR

This paper introduces an error-peaking visualization method for AR view sampling that simplifies data collection, reduces user effort, and improves view synthesis quality in mobile and large scene reconstruction.

Contribution

It proposes a novel error visualization technique that eliminates the need for 3D annotations and expands scene exploration in view sampling for AR and radiance field reconstruction.

Findings

Error-peaking visualization is less invasive and more satisfactory.

Reduces the number of view samples needed for high-quality synthesis.

Enhances scene exploration in large-scale 3D reconstructions.

Abstract

Augmented reality (AR) provides ways to visualize missing view samples for novel view synthesis. Existing approaches present 3D annotations for new view samples and task users with taking images by aligning the AR display. This data collection task is known to be mentally demanding and limits capture areas to pre-defined small areas due to the ideal but restrictive underlying sampling theory. To free users from 3D annotations and limited scene exploration, we propose using locally reconstructed light fields and visualizing errors to be removed by inserting new views. Our results show that the error-peaking visualization is less invasive, reduces disappointment in final results, and is satisfactory with fewer view samples in our mobile view synthesis system. We also show that our approach can contribute to recent radiance field reconstruction for larger scenes, such as 3D Gaussian splatting.