Neural Light Spheres for Implicit Image Stitching and View Synthesis

TL;DR

This paper introduces a spherical neural light field model for implicit panoramic image stitching and view synthesis, capable of handling complex scene dynamics and producing high-quality, real-time wide field-of-view renderings.

Contribution

The work presents a novel single-layer neural light sphere model that efficiently performs panoramic stitching and scene reconstruction with real-time rendering capabilities.

Findings

Outperforms traditional stitching and radiance field methods in quality

Handles depth parallax, view-dependent lighting, and scene motion effectively

Achieves 50 FPS rendering at 1080p resolution

Abstract

Challenging to capture, and challenging to display on a cellphone screen, the panorama paradoxically remains both a staple and underused feature of modern mobile camera applications. In this work we address both of these challenges with a spherical neural light field model for implicit panoramic image stitching and re-rendering; able to accommodate for depth parallax, view-dependent lighting, and local scene motion and color changes during capture. Fit during test-time to an arbitrary path panoramic video capture -- vertical, horizontal, random-walk -- these neural light spheres jointly estimate the camera path and a high-resolution scene reconstruction to produce novel wide field-of-view projections of the environment. Our single-layer model avoids expensive volumetric sampling, and decomposes the scene into compact view-dependent ray offset and color components, with a total model size of 80 MB per scene, and real-time (50 FPS) rendering at 1080p resolution. We demonstrate improved reconstruction quality over traditional image stitching and radiance field methods, with significantly higher tolerance to scene motion and non-ideal capture settings.