Neural Field Representations of Mobile Computational Photography

TL;DR

This paper demonstrates how neural field models can efficiently represent complex scenes captured by mobile devices, enabling advanced imaging applications without extensive pre-processing or labeled data.

Contribution

It introduces neural field models tailored for mobile computational photography, achieving superior scene reconstruction directly from raw smartphone data.

Findings

Outperforms state-of-the-art methods in scene reconstruction tasks

Does not require labeled ground truth or complex pre-processing

Successfully applies to depth estimation, layer separation, and image stitching

Abstract

Over the past two decades, mobile imaging has experienced a profound transformation, with cell phones rapidly eclipsing all other forms of digital photography in popularity. Today's cell phones are equipped with a diverse range of imaging technologies - laser depth ranging, multi-focal camera arrays, and split-pixel sensors - alongside non-visual sensors such as gyroscopes, accelerometers, and magnetometers. This, combined with on-board integrated chips for image and signal processing, makes the cell phone a versatile pocket-sized computational imaging platform. Parallel to this, we have seen in recent years how neural fields - small neural networks trained to map continuous spatial input coordinates to output signals - enable the reconstruction of complex scenes without explicit data representations such as pixel arrays or point clouds. In this thesis, I demonstrate how carefully designed neural field models can compactly represent complex geometry and lighting effects. Enabling applications such as depth estimation, layer separation, and image stitching directly from collected in-the-wild mobile photography data. These methods outperform state-of-the-art approaches without relying on complex pre-processing steps, labeled ground truth data, or machine learning priors. Instead, they leverage well-constructed, self-regularized models that tackle challenging inverse problems through stochastic gradient descent, fitting directly to raw measurements from a smartphone.