Shakes on a Plane: Unsupervised Depth Estimation from Unstabilized Photography

TL;DR

This paper introduces a novel unsupervised method for depth estimation from long-burst, unstabilized mobile photography sequences by leveraging natural hand tremor to recover scene depth without extra hardware.

Contribution

It presents a test-time optimization approach that jointly estimates scene depth and camera motion from long-burst RAW images using a neural RGB-D model trained end-to-end.

Findings

Achieves geometrically accurate depth reconstructions

Works with unstabilized, hand-held mobile photography sequences

No additional hardware or pre-processing required

Abstract

Modern mobile burst photography pipelines capture and merge a short sequence of frames to recover an enhanced image, but often disregard the 3D nature of the scene they capture, treating pixel motion between images as a 2D aggregation problem. We show that in a ''long-burst'', forty-two 12-megapixel RAW frames captured in a two-second sequence, there is enough parallax information from natural hand tremor alone to recover high-quality scene depth. To this end, we devise a test-time optimization approach that fits a neural RGB-D representation to long-burst data and simultaneously estimates scene depth and camera motion. Our plane plus depth model is trained end-to-end, and performs coarse-to-fine refinement by controlling which multi-resolution volume features the network has access to at what time during training. We validate the method experimentally, and demonstrate geometrically accurate depth reconstructions with no additional hardware or separate data pre-processing and pose-estimation steps.