Mesoscopic photogrammetry with an unstabilized phone camera

TL;DR

This paper introduces a novel feature-free photogrammetric method for 3D mesoscopic imaging using a smartphone camera, achieving high accuracy without additional hardware or stabilization, by jointly estimating height maps, camera pose, and distortion.

Contribution

The method is the first to enable accurate 3D mesoscopic imaging with a smartphone camera in freehand motion without extra hardware, using a joint neural and geometric approach.

Findings

Achieves tens-of-micron accuracy in 3D height maps.

Successfully reconstructs complex samples like paintings and circuit boards.

Reduces computation time and memory for multi-frame registration.

Abstract

We present a feature-free photogrammetric technique that enables quantitative 3D mesoscopic (mm-scale height variation) imaging with tens-of-micron accuracy from sequences of images acquired by a smartphone at close range (several cm) under freehand motion without additional hardware. Our end-to-end, pixel-intensity-based approach jointly registers and stitches all the images by estimating a coaligned height map, which acts as a pixel-wise radial deformation field that orthorectifies each camera image to allow homographic registration. The height maps themselves are reparameterized as the output of an untrained encoder-decoder convolutional neural network (CNN) with the raw camera images as the input, which effectively removes many reconstruction artifacts. Our method also jointly estimates both the camera's dynamic 6D pose and its distortion using a nonparametric model, the latter of which is especially important in mesoscopic applications when using cameras not designed for imaging at short working distances, such as smartphone cameras. We also propose strategies for reducing computation time and memory, applicable to other multi-frame registration problems. Finally, we demonstrate our method using sequences of multi-megapixel images captured by an unstabilized smartphone on a variety of samples (e.g., painting brushstrokes, circuit board, seeds).