DP-SfM: Dual-Pixel Structure-from-Motion without Scale Ambiguity

TL;DR

This paper introduces a novel method leveraging dual-pixel sensor data to automatically resolve scale ambiguity in multi-view 3D reconstruction without needing reference objects or prior calibration.

Contribution

It presents a simple linear approach utilizing defocus blur in dual-pixel images to determine absolute scale in structure-from-motion, enhancing reconstruction accuracy.

Findings

Effectively resolves scale ambiguity without reference objects.

Works across diverse scenes and camera setups.

Provides publicly available code and data.

Abstract

Multi-view 3D reconstruction, namely, structure-from-motion followed by multi-view stereo, is a fundamental component of 3D computer vision. In general, multi-view 3D reconstruction suffers from an unknown scale ambiguity unless a reference object of known size is present in the scene. In this article, we show that multi-view images captured using a dual-pixel (DP) sensor can automatically resolve the scale ambiguity, without requiring a reference object or prior calibration. Specifically, the defocus blur observed in DP images provides sufficient information to determine the absolute scale when paired with depth maps (up to scale) recovered from multi-view 3D reconstruction. Based on this observation, we develop a simple yet effective linear method to estimate the absolute scale, followed by the intensity-based optimization stage that aligns the left and right DP images by shifting them back toward each other using cross-view blur kernels. Experiments demonstrate the effectiveness of the proposed approach across diverse scenes captured with different cameras and lenses. Code and data are available at https://github.com/lilika-makabe/dp-sfm-tpami.git