DoubleTake: Geometry Guided Depth Estimation

TL;DR

DoubleTake introduces a geometry-guided depth estimation method that leverages historical 3D scene data and a Hint MLP to improve accuracy and regularization, achieving state-of-the-art results in real-time applications.

Contribution

The paper presents a novel approach that incorporates prior 3D geometry as a hint into depth estimation, enhancing accuracy over traditional multi-view stereo methods.

Findings

Achieves state-of-the-art depth estimation accuracy.

Operates at interactive speeds suitable for real-time applications.

Improves scene reconstruction quality in offline and incremental scenarios.

Abstract

Estimating depth from a sequence of posed RGB images is a fundamental computer vision task, with applications in augmented reality, path planning etc. Prior work typically makes use of previous frames in a multi view stereo framework, relying on matching textures in a local neighborhood. In contrast, our model leverages historical predictions by giving the latest 3D geometry data as an extra input to our network. This self-generated geometric hint can encode information from areas of the scene not covered by the keyframes and it is more regularized when compared to individual predicted depth maps for previous frames. We introduce a Hint MLP which combines cost volume features with a hint of the prior geometry, rendered as a depth map from the current camera location, together with a measure of the confidence in the prior geometry. We demonstrate that our method, which can run at interactive speeds, achieves state-of-the-art estimates of depth and 3D scene reconstruction in both offline and incremental evaluation scenarios.