Flow-Motion and Depth Network for Monocular Stereo and Beyond

TL;DR

This paper introduces a learning-based approach for monocular stereo that jointly estimates optical flow, camera motion, and depth, extending to fuse multiple views, with a novel triangulation layer and dataset tools, achieving state-of-the-art results.

Contribution

The method's novelty lies in joint optical flow and camera motion estimation, a new triangulation layer, and multi-view depth fusion, enabling efficient and accurate monocular stereo and beyond.

Findings

Achieves state-of-the-art accuracy in monocular stereo depth estimation.

Extends to fuse multiple target images for improved depth maps.

Demonstrates strong generalization on real-world and Google Earth images.

Abstract

We propose a learning-based method that solves monocular stereo and can be extended to fuse depth information from multiple target frames. Given two unconstrained images from a monocular camera with known intrinsic calibration, our network estimates relative camera poses and the depth map of the source image. The core contribution of the proposed method is threefold. First, a network is tailored for static scenes that jointly estimates the optical flow and camera motion. By the joint estimation, the optical flow search space is gradually reduced resulting in an efficient and accurate flow estimation. Second, a novel triangulation layer is proposed to encode the estimated optical flow and camera motion while avoiding common numerical issues caused by epipolar. Third, beyond two-view depth estimation, we further extend the above networks to fuse depth information from multiple target images and estimate the depth map of the source image. To further benefit the research community, we introduce tools to generate photorealistic structure-from-motion datasets such that deep networks can be well trained and evaluated. The proposed method is compared with previous methods and achieves state-of-the-art results within less time. Images from real-world applications and Google Earth are used to demonstrate the generalization ability of the method.