Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera

TL;DR

GLNet is a self-supervised framework that jointly learns depth, optical flow, camera pose, and intrinsics from monocular video by leveraging geometric constraints and online refinement, outperforming previous methods on benchmark datasets.

Contribution

The paper introduces new geometric loss functions, extends the model to predict camera intrinsics, and proposes online refinement strategies for improved self-supervised learning.

Findings

Outperforms previous self-supervised methods on KITTI and Cityscapes.

Demonstrates good transfer learning capabilities on YouTube videos.

Effectively integrates geometric constraints for joint task learning.

Abstract

We present GLNet, a self-supervised framework for learning depth, optical flow, camera pose and intrinsic parameters from monocular video - addressing the difficulty of acquiring realistic ground-truth for such tasks. We propose three contributions: 1) we design new loss functions that capture multiple geometric constraints (eg. epipolar geometry) as well as an adaptive photometric loss that supports multiple moving objects, rigid and non-rigid, 2) we extend the model such that it predicts camera intrinsics, making it applicable to uncalibrated video, and 3) we propose several online refinement strategies that rely on the symmetry of our self-supervised loss in training and testing, in particular optimizing model parameters and/or the output of different tasks, thus leveraging their mutual interactions. The idea of jointly optimizing the system output, under all geometric and photometric constraints can be viewed as a dense generalization of classical bundle adjustment. We demonstrate the effectiveness of our method on KITTI and Cityscapes, where we outperform previous self-supervised approaches on multiple tasks. We also show good generalization for transfer learning in YouTube videos.