SfM-Net: Learning of Structure and Motion from Video

TL;DR

SfM-Net is a neural network that learns to estimate scene depth, camera motion, and object motion from video sequences, using various supervision levels, and can produce accurate motion and segmentation results.

Contribution

The paper introduces SfM-Net, a geometry-aware neural network capable of unsupervised, semi-supervised, and supervised learning for motion and depth estimation from videos.

Findings

Accurately estimates depth and camera motion from videos.

Successfully segments moving objects without explicit supervision.

Operates effectively under different supervision regimes.

Abstract

We propose SfM-Net, a geometry-aware neural network for motion estimation in videos that decomposes frame-to-frame pixel motion in terms of scene and object depth, camera motion and 3D object rotations and translations. Given a sequence of frames, SfM-Net predicts depth, segmentation, camera and rigid object motions, converts those into a dense frame-to-frame motion field (optical flow), differentiably warps frames in time to match pixels and back-propagates. The model can be trained with various degrees of supervision: 1) self-supervised by the re-projection photometric error (completely unsupervised), 2) supervised by ego-motion (camera motion), or 3) supervised by depth (e.g., as provided by RGBD sensors). SfM-Net extracts meaningful depth estimates and successfully estimates frame-to-frame camera rotations and translations. It often successfully segments the moving objects in the scene, even though such supervision is never provided.