Monocular Depth Estimation using Multi-Scale Continuous CRFs as Sequential Deep Networks

TL;DR

This paper introduces a novel deep learning approach for monocular depth estimation that integrates multi-scale CNN features using continuous CRFs, enabling end-to-end training and achieving state-of-the-art results.

Contribution

It proposes a new model combining multi-scale CNN outputs with continuous CRFs in a sequential deep network framework, trained end-to-end.

Findings

Achieved new state-of-the-art results on NYUD-V2, Make3D, and KITTI datasets.

Demonstrated the effectiveness of continuous CRFs for multi-scale feature fusion.

Proposed a novel CNN implementation of mean-field updates for continuous CRFs.

Abstract

Depth cues have been proved very useful in various computer vision and robotic tasks. This paper addresses the problem of monocular depth estimation from a single still image. Inspired by the effectiveness of recent works on multi-scale convolutional neural networks (CNN), we propose a deep model which fuses complementary information derived from multiple CNN side outputs. Different from previous methods using concatenation or weighted average schemes, the integration is obtained by means of continuous Conditional Random Fields (CRFs). In particular, we propose two different variations, one based on a cascade of multiple CRFs, the other on a unified graphical model. By designing a novel CNN implementation of mean-field updates for continuous CRFs, we show that both proposed models can be regarded as sequential deep networks and that training can be performed end-to-end. Through an extensive experimental evaluation, we demonstrate the effectiveness of the proposed approach and establish new state of the art results for the monocular depth estimation task on three publicly available datasets, i.e. NYUD-V2, Make3D and KITTI.