NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation

TL;DR

This paper introduces a novel neural window fully-connected CRFs approach for monocular depth estimation, leveraging multi-head attention within windowed regions to improve accuracy and computational efficiency.

Contribution

It proposes a new method combining neural window FC-CRFs with multi-head attention and a transformer-based encoder-decoder structure for better depth estimation.

Findings

Significant performance improvements on KITTI and NYUv2 datasets.

Effective application to panorama images, outperforming previous methods.

Reduced computational complexity enabling fully-connected CRFs.

Abstract

Estimating the accurate depth from a single image is challenging since it is inherently ambiguous and ill-posed. While recent works design increasingly complicated and powerful networks to directly regress the depth map, we take the path of CRFs optimization. Due to the expensive computation, CRFs are usually performed between neighborhoods rather than the whole graph. To leverage the potential of fully-connected CRFs, we split the input into windows and perform the FC-CRFs optimization within each window, which reduces the computation complexity and makes FC-CRFs feasible. To better capture the relationships between nodes in the graph, we exploit the multi-head attention mechanism to compute a multi-head potential function, which is fed to the networks to output an optimized depth map. Then we build a bottom-up-top-down structure, where this neural window FC-CRFs module serves as the decoder, and a vision transformer serves as the encoder. The experiments demonstrate that our method significantly improves the performance across all metrics on both the KITTI and NYUv2 datasets, compared to previous methods. Furthermore, the proposed method can be directly applied to panorama images and outperforms all previous panorama methods on the MatterPort3D dataset. Project page: https://weihaosky.github.io/newcrfs.