Toward A Better Understanding of Monocular Depth Evaluation

TL;DR

This paper analyzes current monocular depth evaluation metrics, revealing their insensitivity to certain surface perturbations, and introduces a new normal-based metric and tools to improve alignment with human judgment.

Contribution

It provides a comprehensive analysis of existing metrics, identifies their limitations, and proposes a novel normal-based metric along with visualization tools for better evaluation.

Findings

Existing metrics are under-sensitive to curvature perturbations.

The new normal-based metric improves alignment with human judgment.

Tools for depth visualization enhance evaluation understanding.

Abstract

Monocular depth estimation is an important task with rapid progress, but how to evaluate it is not fully resolved, as evidenced by a lack of standardization in existing literature and a large selection of evaluation metrics whose trade-offs and behaviors are not fully understood. This paper contributes a novel, quantitative analysis of existing metrics in terms of their sensitivity to various types of perturbations of ground truth, emphasizing comparison to human judgment. Our analysis reveals that existing metrics are severely under-sensitive to curvature perturbation such as making smooth surfaces bumpy. To remedy this, we introduce a new metric based on relative surface normals, along with new depth visualization tools and a principled method to create composite metrics with better human alignment. Code and data are available at: https://github.com/princeton-vl/evalmde.