Counterfactual Depth from a Single RGB Image

TL;DR

This paper introduces a novel method for predicting high-resolution, pixel-level depth maps of hidden scene geometry from a single RGB image, without needing depth sensors, by leveraging scene completion principles.

Contribution

It presents a new approach that predicts counterfactual depth at pixel-level resolution from a single RGB image, surpassing existing scene completion methods in detail and without requiring RGBD data.

Findings

Depth predictions outperform baselines on collected dataset.

Method achieves higher resolution than voxel-based scene completion.

Operates effectively without RGBD inputs.

Abstract

We describe a method that predicts, from a single RGB image, a depth map that describes the scene when a masked object is removed - we call this "counterfactual depth" that models hidden scene geometry together with the observations. Our method works for the same reason that scene completion works: the spatial structure of objects is simple. But we offer a much higher resolution representation of space than current scene completion methods, as we operate at pixel-level precision and do not rely on a voxel representation. Furthermore, we do not require RGBD inputs. Our method uses a standard encoder-decoder architecture, and with a decoder modified to accept an object mask. We describe a small evaluation dataset that we have collected, which allows inference about what factors affect reconstruction most strongly. Using this dataset, we show that our depth predictions for masked objects are better than other baselines.