Multi-Label Stereo Matching for Transparent Scene Depth Estimation

TL;DR

This paper introduces a multi-label stereo matching approach using a Gaussian representation and GRU framework to accurately estimate depths of transparent objects and backgrounds, outperforming previous single-label methods.

Contribution

It proposes a novel multi-label regression formulation with a Gaussian model and iterative prediction for transparent scene depth estimation, addressing limitations of single-label approaches.

Findings

Significantly improves transparent surface depth estimation.

Effectively preserves background scene information.

Validated on a synthesized dataset with 10 scenes and 89 objects.

Abstract

In this paper, we present a multi-label stereo matching method to simultaneously estimate the depth of the transparent objects and the occluded background in transparent scenes.Unlike previous methods that assume a unimodal distribution along the disparity dimension and formulate the matching as a single-label regression problem, we propose a multi-label regression formulation to estimate multiple depth values at the same pixel in transparent scenes. To resolve the multi-label regression problem, we introduce a pixel-wise multivariate Gaussian representation, where the mean vector encodes multiple depth values at the same pixel, and the covariance matrix determines whether a multi-label representation is necessary for a given pixel. The representation is iteratively predicted within a GRU framework. In each iteration, we first predict the update step for the mean parameters and then use both the update step and the updated mean parameters to estimate the covariance matrix. We also synthesize a dataset containing 10 scenes and 89 objects to validate the performance of transparent scene depth estimation. The experiments show that our method greatly improves the performance on transparent surfaces while preserving the background information for scene reconstruction. Code is available at https://github.com/BFZD233/TranScene.