Temporally Consistent Stereo Matching

TL;DR

This paper introduces a novel video stereo matching approach that leverages temporal information through disparity completion and iterative refinement, significantly improving temporal consistency, accuracy, and efficiency in depth estimation from stereo video streams.

Contribution

The paper presents a new method for temporally consistent stereo matching that combines disparity completion with dual-space iterative refinement, addressing limitations of single-frame methods.

Findings

Reduces temporal inconsistency in depth maps

Improves accuracy in ill-posed regions

Enhances computational efficiency

Abstract

Stereo matching provides depth estimation from binocular images for downstream applications. These applications mostly take video streams as input and require temporally consistent depth maps. However, existing methods mainly focus on the estimation at the single-frame level. This commonly leads to temporally inconsistent results, especially in ill-posed regions. In this paper, we aim to leverage temporal information to improve the temporal consistency, accuracy, and efficiency of stereo matching. To achieve this, we formulate video stereo matching as a process of temporal disparity completion followed by continuous iterative refinements. Specifically, we first project the disparity of the previous timestamp to the current viewpoint, obtaining a semi-dense disparity map. Then, we complete this map through a disparity completion module to obtain a well-initialized disparity map. The state features from the current completion module and from the past refinement are fused together, providing a temporally coherent state for subsequent refinement. Based on this coherent state, we introduce a dual-space refinement module to iteratively refine the initialized result in both disparity and disparity gradient spaces, improving estimations in ill-posed regions. Extensive experiments demonstrate that our method effectively alleviates temporal inconsistency while enhancing both accuracy and efficiency.