TemporalStereo: Efficient Spatial-Temporal Stereo Matching Network

TL;DR

TemporalStereo is an efficient stereo matching network that leverages spatial and temporal information to improve accuracy, especially in challenging regions like occlusions and reflections, and performs well on various datasets.

Contribution

It introduces a novel coarse-to-fine network that exploits spatio-temporal data for improved stereo matching, functioning effectively in both single-pair and video modes.

Findings

Achieves state-of-the-art results on multiple datasets

Robust to dynamic objects in videos

Effective in handling occlusions and reflective regions

Abstract

We present TemporalStereo, a coarse-to-fine stereo matching network that is highly efficient, and able to effectively exploit the past geometry and context information to boost matching accuracy. Our network leverages sparse cost volume and proves to be effective when a single stereo pair is given. However, its peculiar ability to use spatio-temporal information across stereo sequences allows TemporalStereo to alleviate problems such as occlusions and reflective regions while enjoying high efficiency also in this latter case. Notably, our model -- trained once with stereo videos -- can run in both single-pair and temporal modes seamlessly. Experiments show that our network relying on camera motion is robust even to dynamic objects when running on videos. We validate TemporalStereo through extensive experiments on synthetic (SceneFlow, TartanAir) and real (KITTI 2012, KITTI 2015) datasets. Our model achieves state-of-the-art performance on any of these datasets. Code is available at \url{https://github.com/youmi-zym/TemporalStereo.git}.