Do End-to-end Stereo Algorithms Under-utilize Information?

TL;DR

This paper enhances end-to-end stereo matching networks by integrating adaptive filtering and semi-global aggregation, utilizing RGB information to improve disparity accuracy, especially near occlusions and thin structures.

Contribution

It introduces a method to incorporate deep adaptive filtering and differentiable semi-global aggregation into existing stereo networks, leveraging RGB cues for better disparity estimation.

Findings

Significant accuracy improvements on KITTI datasets.

Enhanced handling of occlusion boundaries and thin structures.

Effective integration across multiple existing architectures.

Abstract

Deep networks for stereo matching typically leverage 2D or 3D convolutional encoder-decoder architectures to aggregate cost and regularize the cost volume for accurate disparity estimation. Due to content-insensitive convolutions and down-sampling and up-sampling operations, these cost aggregation mechanisms do not take full advantage of the information available in the images. Disparity maps suffer from over-smoothing near occlusion boundaries, and erroneous predictions in thin structures. In this paper, we show how deep adaptive filtering and differentiable semi-global aggregation can be integrated in existing 2D and 3D convolutional networks for end-to-end stereo matching, leading to improved accuracy. The improvements are due to utilizing RGB information from the images as a signal to dynamically guide the matching process, in addition to being the signal we attempt to match across the images. We show extensive experimental results on the KITTI 2015 and Virtual KITTI 2 datasets comparing four stereo networks (DispNetC, GCNet, PSMNet and GANet) after integrating four adaptive filters (segmentation-aware bilateral filtering, dynamic filtering networks, pixel adaptive convolution and semi-global aggregation) into their architectures. Our code is available at https://github.com/ccj5351/DAFStereoNets.