SegStereo: Exploiting Semantic Information for Disparity Estimation

TL;DR

SegStereo leverages semantic information to enhance disparity estimation in stereo images, significantly improving accuracy and robustness across multiple datasets by integrating semantic features and loss regularization.

Contribution

The paper introduces a unified model that incorporates semantic features and a semantic softmax loss to improve disparity prediction in stereo images, applicable in both supervised and unsupervised settings.

Findings

Achieves state-of-the-art results on KITTI benchmark.

Improves disparity accuracy in featureless regions.

Performs well on CityScapes and FlyingThings3D datasets.

Abstract

Disparity estimation for binocular stereo images finds a wide range of applications. Traditional algorithms may fail on featureless regions, which could be handled by high-level clues such as semantic segments. In this paper, we suggest that appropriate incorporation of semantic cues can greatly rectify prediction in commonly-used disparity estimation frameworks. Our method conducts semantic feature embedding and regularizes semantic cues as the loss term to improve learning disparity. Our unified model SegStereo employs semantic features from segmentation and introduces semantic softmax loss, which helps improve the prediction accuracy of disparity maps. The semantic cues work well in both unsupervised and supervised manners. SegStereo achieves state-of-the-art results on KITTI Stereo benchmark and produces decent prediction on both CityScapes and FlyingThings3D datasets.