Towards Segmenting Consumer Stereo Videos: Benchmark, Baselines and Ensembles

TL;DR

This paper introduces a new benchmark for consumer stereo video segmentation, evaluates existing methods, and proposes an adaptive ensemble approach with a learned regressor that outperforms current techniques.

Contribution

It presents the first benchmark dataset and metrics for stereo video segmentation, and develops a novel ensemble method with a learned regressor for improved performance.

Findings

The ensemble method outperforms individual segmentation techniques.

The learned regressor adapts segmentation to each stereo video.

The proposed approach surpasses recent RGB-D segmentation methods.

Abstract

Are we ready to segment consumer stereo videos? The amount of this data type is rapidly increasing and encompasses rich information of appearance, motion and depth cues. However, the segmentation of such data is still largely unexplored. First, we propose therefore a new benchmark: videos, annotations and metrics to measure progress on this emerging challenge. Second, we evaluate several state of the art segmentation methods and propose a novel ensemble method based on recent spectral theory. This combines existing image and video segmentation techniques in an efficient scheme. Finally, we propose and integrate into this model a novel regressor, learnt to optimize the stereo segmentation performance directly via a differentiable proxy. The regressor makes our segmentation ensemble adaptive to each stereo video and outperforms the segmentations of the ensemble as well as a most recent RGB-D segmentation technique.