Online Mutual Foreground Segmentation for Multispectral Stereo Videos

TL;DR

This paper introduces a novel iterative method for simultaneous multispectral stereo video segmentation and registration, leveraging shape and appearance cues with temporal coherence to improve object segmentation in challenging conditions.

Contribution

It presents a new approach that jointly addresses multispectral segmentation and stereo registration through alternating minimization and dynamic priors, enhancing accuracy in low contrast and complex scenes.

Findings

Effective in diverse multispectral datasets

Improves segmentation accuracy in low contrast regions

Enhances temporal coherence of segmentation results

Abstract

The segmentation of video sequences into foreground and background regions is a low-level process commonly used in video content analysis and smart surveillance applications. Using a multispectral camera setup can improve this process by providing more diverse data to help identify objects despite adverse imaging conditions. The registration of several data sources is however not trivial if the appearance of objects produced by each sensor differs substantially. This problem is further complicated when parallax effects cannot be ignored when using close-range stereo pairs. In this work, we present a new method to simultaneously tackle multispectral segmentation and stereo registration. Using an iterative procedure, we estimate the labeling result for one problem using the provisional result of the other. Our approach is based on the alternating minimization of two energy functions that are linked through the use of dynamic priors. We rely on the integration of shape and appearance cues to find proper multispectral correspondences, and to properly segment objects in low contrast regions. We also formulate our model as a frame processing pipeline using higher order terms to improve the temporal coherence of our results. Our method is evaluated under different configurations on multiple multispectral datasets, and our implementation is available online.