Temporally Object-based Video Co-Segmentation

TL;DR

This paper introduces an unsupervised, object-based video co-segmentation framework that leverages temporal consistency and object proposals to accurately extract common foreground objects across videos.

Contribution

It presents a novel approach combining temporal proposal streams and a graphical model for improved co-segmentation of multiple objects in multiple videos.

Findings

Achieves improved performance on benchmark datasets.

Effectively handles multiple foreground objects.

Utilizes temporal consistency for better proposal enrichment.

Abstract

In this paper, we propose an unsupervised video object co-segmentation framework based on the primary object proposals to extract the common foreground object(s) from a given video set. In addition to the objectness attributes and motion coherence our framework exploits the temporal consistency of the object-like regions between adjacent frames to enrich the set of original object proposals. We call the enriched proposal sets temporal proposal streams, as they are composed of the most similar proposals from each frame augmented with predicted proposals using temporally consistent superpixel information. The temporal proposal streams represent all the possible region tubes of the objects. Therefore, we formulate a graphical model to select a proposal stream for each object in which the pairwise potentials consist of the appearance dissimilarity between different streams in the same video and also the similarity between the streams in different videos. This model is suitable for single (multiple) foreground objects in two (more) videos, which can be solved by any existing energy minimization method. We evaluate our proposed framework by comparing it to other video co-segmentation algorithms. Our method achieves improved performance on state-of-the-art benchmark datasets.