Unsupervised Video Segmentation via Spatio-Temporally Nonlocal Appearance Learning

TL;DR

This paper introduces a novel unsupervised video segmentation method that leverages long-term spatio-temporal nonlocal appearance information, improving segmentation stability amidst rapid motion and shape changes.

Contribution

It proposes a simple approach to mine long-term nonlocal appearance features using superpixels and KD-tree search, enhancing segmentation accuracy over existing methods.

Findings

Outperforms several state-of-the-art methods on SegTrack and Youtube-Objects datasets.

Effectively handles large viewpoint changes and non-rigid deformations.

Demonstrates the benefit of long-term nonlocal appearance information in video segmentation.

Abstract

Video object segmentation is challenging due to the factors like rapidly fast motion, cluttered backgrounds, arbitrary object appearance variation and shape deformation. Most existing methods only explore appearance information between two consecutive frames, which do not make full use of the usefully long-term nonlocal information that is helpful to make the learned appearance stable, and hence they tend to fail when the targets suffer from large viewpoint changes and significant non-rigid deformations. In this paper, we propose a simple yet effective approach to mine the long-term sptatio-temporally nonlocal appearance information for unsupervised video segmentation. The motivation of our algorithm comes from the spatio-temporal nonlocality of the region appearance reoccurrence in a video. Specifically, we first generate a set of superpixels to represent the foreground and background, and then update the appearance of each superpixel with its long-term sptatio-temporally nonlocal counterparts generated by the approximate nearest neighbor search method with the efficient KD-tree algorithm. Then, with the updated appearances, we formulate a spatio-temporal graphical model comprised of the superpixel label consistency potentials. Finally, we generate the segmentation by optimizing the graphical model via iteratively updating the appearance model and estimating the labels. Extensive evaluations on the SegTrack and Youtube-Objects datasets demonstrate the effectiveness of the proposed method, which performs favorably against some state-of-art methods.