Iterative Knowledge Exchange Between Deep Learning and Space-Time Spectral Clustering for Unsupervised Segmentation in Videos

TL;DR

This paper introduces an iterative dual system combining space-time spectral clustering and deep learning for unsupervised video object segmentation, achieving state-of-the-art results on multiple datasets.

Contribution

It presents a novel cyclical knowledge exchange framework between spectral clustering and deep networks, enhancing unsupervised and supervised video segmentation performance.

Findings

Achieves state-of-the-art results on DAVIS, SegTrack, YouTube-Objects, and DAVSOD datasets.

Introduces a fast spectral clustering method using power iteration without matrix computation.

Demonstrates effective knowledge exchange improves segmentation accuracy.

Abstract

We propose a dual system for unsupervised object segmentation in video, which brings together two modules with complementary properties: a space-time graph that discovers objects in videos and a deep network that learns powerful object features. The system uses an iterative knowledge exchange policy. A novel spectral space-time clustering process on the graph produces unsupervised segmentation masks passed to the network as pseudo-labels. The net learns to segment in single frames what the graph discovers in video and passes back to the graph strong image-level features that improve its node-level features in the next iteration. Knowledge is exchanged for several cycles until convergence. The graph has one node per each video pixel, but the object discovery is fast. It uses a novel power iteration algorithm computing the main space-time cluster as the principal eigenvector of a special Feature-Motion matrix without actually computing the matrix. The thorough experimental analysis validates our theoretical claims and proves the effectiveness of the cyclical knowledge exchange. We also perform experiments on the supervised scenario, incorporating features pretrained with human supervision. We achieve state-of-the-art level on unsupervised and supervised scenarios on four challenging datasets: DAVIS, SegTrack, YouTube-Objects, and DAVSOD.