Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videos

TL;DR

This paper introduces a novel self-supervised method that combines semantic and temporal cues to improve object-centric learning in videos, achieving state-of-the-art results in unsupervised discovery and label propagation.

Contribution

It proposes a semantic-aware masked slot attention mechanism that integrates semantic segmentation and temporal correspondence for better object instance identification.

Findings

Effective identification of multiple object instances with semantic structure

State-of-the-art performance on dense label propagation tasks

Promising results in unsupervised video object discovery

Abstract

Self-supervised methods have shown remarkable progress in learning high-level semantics and low-level temporal correspondence. Building on these results, we take one step further and explore the possibility of integrating these two features to enhance object-centric representations. Our preliminary experiments indicate that query slot attention can extract different semantic components from the RGB feature map, while random sampling based slot attention can exploit temporal correspondence cues between frames to assist instance identification. Motivated by this, we propose a novel semantic-aware masked slot attention on top of the fused semantic features and correspondence maps. It comprises two slot attention stages with a set of shared learnable Gaussian distributions. In the first stage, we use the mean vectors as slot initialization to decompose potential semantics and generate semantic segmentation masks through iterative attention. In the second stage, for each semantics, we randomly sample slots from the corresponding Gaussian distribution and perform masked feature aggregation within the semantic area to exploit temporal correspondence patterns for instance identification. We adopt semantic- and instance-level temporal consistency as self-supervision to encourage temporally coherent object-centric representations. Our model effectively identifies multiple object instances with semantic structure, reaching promising results on unsupervised video object discovery. Furthermore, we achieve state-of-the-art performance on dense label propagation tasks, demonstrating the potential for object-centric analysis. The code is released at https://github.com/shvdiwnkozbw/SMTC.