From Vicious to Virtuous Cycles: Synergistic Representation Learning for Unsupervised Video Object-Centric Learning

TL;DR

This paper introduces Synergistic Representation Learning (SRL), a novel method that enables mutual refinement between encoder and decoder in unsupervised video object-centric models, overcoming the reconstruction conflict and achieving state-of-the-art results.

Contribution

SRL establishes a virtuous cycle between encoder and decoder, improving scene decomposition by mutual refinement and stabilizing training with a warm-up phase.

Findings

Achieves state-of-the-art results on video object-centric benchmarks.

Effectively deblurs semantic boundaries in decoder outputs.

Reduces noise in encoder features through decoder supervision.

Abstract

Unsupervised object-centric learning models, particularly slot-based architectures, have shown great promise in decomposing complex scenes. However, their reliance on reconstruction-based training creates a fundamental conflict between the sharp, high-frequency attention maps of the encoder and the spatially consistent but blurry reconstruction maps of the decoder. We identify that this discrepancy gives rise to a vicious cycle: the noisy feature map from the encoder forces the decoder to average over possibilities and produce even blurrier outputs, while the gradient computed from blurry reconstruction maps lacks high-frequency details necessary to supervise encoder features. To break this cycle, we introduce Synergistic Representation Learning (SRL) that establishes a virtuous cycle where the encoder and decoder mutually refine one another. SRL leverages the encoder's sharpness to deblur the semantic boundary within the decoder output, while exploiting the decoder's spatial consistency to denoise the encoder's features. This mutual refinement process is stabilized by a warm-up phase with a slot regularization objective that initially allocates distinct entities per slot. By bridging the representational gap between the encoder and decoder, SRL achieves state-of-the-art results on video object-centric learning benchmarks. Codes are available at https://github.com/hynnsk/SRL.