Reasoning-Enhanced Object-Centric Learning for Videos

TL;DR

This paper introduces a novel reasoning module called STATM that enhances object-centric video models by improving their perception, prediction, and reasoning abilities, especially in complex scenes, and demonstrates its effectiveness across multiple tasks.

Contribution

The paper proposes the STATM module, a new reasoning component that significantly boosts the performance of object-centric video models in perception and reasoning tasks.

Findings

STATM improves object segmentation and tracking accuracy.

Enhanced performance in downstream prediction tasks.

Effective in Visual Question Answering (VQA) applications.

Abstract

Object-centric learning aims to break down complex visual scenes into more manageable object representations, enhancing the understanding and reasoning abilities of machine learning systems toward the physical world. Recently, slot-based video models have demonstrated remarkable proficiency in segmenting and tracking objects, but they overlook the importance of the effective reasoning module. In the real world, reasoning and predictive abilities play a crucial role in human perception and object tracking; in particular, these abilities are closely related to human intuitive physics. Inspired by this, we designed a novel reasoning module called the Slot-based Time-Space Transformer with Memory buffer (STATM) to enhance the model's perception ability in complex scenes. The memory buffer primarily serves as storage for slot information from upstream modules, the Slot-based Time-Space Transformer makes predictions through slot-based spatiotemporal attention computations and fusion. Our experimental results on various datasets indicate that the STATM module can significantly enhance the capabilities of multiple state-of-the-art object-centric learning models for video. Moreover, as a predictive model, the STATM module also performs well in downstream prediction and Visual Question Answering (VQA) tasks. We will release our codes and data at https://github.com/intell-sci-comput/STATM.