SlotGNN: Unsupervised Discovery of Multi-Object Representations and Visual Dynamics

TL;DR

This paper introduces SlotGNN, an unsupervised framework combining SlotTransport for object discovery and SlotGNN for predicting multi-object dynamics, demonstrating robustness and accuracy in both simulated and real-world robotic tasks.

Contribution

The paper presents two novel architectures, SlotTransport and SlotGNN, enabling unsupervised discovery of object representations and their dynamics from visual data in robotic environments.

Findings

SlotTransport accurately encodes visual and positional information of objects.

SlotGNN effectively predicts future states and dynamics of multi-object scenes.

Framework performs well in real-world robotic control tasks.

Abstract

Learning multi-object dynamics from visual data using unsupervised techniques is challenging due to the need for robust, object representations that can be learned through robot interactions. This paper presents a novel framework with two new architectures: SlotTransport for discovering object representations from RGB images and SlotGNN for predicting their collective dynamics from RGB images and robot interactions. Our SlotTransport architecture is based on slot attention for unsupervised object discovery and uses a feature transport mechanism to maintain temporal alignment in object-centric representations. This enables the discovery of slots that consistently reflect the composition of multi-object scenes. These slots robustly bind to distinct objects, even under heavy occlusion or absence. Our SlotGNN, a novel unsupervised graph-based dynamics model, predicts the future state of multi-object scenes. SlotGNN learns a graph representation of the scene using the discovered slots from SlotTransport and performs relational and spatial reasoning to predict the future appearance of each slot conditioned on robot actions. We demonstrate the effectiveness of SlotTransport in learning object-centric features that accurately encode both visual and positional information. Further, we highlight the accuracy of SlotGNN in downstream robotic tasks, including challenging multi-object rearrangement and long-horizon prediction. Finally, our unsupervised approach proves effective in the real world. With only minimal additional data, our framework robustly predicts slots and their corresponding dynamics in real-world control tasks.