Object-Centric World Models for Causality-Aware Reinforcement Learning

TL;DR

STICA introduces an object-centric, causality-aware world model for reinforcement learning, improving sample efficiency and performance by decomposing environments into objects and modeling their interactions.

Contribution

It proposes a novel framework combining object-centric Transformers with causality-aware policy networks for better environment understanding.

Findings

STICA outperforms state-of-the-art agents on object-rich benchmarks.

It achieves higher sample efficiency and final performance.

Token-level causality modeling enhances decision-making.

Abstract

World models have been developed to support sample-efficient deep reinforcement learning agents. However, it remains challenging for world models to accurately replicate environments that are high-dimensional, non-stationary, and composed of multiple objects with rich interactions since most world models learn holistic representations of all environmental components. By contrast, humans perceive the environment by decomposing it into discrete objects, facilitating efficient decision-making. Motivated by this insight, we propose \emph{Slot Transformer Imagination with CAusality-aware reinforcement learning} (STICA), a unified framework in which object-centric Transformers serve as the world model and causality-aware policy and value networks. STICA represents each observation as a set of object-centric tokens, together with tokens for the agent action and the resulting reward, enabling the world model to predict token-level dynamics and interactions. The policy and value networks then estimate token-level cause--effect relations and use them in the attention layers, yielding causality-guided decision-making. Experiments on object-rich benchmarks demonstrate that STICA consistently outperforms state-of-the-art agents in both sample efficiency and final performance.