Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulation

TL;DR

This paper introduces a structured world model-based approach for intrinsic motivation-driven exploration in multi-object environments, enabling zero-shot generalization to complex manipulation tasks without additional training.

Contribution

It presents a novel method that combines relational inductive biases with model-based planning for efficient exploration and zero-shot task generalization in object manipulation.

Findings

Achieves sample-efficient exploration in multi-object scenarios.

Enables zero-shot generalization to tasks like stacking and flipping.

Develops interaction-rich behaviors starting from early exploration.

Abstract

It has been a long-standing dream to design artificial agents that explore their environment efficiently via intrinsic motivation, similar to how children perform curious free play. Despite recent advances in intrinsically motivated reinforcement learning (RL), sample-efficient exploration in object manipulation scenarios remains a significant challenge as most of the relevant information lies in the sparse agent-object and object-object interactions. In this paper, we propose to use structured world models to incorporate relational inductive biases in the control loop to achieve sample-efficient and interaction-rich exploration in compositional multi-object environments. By planning for future novelty inside structured world models, our method generates free-play behavior that starts to interact with objects early on and develops more complex behavior over time. Instead of using models only to compute intrinsic rewards, as commonly done, our method showcases that the self-reinforcing cycle between good models and good exploration also opens up another avenue: zero-shot generalization to downstream tasks via model-based planning. After the entirely intrinsic task-agnostic exploration phase, our method solves challenging downstream tasks such as stacking, flipping, pick & place, and throwing that generalizes to unseen numbers and arrangements of objects without any additional training.