EC-Diffuser: Multi-Object Manipulation via Entity-Centric Behavior Generation

TL;DR

EC-Diffuser introduces an entity-centric Transformer with diffusion-based optimization for multi-object manipulation, enabling efficient offline learning and zero-shot generalization to unseen object configurations and goals.

Contribution

The paper presents a novel behavioral cloning approach using object-centric representations and an entity-centric Transformer with diffusion models for improved multi-object manipulation.

Findings

Achieves significant performance improvements in multi-object tasks.

Enables zero-shot generalization to unseen object configurations.

Handles larger numbers of objects than during training.

Abstract

Object manipulation is a common component of everyday tasks, but learning to manipulate objects from high-dimensional observations presents significant challenges. These challenges are heightened in multi-object environments due to the combinatorial complexity of the state space as well as of the desired behaviors. While recent approaches have utilized large-scale offline data to train models from pixel observations, achieving performance gains through scaling, these methods struggle with compositional generalization in unseen object configurations with constrained network and dataset sizes. To address these issues, we propose a novel behavioral cloning (BC) approach that leverages object-centric representations and an entity-centric Transformer with diffusion-based optimization, enabling efficient learning from offline image data. Our method first decomposes observations into an object-centric representation, which is then processed by our entity-centric Transformer that computes attention at the object level, simultaneously predicting object dynamics and the agent's actions. Combined with the ability of diffusion models to capture multi-modal behavior distributions, this results in substantial performance improvements in multi-object tasks and, more importantly, enables compositional generalization. We present BC agents capable of zero-shot generalization to tasks with novel compositions of objects and goals, including larger numbers of objects than seen during training. We provide video rollouts on our webpage: https://sites.google.com/view/ec-diffuser.