Representing Positional Information in Generative World Models for Object Manipulation

TL;DR

This paper enhances generative world models with explicit positional representations to improve object manipulation, enabling more accurate goal achievement in robotics tasks.

Contribution

It introduces position-conditioned and latent-conditioned approaches that explicitly encode object positions, improving manipulation success over existing models.

Findings

LCP captures object positions effectively for manipulation tasks.

Methods outperform current model-based control approaches.

Enables multimodal goal specification through spatial or visual cues.

Abstract

Object manipulation capabilities are essential skills that set apart embodied agents engaging with the world, especially in the realm of robotics. The ability to predict outcomes of interactions with objects is paramount in this setting. While model-based control methods have started to be employed for tackling manipulation tasks, they have faced challenges in accurately manipulating objects. As we analyze the causes of this limitation, we identify the cause of underperformance in the way current world models represent crucial positional information, especially about the target's goal specification for object positioning tasks. We introduce a general approach that empowers world model-based agents to effectively solve object-positioning tasks. We propose two declinations of this approach for generative world models: position-conditioned (PCP) and latent-conditioned (LCP) policy learning. In particular, LCP employs object-centric latent representations that explicitly capture object positional information for goal specification. This naturally leads to the emergence of multimodal capabilities, enabling the specification of goals through spatial coordinates or a visual goal. Our methods are rigorously evaluated across several manipulation environments, showing favorable performance compared to current model-based control approaches.