Neural Field Representations of Articulated Objects for Robotic Manipulation Planning

TL;DR

This paper introduces a Neural Field Representation that allows robots to plan and execute manipulation tasks on articulated objects directly from images, bypassing explicit geometric modeling.

Contribution

The paper presents a novel neural representation for articulated objects that supports manipulation planning, shape reconstruction, and segmentation directly from images, trained solely on synthetic data.

Findings

Model generalizes to unseen objects within the same class

Enables real-world robotic manipulation from images

Supports shape reconstruction and semantic segmentation

Abstract

Traditional approaches for manipulation planning rely on an explicit geometric model of the environment to formulate a given task as an optimization problem. However, inferring an accurate model from raw sensor input is a hard problem in itself, in particular for articulated objects (e.g., closets, drawers). In this paper, we propose a Neural Field Representation (NFR) of articulated objects that enables manipulation planning directly from images. Specifically, after taking a few pictures of a new articulated object, we can forward simulate its possible movements, and, therefore, use this neural model directly for planning with trajectory optimization. Additionally, this representation can be used for shape reconstruction, semantic segmentation and image rendering, which provides a strong supervision signal during training and generalization. We show that our model, which was trained only on synthetic images, is able to extract a meaningful representation for unseen objects of the same class, both in simulation and with real images. Furthermore, we demonstrate that the representation enables robotic manipulation of an articulated object in the real world directly from images.