Interpretable Robotic Manipulation from Language

TL;DR

This paper presents Ex-PERACT, an explainable hierarchical agent that uses natural language to improve robotic manipulation learning, achieving competitive performance and better human-machine instruction translation.

Contribution

Introduction of Ex-PERACT, a hierarchical, language-integrated behavior cloning agent for manipulation tasks, enhancing interpretability and instruction understanding in robots.

Findings

Achieves competitive performance on RLBench tasks.

Effectively bridges human instructions and robotic actions.

Demonstrates interpretability in manipulation policies.

Abstract

Humans naturally employ linguistic instructions to convey knowledge, a process that proves significantly more complex for machines, especially within the context of multitask robotic manipulation environments. Natural language, moreover, serves as the primary medium through which humans acquire new knowledge, presenting a potentially intuitive bridge for translating concepts understandable by humans into formats that can be learned by machines. In pursuit of facilitating this integration, we introduce an explainable behavior cloning agent, named Ex-PERACT, specifically designed for manipulation tasks. This agent is distinguished by its hierarchical structure, which incorporates natural language to enhance the learning process. At the top level, the model is tasked with learning a discrete skill code, while at the bottom level, the policy network translates the problem into a voxelized grid and maps the discretized actions to voxel grids. We evaluate our method across eight challenging manipulation tasks utilizing the RLBench benchmark, demonstrating that Ex-PERACT not only achieves competitive policy performance but also effectively bridges the gap between human instructions and machine execution in complex environments.