Visual Robot Task Planning

TL;DR

This paper introduces a neural network-based planning system that predicts future states, enabling robots to simulate and evaluate action sequences for complex tasks across various environments using visual data.

Contribution

It presents a novel neural architecture and planning algorithm that learns a generative world model for prospective simulation and decision-making in robotics.

Findings

Successfully learned world representations for multiple tasks

Enabled visualization of intermediate goals during planning

Demonstrated effective planning in diverse environments

Abstract

Prospection, the act of predicting the consequences of many possible futures, is intrinsic to human planning and action, and may even be at the root of consciousness. Surprisingly, this idea has been explored comparatively little in robotics. In this work, we propose a neural network architecture and associated planning algorithm that (1) learns a representation of the world useful for generating prospective futures after the application of high-level actions, (2) uses this generative model to simulate the result of sequences of high-level actions in a variety of environments, and (3) uses this same representation to evaluate these actions and perform tree search to find a sequence of high-level actions in a new environment. Models are trained via imitation learning on a variety of domains, including navigation, pick-and-place, and a surgical robotics task. Our approach allows us to visualize intermediate motion goals and learn to plan complex activity from visual information.