Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Image

TL;DR

This paper introduces a deep neural network that predicts action sequences from scene images to improve the efficiency and scalability of task and motion planning, generalizing across scenes with varying objects.

Contribution

The novel neural network directly predicts promising action sequences from scene images, reducing the need for multiple optimization problems in TAMP and generalizing to scenes with many objects.

Findings

Significant runtime improvements in TAMP tasks.

Effective generalization to scenes with many objects.

Successful prediction of action sequences from initial scene images.

Abstract

In this paper, we propose a deep convolutional recurrent neural network that predicts action sequences for task and motion planning (TAMP) from an initial scene image. Typical TAMP problems are formalized by combining reasoning on a symbolic, discrete level (e.g. first-order logic) with continuous motion planning such as nonlinear trajectory optimization. Due to the great combinatorial complexity of possible discrete action sequences, a large number of optimization/motion planning problems have to be solved to find a solution, which limits the scalability of these approaches. To circumvent this combinatorial complexity, we develop a neural network which, based on an initial image of the scene, directly predicts promising discrete action sequences such that ideally only one motion planning problem has to be solved to find a solution to the overall TAMP problem. A key aspect is that our method generalizes to scenes with many and varying number of objects, although being trained on only two objects at a time. This is possible by encoding the objects of the scene in images as input to the neural network, instead of a fixed feature vector. Results show runtime improvements of several magnitudes. Video: https://youtu.be/i8yyEbbvoEk