Neural Task Planning with And-Or Graph Representations

TL;DR

This paper introduces a novel approach for semantic task planning in computer vision by combining LSTM networks with hierarchical And-Or Graph representations to generate training data and improve action sequence prediction.

Contribution

It proposes using an And-Or Graph to generate training samples for LSTM-based task planning, reducing the need for extensive annotated data.

Findings

Effective generation of diverse action sequences from limited annotations

Improved accuracy in predicting task-specific action sequences

Creation of a new dataset for daily task planning

Abstract

This paper focuses on semantic task planning, i.e., predicting a sequence of actions toward accomplishing a specific task under a certain scene, which is a new problem in computer vision research. The primary challenges are how to model task-specific knowledge and how to integrate this knowledge into the learning procedure. In this work, we propose training a recurrent long short-term memory (LSTM) network to address this problem, i.e., taking a scene image (including pre-located objects) and the specified task as input and recurrently predicting action sequences. However, training such a network generally requires large numbers of annotated samples to cover the semantic space (e.g., diverse action decomposition and ordering). To overcome this issue, we introduce a knowledge and-or graph (AOG) for task description, which hierarchically represents a task as atomic actions. With this AOG representation, we can produce many valid samples (i.e., action sequences according to common sense) by training another auxiliary LSTM network with a small set of annotated samples. Furthermore, these generated samples (i.e., task-oriented action sequences) effectively facilitate training of the model for semantic task planning. In our experiments, we create a new dataset that contains diverse daily tasks and extensively evaluate the effectiveness of our approach.