Specification mining and automated task planning for autonomous robots based on a graph-based spatial temporal logic

TL;DR

This paper introduces a method for autonomous robots to learn skills from videos using graph-based spatial temporal logic, enabling high-level task planning through specification mining and automated planning.

Contribution

It presents a novel approach combining GSTL-based specification mining with automated task planning for robots learning from demonstration videos.

Findings

Successfully mined GSTL formulas from demo videos.

Generated executable plans for a table setting task.

Demonstrated the approach's effectiveness in a practical example.

Abstract

We aim to enable an autonomous robot to learn new skills from demo videos and use these newly learned skills to accomplish non-trivial high-level tasks. The goal of developing such autonomous robot involves knowledge representation, specification mining, and automated task planning. For knowledge representation, we use a graph-based spatial temporal logic (GSTL) to capture spatial and temporal information of related skills demonstrated by demo videos. We design a specification mining algorithm to generate a set of parametric GSTL formulas from demo videos by inductively constructing spatial terms and temporal formulas. The resulting parametric GSTL formulas from specification mining serve as a domain theory, which is used in automated task planning for autonomous robots. We propose an automatic task planning based on GSTL where a proposer is used to generate ordered actions, and a verifier is used to generate executable task plans. A table setting example is used throughout the paper to illustrate the main ideas.