PLOTS: Procedure Learning from Observations using Subtask Structure

TL;DR

This paper introduces PLOTS, a method for procedural learning from observation that efficiently constructs open-loop action plans, leveraging repeated subsequences and optimistic exploration to outperform existing approaches in speed.

Contribution

The paper presents a novel approach for procedural learning from observation that is significantly faster than policy-gradient and model-based methods, especially in structured environments.

Findings

PLOTS is about 100 times faster than policy-gradient approaches.

Explicit procedural learning improves speed over traditional methods.

Optimistic action selection enhances performance in environments with latent structure.

Abstract

In many cases an intelligent agent may want to learn how to mimic a single observed demonstrated trajectory. In this work we consider how to perform such procedural learning from observation, which could help to enable agents to better use the enormous set of video data on observation sequences. Our approach exploits the properties of this setting to incrementally build an open loop action plan that can yield the desired subsequence, and can be used in both Markov and partially observable Markov domains. In addition, procedures commonly involve repeated extended temporal action subsequences. Our method optimistically explores actions to leverage potential repeated structure in the procedure. In comparing to some state-of-the-art approaches we find that our explicit procedural learning from observation method is about 100 times faster than policy-gradient based approaches that learn a stochastic policy and is faster than model based approaches as well. We also find that performing optimistic action selection yields substantial speed ups when latent dynamical structure is present.