Hierarchical Variational Imitation Learning of Control Programs

TL;DR

This paper introduces a variational inference approach for hierarchical imitation learning, enabling autonomous agents to learn structured control programs more efficiently and accurately from demonstrations.

Contribution

It proposes a novel variational inference method for learning hierarchical control policies represented as program-like structures, improving data efficiency and generalization.

Findings

Outperforms LSTM baselines in data efficiency and accuracy

Achieves 24% error in bubble sort task with less data

Executes Karel programs flawlessly

Abstract

Autonomous agents can learn by imitating teacher demonstrations of the intended behavior. Hierarchical control policies are ubiquitously useful for such learning, having the potential to break down structured tasks into simpler sub-tasks, thereby improving data efficiency and generalization. In this paper, we propose a variational inference method for imitation learning of a control policy represented by parametrized hierarchical procedures (PHP), a program-like structure in which procedures can invoke sub-procedures to perform sub-tasks. Our method discovers the hierarchical structure in a dataset of observation-action traces of teacher demonstrations, by learning an approximate posterior distribution over the latent sequence of procedure calls and terminations. Samples from this learned distribution then guide the training of the hierarchical control policy. We identify and demonstrate a novel benefit of variational inference in the context of hierarchical imitation learning: in decomposing the policy into simpler procedures, inference can leverage acausal information that is unused by other methods. Training PHP with variational inference outperforms LSTM baselines in terms of data efficiency and generalization, requiring less than half as much data to achieve a 24% error rate in executing the bubble sort algorithm, and to achieve no error in executing Karel programs.