Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information

TL;DR

This paper introduces a novel imitation learning algorithm that automatically discovers hierarchical sub-task policies from unsegmented demonstrations by maximizing directed information flow, improving learning efficiency for complex tasks.

Contribution

The paper proposes a new algorithm that leverages directed information in a generative adversarial framework to learn hierarchical policies without pre-segmented data.

Findings

Automatically learns sub-task policies from unsegmented demonstrations

Connects directed information maximization with hierarchical policy learning

Improves imitation learning for complex, multi-modal tasks

Abstract

The use of imitation learning to learn a single policy for a complex task that has multiple modes or hierarchical structure can be challenging. In fact, previous work has shown that when the modes are known, learning separate policies for each mode or sub-task can greatly improve the performance of imitation learning. In this work, we discover the interaction between sub-tasks from their resulting state-action trajectory sequences using a directed graphical model. We propose a new algorithm based on the generative adversarial imitation learning framework which automatically learns sub-task policies from unsegmented demonstrations. Our approach maximizes the directed information flow in the graphical model between sub-task latent variables and their generated trajectories. We also show how our approach connects with the existing Options framework, which is commonly used to learn hierarchical policies.