Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning

TL;DR

This paper introduces DISH, a hierarchical policy framework that uses representation and reinforcement learning to enable flexible task adaptation and planning in high-dimensional environments by operating in low-dimensional latent spaces.

Contribution

DISH is a novel hierarchical policy distillation method that leverages latent variable models for flexible planning and control across multiple tasks without retraining.

Findings

Learned compact latent representations for complex humanoid control

Policy can adapt to new tasks like navigation without retraining

Effective in high-dimensional observation and action spaces

Abstract

We present a hierarchical planning and control framework that enables an agent to perform various tasks and adapt to a new task flexibly. Rather than learning an individual policy for each particular task, the proposed framework, DISH, distills a hierarchical policy from a set of tasks by representation and reinforcement learning. The framework is based on the idea of latent variable models that represent high-dimensional observations using low-dimensional latent variables. The resulting policy consists of two levels of hierarchy: (i) a planning module that reasons a sequence of latent intentions that would lead to an optimistic future and (ii) a feedback control policy, shared across the tasks, that executes the inferred intention. Because the planning is performed in low-dimensional latent space, the learned policy can immediately be used to solve or adapt to new tasks without additional training. We demonstrate the proposed framework can learn compact representations (3- and 1-dimensional latent states and commands for a humanoid with 197- and 36-dimensional state features and actions) while solving a small number of imitation tasks, and the resulting policy is directly applicable to other types of tasks, i.e., navigation in cluttered environments. Video: https://youtu.be/HQsQysUWOhg