Hierarchical Neural Dynamic Policies

TL;DR

This paper introduces Hierarchical Neural Dynamical Policies (H-NDPs), a hierarchical framework that learns local dynamical policies and distills them into a global policy from high-dimensional images, improving generalization and safety in dynamic tasks.

Contribution

The paper proposes a hierarchical deep policy learning framework that embeds dynamical system structure, enabling better generalization to unseen configurations from image inputs.

Findings

H-NDPs achieve state-of-the-art results in real-world and simulation dynamic tasks.

H-NDPs provide smooth trajectories, enhancing safety in real-world applications.

The approach integrates seamlessly with imitation and reinforcement learning methods.

Abstract

We tackle the problem of generalization to unseen configurations for dynamic tasks in the real world while learning from high-dimensional image input. The family of nonlinear dynamical system-based methods have successfully demonstrated dynamic robot behaviors but have difficulty in generalizing to unseen configurations as well as learning from image inputs. Recent works approach this issue by using deep network policies and reparameterize actions to embed the structure of dynamical systems but still struggle in domains with diverse configurations of image goals, and hence, find it difficult to generalize. In this paper, we address this dichotomy by leveraging embedding the structure of dynamical systems in a hierarchical deep policy learning framework, called Hierarchical Neural Dynamical Policies (H-NDPs). Instead of fitting deep dynamical systems to diverse data directly, H-NDPs form a curriculum by learning local dynamical system-based policies on small regions in state-space and then distill them into a global dynamical system-based policy that operates only from high-dimensional images. H-NDPs additionally provide smooth trajectories, a strong safety benefit in the real world. We perform extensive experiments on dynamic tasks both in the real world (digit writing, scooping, and pouring) and simulation (catching, throwing, picking). We show that H-NDPs are easily integrated with both imitation as well as reinforcement learning setups and achieve state-of-the-art results. Video results are at https://shikharbahl.github.io/hierarchical-ndps/