Hierarchical Universal Value Function Approximators

TL;DR

This paper introduces hierarchical universal value function approximators (H-UVFAs) for hierarchical reinforcement learning, enabling better scaling, planning, and generalization in multi-goal settings by leveraging the options framework.

Contribution

It extends universal value function approximators to hierarchical settings using the options framework, with new methods for learning hierarchical embeddings and demonstrating improved generalization.

Findings

H-UVFAs outperform UVFAs in generalization tasks

Developed supervised and reinforcement learning methods for hierarchical embeddings

Enabled scaling and planning benefits in hierarchical reinforcement learning

Abstract

There have been key advancements to building universal approximators for multi-goal collections of reinforcement learning value functions -- key elements in estimating long-term returns of states in a parameterized manner. We extend this to hierarchical reinforcement learning, using the options framework, by introducing hierarchical universal value function approximators (H-UVFAs). This allows us to leverage the added benefits of scaling, planning, and generalization expected in temporal abstraction settings. We develop supervised and reinforcement learning methods for learning embeddings of the states, goals, options, and actions in the two hierarchical value functions: $Q(s, g, o; \theta)$ and $Q(s, g, o, a; \theta)$. Finally we demonstrate generalization of the HUVFAs and show they outperform corresponding UVFAs.