Possibility Before Utility: Learning And Using Hierarchical Affordances

TL;DR

This paper introduces Hierarchical Affordance Learning (HAL), a method enabling reinforcement learning agents to focus on feasible actions, improving efficiency and adaptability in complex hierarchical tasks by modeling and pruning impossible subtasks.

Contribution

The paper proposes a novel affordance-aware hierarchical reinforcement learning approach that handles incomplete symbolic specifications and enhances learning efficiency and flexibility.

Findings

HAL improves learning efficiency in complex tasks

HAL enables better navigation in stochastic environments

HAL facilitates acquisition of diverse skills without extrinsic supervision

Abstract

Reinforcement learning algorithms struggle on tasks with complex hierarchical dependency structures. Humans and other intelligent agents do not waste time assessing the utility of every high-level action in existence, but instead only consider ones they deem possible in the first place. By focusing only on what is feasible, or "afforded", at the present moment, an agent can spend more time both evaluating the utility of and acting on what matters. To this end, we present Hierarchical Affordance Learning (HAL), a method that learns a model of hierarchical affordances in order to prune impossible subtasks for more effective learning. Existing works in hierarchical reinforcement learning provide agents with structural representations of subtasks but are not affordance-aware, and by grounding our definition of hierarchical affordances in the present state, our approach is more flexible than the multitude of approaches that ground their subtask dependencies in a symbolic history. While these logic-based methods often require complete knowledge of the subtask hierarchy, our approach is able to utilize incomplete and varying symbolic specifications. Furthermore, we demonstrate that relative to non-affordance-aware methods, HAL agents are better able to efficiently learn complex tasks, navigate environment stochasticity, and acquire diverse skills in the absence of extrinsic supervision -- all of which are hallmarks of human learning.