Hierarchical Orchestra of Policies

TL;DR

The paper introduces Hierarchical Orchestra of Policies (HOP), a modular reinforcement learning approach that dynamically forms policy hierarchies to prevent catastrophic forgetting without needing task labels, showing strong results across diverse environments.

Contribution

HOP is a novel modular hierarchy-based method that mitigates catastrophic forgetting in continual reinforcement learning without requiring task labels.

Findings

HOP outperforms baseline methods in knowledge retention across tasks.

HOP performs comparably to state-of-the-art transfer methods.

HOP maintains performance when tasks are unchanged.

Abstract

Continual reinforcement learning poses a major challenge due to the tendency of agents to experience catastrophic forgetting when learning sequential tasks. In this paper, we introduce a modularity-based approach, called Hierarchical Orchestra of Policies (HOP), designed to mitigate catastrophic forgetting in lifelong reinforcement learning. HOP dynamically forms a hierarchy of policies based on a similarity metric between the current observations and previously encountered observations in successful tasks. Unlike other state-of-the-art methods, HOP does not require task labelling, allowing for robust adaptation in environments where boundaries between tasks are ambiguous. Our experiments, conducted across multiple tasks in a procedurally generated suite of environments, demonstrate that HOP significantly outperforms baseline methods in retaining knowledge across tasks and performs comparably to state-of-the-art transfer methods that require task labelling. Moreover, HOP achieves this without compromising performance when tasks remain constant, highlighting its versatility.