The Logical Options Framework

TL;DR

The Logical Options Framework (LOF) is a hierarchical reinforcement learning approach that learns composable, satisfying, and optimal policies by integrating automata representations of tasks, enabling efficient adaptation to new tasks.

Contribution

LOF introduces a novel hierarchical RL framework that incorporates automata to learn and compose policies for complex tasks with theoretical guarantees.

Findings

LOF efficiently learns satisfying policies in various domains.

LOF can compose learned policies to solve unseen tasks with minimal retraining.

LOF demonstrates strong performance on discrete and continuous tasks, including 3D environments.

Abstract

Learning composable policies for environments with complex rules and tasks is a challenging problem. We introduce a hierarchical reinforcement learning framework called the Logical Options Framework (LOF) that learns policies that are satisfying, optimal, and composable. LOF efficiently learns policies that satisfy tasks by representing the task as an automaton and integrating it into learning and planning. We provide and prove conditions under which LOF will learn satisfying, optimal policies. And lastly, we show how LOF's learned policies can be composed to satisfy unseen tasks with only 10-50 retraining steps. We evaluate LOF on four tasks in discrete and continuous domains, including a 3D pick-and-place environment.