Principled Option Learning in Markov Decision Processes

TL;DR

This paper introduces a mathematically principled method for discovering useful options in Markov Decision Processes using information theory, leading to more efficient planning and autonomous option discovery.

Contribution

It provides a novel information-theoretic framework for identifying optimal sets of options, moving beyond heuristic approaches.

Findings

Proposes a mathematical characterization of good options sets.

Derives conditions for optimal options sets.

Demonstrates the algorithm's effectiveness in simulations.

Abstract

It is well known that options can make planning more efficient, among their many benefits. Thus far, algorithms for autonomously discovering a set of useful options were heuristic. Naturally, a principled way of finding a set of useful options may be more promising and insightful. In this paper we suggest a mathematical characterization of good sets of options using tools from information theory. This characterization enables us to find conditions for a set of options to be optimal and an algorithm that outputs a useful set of options and illustrate the proposed algorithm in simulation.