Options Discovery with Budgeted Reinforcement Learning

TL;DR

This paper introduces BONN, a novel neural network model that automatically discovers hierarchical options in reinforcement learning by optimizing a budgeted learning objective, improving policy efficiency.

Contribution

The paper presents BONN, a new model for automatic option discovery in RL that does not rely on predefined options, advancing hierarchical policy learning.

Findings

BONN effectively discovers meaningful options in classical RL tasks.

The model achieves both quantitative improvements and qualitative insights.

Results demonstrate the potential of budgeted learning for hierarchical policy discovery.

Abstract

We consider the problem of learning hierarchical policies for Reinforcement Learning able to discover options, an option corresponding to a sub-policy over a set of primitive actions. Different models have been proposed during the last decade that usually rely on a predefined set of options. We specifically address the problem of automatically discovering options in decision processes. We describe a new learning model called Budgeted Option Neural Network (BONN) able to discover options based on a budgeted learning objective. The BONN model is evaluated on different classical RL problems, demonstrating both quantitative and qualitative interesting results.