MO2: Model-Based Offline Options

TL;DR

MO2 introduces a model-based offline framework for discovering and transferring bottleneck options in continuous control tasks, improving sample efficiency and performance in long-horizon, sparse reward environments.

Contribution

It presents the first offline method for bottleneck option discovery in continuous spaces, enabling transfer and improved exploration in complex tasks.

Findings

MO2 outperforms recent option learning methods on complex tasks.

Offline bottleneck option discovery enhances exploration and value estimation.

Ablations show improved option predictability and credit assignment.

Abstract

The ability to discover useful behaviours from past experience and transfer them to new tasks is considered a core component of natural embodied intelligence. Inspired by neuroscience, discovering behaviours that switch at bottleneck states have been long sought after for inducing plans of minimum description length across tasks. Prior approaches have either only supported online, on-policy, bottleneck state discovery, limiting sample-efficiency, or discrete state-action domains, restricting applicability. To address this, we introduce Model-Based Offline Options (MO2), an offline hindsight framework supporting sample-efficient bottleneck option discovery over continuous state-action spaces. Once bottleneck options are learnt offline over source domains, they are transferred online to improve exploration and value estimation on the transfer domain. Our experiments show that on complex long-horizon continuous control tasks with sparse, delayed rewards, MO2's properties are essential and lead to performance exceeding recent option learning methods. Additional ablations further demonstrate the impact on option predictability and credit assignment.