A Study of Value-Aware Eigenoptions

TL;DR

This paper explores how eigenoptions can improve credit assignment and exploration in reinforcement learning, revealing their benefits and challenges through empirical evaluation and proposing methods for deep RL.

Contribution

It demonstrates that pre-specified eigenoptions enhance credit assignment and exploration, and introduces a new approach for learning option-values with non-linear function approximation.

Findings

Pre-specified eigenoptions aid exploration and credit assignment.

Online discovery of eigenoptions can bias experience and hinder learning.

Proposed method for learning option-values impacts deep RL performance.

Abstract

Options, which impose an inductive bias toward temporal and hierarchical structure, offer a powerful framework for reinforcement learning (RL). While effective in sequential decision-making, they are often handcrafted rather than learned. Among approaches for discovering options, eigenoptions have shown strong performance in exploration, but their role in credit assignment remains underexplored. In this paper, we investigate whether eigenoptions can accelerate credit assignment in model-free RL, evaluating them in tabular and pixel-based gridworlds. We find that pre-specified eigenoptions aid not only exploration but also credit assignment, whereas online discovery can bias the agent's experience too strongly and hinder learning. In the context of deep RL, we also propose a method for learning option-values under non-linear function approximation, highlighting the impact of termination conditions on performance. Our findings reveal both the promise and complexity of using eigenoptions, and options more broadly, to simultaneously support credit assignment and exploration in reinforcement learning.