When Do Skills Help Reinforcement Learning? A Theoretical Analysis of Temporal Abstractions

TL;DR

This paper provides a theoretical framework for understanding when skills, as temporal abstractions, improve reinforcement learning, highlighting that their usefulness depends on environment properties like state compressibility and exploration needs.

Contribution

It offers the first formal characterization of the conditions under which deterministic skills enhance RL performance, including insights on the limitations of macroactions.

Findings

Skills are less useful in environments with less compressible solutions.

Skills primarily aid exploration rather than learning from experience.

Unexpressive skills like macroactions may hinder RL performance.

Abstract

Skills are temporal abstractions that are intended to improve reinforcement learning (RL) performance through hierarchical RL. Despite our intuition about the properties of an environment that make skills useful, a precise characterization has been absent. We provide the first such characterization, focusing on the utility of deterministic skills in deterministic sparse-reward environments with finite action spaces. We show theoretically and empirically that RL performance gain from skills is worse in environments where solutions to states are less compressible. Additional theoretical results suggest that skills benefit exploration more than they benefit learning from existing experience, and that using unexpressive skills such as macroactions may worsen RL performance. We hope our findings can guide research on automatic skill discovery and help RL practitioners better decide when and how to use skills.