Efficient Planning in Reinforcement Learning via Model Introspection

TL;DR

This paper introduces a novel approach that leverages model introspection, akin to program analysis, to enhance goal-oriented planning efficiency in reinforcement learning by connecting it with classical planning techniques.

Contribution

It proposes a new algorithm that enables efficient planning in relational reinforcement learning through model introspection, bridging reinforcement learning and classical planning.

Findings

Demonstrates improved planning efficiency in relational reinforcement learning.

Establishes a theoretical link between reinforcement learning and classical planning.

Provides a framework for applying introspection to various RL models.

Abstract

Reinforcement learning and classical planning are typically seen as two distinct problems, with differing formulations necessitating different solutions. Yet, when humans are given a task, regardless of the way it is specified, they can often derive the additional information needed to solve the problem efficiently. The key to this ability is introspection: by reasoning about their internal models of the problem, humans directly synthesize additional task-relevant information. In this paper, we propose that this introspection can be thought of as program analysis. We discuss examples of how this approach can be applied to various kinds of models used in reinforcement learning. We then describe an algorithm that enables efficient goal-oriented planning over the class of models used in relational reinforcement learning, demonstrating a novel link between reinforcement learning and classical planning.