SPOTTER: Extending Symbolic Planning Operators through Targeted Reinforcement Learning

TL;DR

SPOTTER is a framework that combines symbolic planning and reinforcement learning to automatically discover and extend planning operators, improving goal achievement in dynamic domains without supervision.

Contribution

It introduces SPOTTER, a novel integrated approach that uses RL to identify and add missing planning operators, enhancing planning robustness and transferability.

Findings

SPOTTER outperforms pure RL approaches in goal achievement.

It discovers transferable symbolic operators without supervision.

The framework effectively resolves planning model discrepancies.

Abstract

Symbolic planning models allow decision-making agents to sequence actions in arbitrary ways to achieve a variety of goals in dynamic domains. However, they are typically handcrafted and tend to require precise formulations that are not robust to human error. Reinforcement learning (RL) approaches do not require such models, and instead learn domain dynamics by exploring the environment and collecting rewards. However, RL approaches tend to require millions of episodes of experience and often learn policies that are not easily transferable to other tasks. In this paper, we address one aspect of the open problem of integrating these approaches: how can decision-making agents resolve discrepancies in their symbolic planning models while attempting to accomplish goals? We propose an integrated framework named SPOTTER that uses RL to augment and support ("spot") a planning agent by discovering new operators needed by the agent to accomplish goals that are initially unreachable for the agent. SPOTTER outperforms pure-RL approaches while also discovering transferable symbolic knowledge and does not require supervision, successful plan traces or any a priori knowledge about the missing planning operator.