Exploiting Symbolic Heuristics for the Synthesis of Domain-Specific Temporal Planning Guidance using Reinforcement Learning

TL;DR

This paper advances temporal planning by integrating symbolic heuristics with reinforcement learning, enabling more effective heuristic guidance through residual learning and combined search strategies, thereby improving planning performance.

Contribution

It introduces a novel framework that exploits symbolic heuristics during RL and planning, including residual learning and multiple-queue search, to enhance domain-specific temporal planning.

Findings

Residual heuristic learning improves guidance accuracy.

Combining symbolic and learned heuristics balances search efficiency.

Experimental results outperform previous approaches.

Abstract

Recent work investigated the use of Reinforcement Learning (RL) for the synthesis of heuristic guidance to improve the performance of temporal planners when a domain is fixed and a set of training problems (not plans) is given. The idea is to extract a heuristic from the value function of a particular (possibly infinite-state) MDP constructed over the training problems. In this paper, we propose an evolution of this learning and planning framework that focuses on exploiting the information provided by symbolic heuristics during both the RL and planning phases. First, we formalize different reward schemata for the synthesis and use symbolic heuristics to mitigate the problems caused by the truncation of episodes needed to deal with the potentially infinite MDP. Second, we propose learning a residual of an existing symbolic heuristic, which is a "correction" of the heuristic value, instead of eagerly learning the whole heuristic from scratch. Finally, we use the learned heuristic in combination with a symbolic heuristic using a multiple-queue planning approach to balance systematic search with imperfect learned information. We experimentally compare all the approaches, highlighting their strengths and weaknesses and significantly advancing the state of the art for this planning and learning schema.