Reinforcement Learning for Classical Planning: Viewing Heuristics as Dense Reward Generators

TL;DR

This paper introduces a method that uses classical planning heuristics as dense rewards in reinforcement learning, significantly improving sample efficiency and generalization in classical planning tasks.

Contribution

It proposes a novel approach to integrate classical heuristics into RL as dense reward signals, enhancing learning efficiency and generalization in planning domains.

Findings

Improved sample efficiency over sparse-reward RL methods.

Effective generalization to new problem instances within the same domain.

Successful implementation using Neural Logic Machines.

Abstract

Recent advances in reinforcement learning (RL) have led to a growing interest in applying RL to classical planning domains or applying classical planning methods to some complex RL domains. However, the long-horizon goal-based problems found in classical planning lead to sparse rewards for RL, making direct application inefficient. In this paper, we propose to leverage domain-independent heuristic functions commonly used in the classical planning literature to improve the sample efficiency of RL. These classical heuristics act as dense reward generators to alleviate the sparse-rewards issue and enable our RL agent to learn domain-specific value functions as residuals on these heuristics, making learning easier. Correct application of this technique requires consolidating the discounted metric used in RL and the non-discounted metric used in heuristics. We implement the value functions using Neural Logic Machines, a neural network architecture designed for grounded first-order logic inputs. We demonstrate on several classical planning domains that using classical heuristics for RL allows for good sample efficiency compared to sparse-reward RL. We further show that our learned value functions generalize to novel problem instances in the same domain.