Sketch Decompositions for Classical Planning via Deep Reinforcement Learning

TL;DR

This paper introduces a deep reinforcement learning approach to learn sketch decompositions in classical planning, addressing scalability and expressivity limitations of previous methods, and demonstrating effective problem-solving across various domains.

Contribution

It formulates sketch learning as a DRL task, improving scalability and expressivity over prior feature-based rule methods in classical planning.

Findings

Decompositions often lead to successful goal achievement via greedy IW(k) searches.

The DRL approach produces understandable and effective problem decompositions.

Experimental results show improved scalability and applicability across domains.

Abstract

In planning and reinforcement learning, the identification of common subgoal structures across problems is important when goals are to be achieved over long horizons. Recently, it has been shown that such structures can be expressed as feature-based rules, called sketches, over a number of classical planning domains. These sketches split problems into subproblems which then become solvable in low polynomial time by a greedy sequence of IW$(k)$ searches. Methods for learning sketches using feature pools and min-SAT solvers have been developed, yet they face two key limitations: scalability and expressivity. In this work, we address these limitations by formulating the problem of learning sketch decompositions as a deep reinforcement learning (DRL) task, where general policies are sought in a modified planning problem where the successor states of a state s are defined as those reachable from s through an IW$(k)$ search. The sketch decompositions obtained through this method are experimentally evaluated across various domains, and problems are regarded as solved by the decomposition when the goal is reached through a greedy sequence of IW$(k)$ searches. While our DRL approach for learning sketch decompositions does not yield interpretable sketches in the form of rules, we demonstrate that the resulting decompositions can often be understood in a crisp manner.