Using Large Language Models for Abstraction of Planning Domains - Extended Version

TL;DR

This paper explores using large language models, specifically GPT-4o, to generate abstract planning domains in PDDL from natural language objectives, validated by symbolic tools and experts, highlighting strengths and limitations.

Contribution

It introduces a novel approach employing in-context learning with LLMs to create abstract PDDL domains tailored to specific purposes, with new benchmark examples not seen during training.

Findings

GPT-4o can synthesize useful planning abstractions in simple settings

Better at abstracting actions than fluents

Validated by symbolic tools and human experts

Abstract

Generating an abstraction of a dynamic domain that aligns with a given purpose remains a significant challenge given that the choice of such an abstraction can impact an agent's ability to plan, reason, and provide explanations effectively. We model the agent's concrete behaviors in PDDL and investigate the use of in-context learning with large language models (LLMs) for the generation of abstract PDDL domains and problem instances, given an abstraction objective specified in natural language. The benchmark examples we use are new and have not been part of the data any LLMs have been trained on. We consider three categories of abstractions: abstraction of choice of alternative concrete actions, abstraction of sequences of concrete actions, and abstraction of action/predicate parameters, as well as combinations of these. The generated abstract PDDL domains and problem instances are then checked by symbolic validation tools as well as human experts. Our experiments show that GPT-4o can generally synthesize useful planning domain abstractions in simple settings, although it is better at abstracting over actions than over the associated fluents.