Generalized Planning in PDDL Domains with Pretrained Large Language Models

TL;DR

This paper explores using GPT-4 to synthesize Python programs as generalized planners in PDDL domains, demonstrating strong performance with minimal training tasks and emphasizing the importance of automated debugging.

Contribution

It introduces a novel approach of using GPT-4 for program synthesis as generalized planners in PDDL domains, incorporating CoT summarization and automated debugging techniques.

Findings

GPT-4 effectively generalizes across multiple PDDL domains.

Automated debugging significantly improves planning accuracy.

Two training tasks often suffice for strong generalization.

Abstract

Recent work has considered whether large language models (LLMs) can function as planners: given a task, generate a plan. We investigate whether LLMs can serve as generalized planners: given a domain and training tasks, generate a program that efficiently produces plans for other tasks in the domain. In particular, we consider PDDL domains and use GPT-4 to synthesize Python programs. We also consider (1) Chain-of-Thought (CoT) summarization, where the LLM is prompted to summarize the domain and propose a strategy in words before synthesizing the program; and (2) automated debugging, where the program is validated with respect to the training tasks, and in case of errors, the LLM is re-prompted with four types of feedback. We evaluate this approach in seven PDDL domains and compare it to four ablations and four baselines. Overall, we find that GPT-4 is a surprisingly powerful generalized planner. We also conclude that automated debugging is very important, that CoT summarization has non-uniform impact, that GPT-4 is far superior to GPT-3.5, and that just two training tasks are often sufficient for strong generalization.