On the Planning, Search, and Memorization Capabilities of Large Language Models

TL;DR

This paper evaluates GPT-4's capabilities in planning tasks, identifying strengths and limitations, and proposes fine-tuning methods to enhance its Chain of Thought reasoning for planning applications.

Contribution

It provides a comprehensive analysis of GPT-4's performance in planning, introduces domain-specific fine-tuning techniques, and offers insights into improving large language models for planning tasks.

Findings

GPT-4 performs well in domain extraction and graph search planning.

Limitations are identified in adversarial planning scenarios.

Fine-tuning enhances Chain of Thought reasoning in planning tasks.

Abstract

The rapid advancement of large language models, such as the Generative Pre-trained Transformer (GPT) series, has had significant implications across various disciplines. In this study, we investigate the potential of the state-of-the-art large language model (GPT-4) for planning tasks. We explore its effectiveness in multiple planning subfields, highlighting both its strengths and limitations. Through a comprehensive examination, we identify areas where large language models excel in solving planning problems and reveal the constraints that limit their applicability. Our empirical analysis focuses on GPT-4's performance in planning domain extraction, graph search path planning, and adversarial planning. We then propose a way of fine-tuning a domain-specific large language model to improve its Chain of Thought (CoT) capabilities for the above-mentioned tasks. The results provide valuable insights into the potential applications of large language models in the planning domain and pave the way for future research to overcome their limitations and expand their capabilities.