Look Further Ahead: Testing the Limits of GPT-4 in Path Planning

TL;DR

This paper evaluates GPT-4's ability to perform long-horizon path planning using a new benchmark, revealing that specific prompting strategies improve performance but do not fully overcome the challenges of complex, extended trajectory tasks.

Contribution

It introduces a systematic benchmark for LLMs in path planning and analyzes GPT-4's capabilities and limitations with different prompt engineering techniques.

Findings

Prompting GPT-4 as Python code enhances planning effectiveness.

Decomposition of tasks improves trajectory generation.

GPT-4 struggles with optimality and generalization in long-horizon planning.

Abstract

Large Language Models (LLMs) have shown impressive capabilities across a wide variety of tasks. However, they still face challenges with long-horizon planning. To study this, we propose path planning tasks as a platform to evaluate LLMs' ability to navigate long trajectories under geometric constraints. Our proposed benchmark systematically tests path-planning skills in complex settings. Using this, we examined GPT-4's planning abilities using various task representations and prompting approaches. We found that framing prompts as Python code and decomposing long trajectory tasks improve GPT-4's path planning effectiveness. However, while these approaches show some promise toward improving the planning ability of the model, they do not obtain optimal paths and fail at generalizing over extended horizons.